The one where all hell breaks loose

guy

Don’t you love a good story? This one is a DevOps story in a fictional enterprise. It’s the kind of story you’ve seen replayed countless times in large organizations.

The backstory

DChip Tech is a semiconductor company. They’re in the business of making chips for wearable computing since acquiring Enogo, a startup in France for simulated testing in wearable technology.

Three months into the acquisition on a late Sunday night in France, the VP of Operations at Enogo considers his late evening ahead.

Chandler is not a happy man despite his recent appointment as the VP of Operations. How can he be? How can a 40-person company in France give him so many sleepless nights? And tonight is no exception.

DChip’s strategic momentum in the last months had everything to do with it. To leverage the cream of the crop engineering talent in wearable tech, DChip, the San Francisco Bay area chip giant invested in Enogo.

  • Now DChip wants Enogo systems fully integrated into its OpenStack datacenter to reduce infrastructure costs.
  • Although Enogo hosts on AWS, DChip doesn’t want to integrate AWS Europe into the corporate network. They want to keep the intellectual property in the US.
  • It falls on Chandler’s lap to integrate the systems of both the companies and reorganize IT, to speed up DChip releases from once every six months to the weekly push like at Enogo. Imagine all that.

Tall order, he thinks wryly, gulping down another cup of coffee to keep him awake for the Monday morning US time leadership meeting.

Where all the action happens

He dials into the video conference warily, his head already starting to throb. First to appear on the screen, Phoebe, the CIO is looking none too happy either. Soon all the division heads appear on separate screens including the engineering and QA heads from Enogo.

Dismissing the usual small talk, Phoebe launches right to the point. “We have a firm deadline to move out of AWS cloud in three months. Yet, where do we stand now? Can someone explain why the delay in integrating the systems of both companies? Do you know how much this is slowing down our go-to-market strategy?”

Monica, the Chief Architect in charge of the Datacenter OpenStack cluster, says, “We’d love Enogo to move to OpenStack today. But it’s not fully stable.”

As if to emphasize the point, Rachel the engineering manager whose team is building the wearable emulation test software adds, “We can’t wait for the OpenStack cluster to work. Our engineers need test environments like yesterday. That’s why using AWS makes sense.”

The marketing and sales VPs quietly watch as everyone joins in the blame game.

Not to be left out, Joey the Enogo QA Manager complains that he can’t deploy the image that contains the emulation software because OpenStack is corrupting it. “All the more reason to support AWS,” he quips.

The ultimatum

I’ve heard enough; Phoebe bellows over the din of the meeting. She turns slowly to glare at Chandler through the video camera. “Chandler, do you care to explain why we have an unforeseen $100,000 dollar expense this month? I want an explanation.”

“It’s a case of a simple, human error; it can happen to anyone Phoebe,” Chandler tried calmly to explain. “One of the devs launched instances and forgot about it until we got the bill.”

“Yea” piped up Janice, the VP of Finance. You’ve been paying the AWS bills out of your credit card, Chandler, and then billing the company after the fact,” she so helpfully points out.

Chandler stands dumbfounded as Phoebe’s final words struck his heart: “I told you to shut down operations on AWS. I want to move to our datacenter on OpenStack. No more bills from AWS Chandler, you hear me?”

Her words rang late into the night and the next day.

(What do you think happens next? Catch the next episode of the DevOps story. Any resemblance to these characters or events in real life is wholly intentional.)

Hacker News

Categories: AWS, DevOps, OpenStack
Tags: , , , , ,
  • Raul Lapeira

    lol, this was a good one… 100.000 in instances sounds like a big doh!.

    • Mrina N

      It’s possible to hit that dollar figure if you run for example 80 such C3 extra large instances all month long. We sure hope you never do! Thanks for enjoying the story Raul. Catch the second episode here: https://elasticbox.com/blog/all-the-deployments-stall/

  • Hévellyn Natasha

    Okay, so we also have a FRIENDS fan on this :)