Skytap was in full force at Electric Cloud’s second annual DevOps Enterprise Summit (DOES) last week—in the Expo, recording podcasts with the show’s amazing speakers, and sitting in on every keynote and track session possible. Our goal was to get a better grasp on how DevOps is being defined (a hundred different ways), how DevOps being implemented (a thousand different ways), and how DevOps success is being measured (the sky’s the limit).
I kept a running diary of my thoughts, and those shared by others, in hopes of getting a clearer picture of where DevOps is today, and where it’s going. My hope is that by sharing the amazing stories I heard throughout the week, I can do my part in helping others reach the value that DevOps is bringing to a quickly growing number of the world’s most successful enterprises.
Like the title of Elisabeth Hendrickson’s closing keynote, “It’s all about feedback”—so, provide your own! What did you think of the show? What impressed you the most? And how awesome was Jason Cox’s story of DevOps at Disney? Reach out to me on Twitter @Skytap, and let’s shorten our own feedback loop by keeping the conversation going!
Note: All italicized content below was jotted down live during the presentations I attended.
Day 1
Opening Keynote
Gene Kim
Gene just pointed out that the conference grew from selling out all 600 tickets in 2014 to 1200 in 2015—impressive. I would bet that’s largely due to Gene’s other point, “We no longer believe that DevOps is capable in large organizations, we know it works in large organizations.” Looking forward to seeing just how valuable it’s been.
(Re)building an Engineering Culture: DevOps at Target
Ross Clanton, Director
Heather Mickman, Senior Group Manager
I love that Target’s Ross Clanton and Heather Mickman are being so blunt and honest about the “pre-DevOps” days at Target. Heather:
“Target lost site of the importance of engineers years ago. We knew that had to change…We had silos within silos…It took ten teams to provision a single server—our IT organization was a mess. Our development process was mostly waiting in queues.”
Ross: “We get a lot of credit, but we’re still just starting this journey – we’re doing cool things, but we’ve got a long way to go.”
Target really set the tone of the show, not just by sharing their incredible successes after getting their development and operations teams aligned, but by telling stories of just how bad things were a relatively short time ago. Though some say that the principles around DevOps aren’t new, the rewards that come from successful DevOps initiatives certainly are, and this makes the struggles and failures far more relatable to many.
DevOps in the Enterprise: A Transformation Journey Powered by Metrics, Empathy and Empowerment
Jody Mulkey, Ticketmaster Chief Technology Officer
Jody’s football analogy: For too long, development and operations have been viewed as opposite teams. In football, this would be operations seen as the defense trying to stop the developers (offense) from scoring. We need to view operations as the offensive line, trying their hardest to give the developers the time they need to score. Love this.
From 2011 to 2014, Ticketmaster had a 230% increase in the number of developers, and a 12% increase in Ops. Mulkey: “That’s…bad.”
Average mean time to repair a bug was 47 minutes, now it’s 3.8.
More great stories of real struggles: Taking forever to repair bugs, departments that view themselves as opponents of others, long wait times around provisioning—you’re going to keep hearing these same stories, because it’s unfortunately still more common to have these struggles than it is to have eliminated them.
John’s story was also great because he talked about how Ticketmaster had become, “burdened by our legacy,” and how DevOps is absolutely applicable to legacy mainframe systems. Ticketmaster’s ticket engine has generated $25 billion in revenue, and had its first code commit in 1976. This is the system and the teams that support it that Ticketmaster has been able to take an average mean time to repair a bug down from 47 minutes to only 3.8 minutes today.
DevOps and Innovation at USAA and IBM
Michael Bueche, AVP IT Operational Excellence, USAA
Dibbe Edwards, VP Development, DevOps for Hybrid, IBM
Suggests starting small and approaching in layers when introducing a cultural change as large as DevOps to your organization. (Like our own suggestion of, “Biting DevOps into chunks you can chew and not choke on”)
“We as humans are really bad at the time between an action or decision, and getting the outcome. It’s like the ‘hot stove analogy’. When you put your hand on a hot stove, how long does it take to realize that it hurts? It doesn’t take a week. It’s just like if you can’t discover a bug until 6 weeks after a developer put it in there, how hard is it to find who contributed it, and even when you do, how hard will it be for them to remember doing it, or how it happened? This is the shorter feedback loop needed so that actions/outcomes are easy to connect” – Michael Bueche
Dibbe: “We had to make sure we had scalable environments within our organization in order to be successful with our own DevOps initiatives…and we’re always looking for ways to improve.”
I’d somehow never heard the hot stove analogy before Michael shared it, but it’s so applicable to DevOps, agile, or just modernized software delivery in general. This is one feedback loop that absolutely must be shortened in order to meet deadlines and keep bugs out of production—and as Dibbe pointed out, you have to always look for ways to shorten it further.
By providing your dev/test teams with the ability to self-provision the scalable environments they need, they’re then testing earlier and more often, and snapshotting those bugs in state, so that developers can easily reproduce bugs and eliminate them. As Jez Humble said later in the week, “Make ‘It works on my machine,’ mean something useful.”
How DevOps is Enabling Lean Application Development
Carmen DeArdo, DevOps Technology Leader, Nationwide Insurance
Carmen says that Nationwide was looking to outsource software delivery, and that departments were under a lot of pressure to prove that was unnecessary—budget could be reduced from reducing dependencies, wait times, and unplanned work.
Another great sports/software analogy (are there more of these I don’t know? These are great!): “Without a lack of integration of the countless tools being used by your teams, it’s like trying to coach a basketball team when you can’t even see the game.”
I really loved Carmen’s presentation. With over 200 agile teams making significant quality and productivity improvements, Nationwide still became burdened by the same wait states that organizations of any size commonly feel—just on an obviously very large scale. So what did Carmen and Nationwide do? They drove, “continuous delivery utilizing DevOps, Lean and Agile techniques across Mobile, Distributed and Mainframe and other technologies.”
How’s that working for them?
This was all just from Day 1 – and according to my Skytap colleagues who were also in attendance, I missed other sessions that were equally impressive! Look out for two podcasts we recorded on-site to publish soon that will cover these additional sessions!
Day 2
After a calm and peaceful breakfast at Boulettes Larder at The Ferry Building, day 2 took off at a blistering pace, right where day 1 left off.
Banking on Innovation & DevOps
Tapabrata Pal, Product Manager, Capital One
Tababrata: “Why are we open-sourcing our tools? It’s the right thing to do. They contribute to a culture of continuous experimentation and learning, and open sourcing makes it better.”
Those were actually the only notes I took during Tapabrata’s keynote, but that doesn’t mean there wasn’t anything else memorable about it. Far from it, to be honest. But I loved that the mere mention of open source tools, and the simplistic answer, “It’s the right thing to do…because they make it better,” drew tremendous applause from the packed ballroom. There was almost a standing ovation.
Tapabrata went on to point out other areas that Capital One is excelling at getting the fast feedback that they need to keep both their employees and their customers delighted. Teams have a resource called, “Office Hours,” where they can go for help with any project they’re working on, and the “voice of the customer” program allows customers to point out where bottlenecks exist—something traditionally only pointed out internally. I love this idea.
“I’m Not Building Web Software – Why Should I Care About Continuous Delivery?”
Panel hosted by Jez Humble
“If you build quality in, it’s going to get to market faster”
“Things get more expensive the later you find the bugs. In embedded software, this can become much more serious. Automobiles, medical devices—the need for high quality, secure software is an absolute must.”
“What culture is needed to make these changes?” (Question from the audience)
“Products are very easy to invest in, and IT cannot be looked at as only a cost center…Start by doing what’s right for the business to get them on board.”
Even though this panel was designed to speak specifically to the embedded software industry, the group’s discussion was applicable from mainframes to mobile and everything in between. Everyone is talking these days about how the cost of finding bugs late in the delivery lifecycle far exceeds when they’re found earlier on, and well before they make it into customers’ hands. “Building quality in” may require serious disruption of the status quo, but no matter how familiar or comfortable teams are with doing things, “the way they’ve always been done,” how much longer can you afford to continue down that path?
And like this panel said, “IT cannot be looked at as only a cost center.” The same goes for software testing—a stage of software delivery also often looked at as a cost center, or roadblock to getting features or releases out on time. On-demand virtual environments, DevOps, continuous testing, and other modernizations to the entire delivery process are changing the way these teams are looked at, and are giving them the power to have a real impact on software speed and quality.
Disney DevOps – The Enterprise Awakens
Jason Cox – Systems Engineering Disney Internet Group – Web Operations
I didn’t actually take a single note during this session because I didn’t want to miss a single word that Jason said. Obviously his incredible Star Wars tie-ins only two months before The Force Awakens hits theatres played tremendously to Jason’s favor, but his session would’ve still been a huge hit even without them.
I’m not sure anyone all week was more revealing about how prevalent red tape, bureaucracies, silos, and weekend war rooms were at their organizations. But it was Jason’s obvious honesty and passion when he said, “It takes being inspired to change,” that had all of us in attendance ready to take that level of inspiration back to our own teams.
Like so many of us have pointed out a million times: none of this is easy. DevOps, agile, continuous integration/testing/deployment/delivery—it’s all hard. Sometimes saying, “just start somewhere,” isn’t enough. The value that these changes bring doesn’t come from simply wanting things to be better. As Jason said, you have to be inspired. If you’re lacking that inspiration, I highly recommend checking out one of Jason’s sessions anytime you get the chance to see one.
Architecting for Continuous Delivery
Jez Humble, Author, Continuous Delivery
“Who is doing continuous integration?” (almost everyone, 1000 people, raise their hand)
“Who is fixing bugs within ten minutes of them being found?” (everyone laughs and puts their hands down)
“You should be able to push a button to release to production”
“Every change is in the build, and every build is a release candidate…Software should always be testable, and always deployable”
“Developers have to care about this stuff from the start. The DevOps fairies cannot ‘secure it’, ‘reliable it’, or ‘deployable it’. You HAVE to build this stuff in as early as possible.”
“We have to go back to some very practical aspects of software development that we’ve known about since the 1970’s.”
“Most unicorns are just horses with really good PR”
I have no real problem with the ambiguity around the definition of DevOps. I know the lack of a concrete, universally accepted definition drives some people mad, but I like Gene Kim’s approach of defining DevOps by its outcomes.
It’s no surprise that I also love Jez Humble’s definition of continuous delivery (CD):
Perhaps the reason this definition is so popular is that Jez also uses the outcomes to define CD, and not a single prescriptive all-or-nothing approach.
I don’t know how many of you are fellow Jez Humble fanatics (there’s a lot of us) but this is what it’s like every time he speaks or writes a new book. The whole world goes crazy with, “why didn’t I think of that?” “Holy cow, that was great,” and, “Okay, we’re doing that where I work.”
After second day of awesome sessions, it was time to head out for “An Evening at the Exploratorium,” Electric Cloud’s party for DOES attendees!
Day 3
Test Automation for Mainframe Applications
Rosalind Radcliffe- Distinguished Engineer, Chief Architect for DevOps and CLM, IBM
Rosalind Radcliffe kicked off the final day at DOES, and she quickly impressed the entire room with both her 26 years of engineering experience at IBM, and her approach to modernizing mainframe systems through virtualization. This is the same approach that Skytap and many of our partners prescribe to any development and test teams who are being constrained by hardware that can be hard or even impossible to access when it’s needed most.
Rosalind’s keynote was outstanding, and she was one of numerous speakers from companies who proved that DevOps practices are being successfully introduced even at the mainframe level.
Immutable Awesomeness?: Where Containers Collide with SW Supply Chains
Joshua Corman – CTO, Sonatype
John Willis – Director of Ecosystem Development, Docker
“IT Ops has been lost at sea for 20 years; DevOps is how we make our way back to land.” – John Willis
“I do what I do because I want to save lives. Our dependence on software is growing due to embedded medical devices, our cars, homes—I want these things to work.” – Josh Corman
“We need building codes for building code.”
“If you’re not excited about this—leave now” – John Willis
It would be easy to give the “Best Session” award to Disney’s Jason Cox, especially as the Star Wars fanatic that I am, but I think it actually goes to Joshua and John’s “Immutable Awesomeness” Wednesday keynote. When Joshua tells you that he’s not just passionate about software, or DevOps, and that it’s saving lives that he’s passionate about—you believe him.
Using the 2010 earthquakes in Haiti and Chile as examples of the difference that quality being, “built in,” can make—Joshua pointed out that while Haiti’s 7.0 magnitude quake killed 230,000 people, Chile’s significantly stronger 8.8 quake resulted in 279 deaths. One reason for the incredible disparity between the two numbers was the strict building codes in Chile, and the lack of such codes in Haiti.
“We need building codes for building code,” said Corman. Our reliance on software to facilitate social interactions or assist mobile business transactions, is rapidly moving into also depending on the software found inside complex medical devices, and our automobiles. Those responsible for keeping these connected devices running and protected from hacking and failures have an enormous responsibility for the quality of their work.
It’s All About Feedback
Elisabeth Hendrickson – VP of Engineering, Pivotal’s Big Data Suite
“More testers does not equal better quality”
Avoid: Polluted feedback streams, false alarms/failures, distortion, lost information
It was fitting that DOES came to a close with the mantra, “It’s all about feedback.” Gene Kim explained at the start of the week, that he’d asked each speaker to share with their audience the areas where they, too, are still looking for help and feedback, no matter what stage of DevOps they and their organizations were at. Scaling DevOps was hands-down the most popular request for help.
It always sounds a bit simplistic when anyone says that anything is, “all about” a single something. But I’m not sure that stating that DevOps is all about feedback oversimplifies it at all. There are so many sources of valuable feedback that have to be considered when beginning any new initiative, especially in software delivery.
From developers, to testers, to operations, to the users/customers of your software, to security, to the system itself—DevOps requires fast feedback from each of these sources, and with the amazing stories we heard this week from organizations who’ve been able to act upon that feedback—maybe this really is what it at least is largely all about.
Here’s hoping my own feedback helps you in some way!
