In this chapter of our technical series, “Scaling Modern Software Delivery,” we sit down with Derek Weeks, Sonatype VP and DevOps advocate, to discuss the benefits and challenges of building—and governing—a modern DevOps toolchain. Derek also provides some incredible thoughts on the responsibility we all share for delivering the highest quality software possible as software is embedded into more and more devices.
We hope you enjoy this episode, our previous DevHops episodes, and the rest of this technical series!
Noel: Our theme today is, “What’s in your Toolchain?” If you start off a DevOps strategy by discussing only the technologies involved, you end up really missing the point. There’s no standard vendor solution available today that can fit all of your needs just by itself. But we are starting to see Toolchain’s that contain many of the elements needed for successful transformation. Those tools encompass organizational process and technology implements kind of working hand in hand. Before we dive into toolchains specifically, I was going to ask you how you seeing that DevOps has changed software delivery as we know it today.
Derek: Yeah. That’s a great question. It’s obviously changed software as we know it today in a lot of ways. I think one is, as I work in this space, I’m very excited about the amazing amount of innovation that’s going on. This is probably one of the greatest innovation waves that we’ve ever, ever been in. Even at the start of this wave, there is tremendous energy behind DevOps coming into play. For those that have headed down the path and have certainly seen their early successes in it, I think there is no going back, right?
The ways of old and doing things as we used to is no longer acceptable. There are organizations that are learning to swim with the tide and those that aren’t are going to get crushed. I think the way that it’s changed is really … There are organizations that have figured out how to become very high performance organizations where they’re moving from software development and operations as being less of an art and more of production like as in manufacturing.
Where they look at … we have certain components that we are producing and those components can be systems, applications, containers, microservices, what have you, and they’re saying, “How do I organize these components, manage and store these components, distribute the components, assemble the components into finished products or goods to make my business more higher performance?”
They’re taking a lot of lessons learned from even those as way back or as far back as Deming, these manufacturing, lean and agile manufacturing processes that have made it into every other modern manufacturing industry. From automobiles to food services to pharmaceuticals, where they know absolutely everything that’s going into their products. They are using the highest quality components for those. They can track and trace and monitor the health of those components and their ecosystem, their manufacturing lines, to such a degree that high precision is part of how they normally work.
I think that there are software organizations that are fully embracing these principles and really coming out as much more high performance organizations, performing at a lower cost, being much more innovative, removing considerable waste from their infrastructure and from their operations. I think that is a huge amount of how it’s changed is software delivery taking on that more of the production style or manufacturing view and less of the art kind of a view that everything’s unique and as unique as a snowflake. They’re saying, “We can really control these environments if we treat it more in a systematic way.”
Jason: That’s great Derek. I think that we’ve … Basically, software development has kind of given itself a pass on that kind of discipline over the last two decades and it’s really coming around now. What’s very interesting about what’s changed about software delivery itself is it also it’s become so collaborative and that you really are … The art is really in getting these teams to work together, and have that shared empathy, and have the ability to put themselves in each other’s shoes so that you have development and operations and security and everybody kind of coming to the table to have that kind of goal.
That’s really a lot of how it’s changed things, but everything you’re talking about, bringing the discipline of manufacturing forward into the DevOps today is just exactly what I would say.
Derek: Yeah. I think there are a lot of organizations, when they hear, if you kind of take it down to the manufacturing analogy that you’re looking at and dealing with parts from suppliers that you’re bringing into your DevOps toolchain or software supply chain or what have you, and those parts can be the systems that you configure, the applications that get configured, the software components that go into your applications, the security and governance policies that can be created as code. A lot of people find, “Well that sounds kind of restrictive,” that you know, you’re standardizing everything.
Once you realize the freedom that it gives you, and the time that it frees up for you to do those kind of things, and the velocity at which you can move when you use these kind of technologies to your advantage, and you manage them to your advantage—that’s really where it becomes exciting. I don’t see it as constraining whatsoever. I see it as you can move faster and be more innovative than anyone else and you can come out a winner which I think all of us want to do.
We want to develop software that people want, and that really meets customer’s needs and we can deliver it at a speed that they expect.
Noel: That’s great. That’s a great kickoff into this topic and looking at the toolchains specifically. Let’s talk about “borrowing what works.” I was curious, Derek, as to what extent have you seen that become prevalent in the software market today?
Derek: Oh my gosh. It’s everywhere. One is, I think Jason just mentioned, the collaboration between people not only internally but across organizations, trying to figure out, “What’s working in your organization and can I use some of that over here?”
We see it in the organizations that have built something like a software tool, whether that’s an automation tool or a dashboard or something like that, and making it available as open source so that others can borrow what works in that regard to better innovate and automate their operations.
I think one of the things that we certainly see at Sonatype in mass is the, if you will, borrowing of open source components to be used in application development. Some of the listeners might not know, Sonatype and part of what we do, is host the Maven Central Repository which is the largest open source repository in the world for Java components. Last year we served 31 billion download requests for open source components.
Noel: Wow.
Derek: And there are only 11 million developers in the world that would need these billions of components. Basically, there are organizations and open source projects out there that are developing pieces of code that other developers no longer have to write themselves. This is creating a massive change in the way software is being developed, where now it’s being more assembled than written from scratch. That’s allowing us to move much faster.
We’re seeing that in the organizations that are delivering new products faster. We’re also seeing it in the products that didn’t normally have software in them, now have software in them because it’s easier to create some of the products that are coming to market, because of these reusable components and so forth. The other thing that I see which is really interesting for me and Jason, I might have shared this with you in our initial conversations, but I’ve put together a DevOps reference architecture deck. DevOps and continuous delivery reference architectures.
I put it out on SlideShare earlier last year. That slide deck has 67,000 views on it in the last 10 months or so. A lot of what’s in there, they’re just reference architectures from a lot of different companies using a lot of different tools and a lot of different ways. There are some commonalities between them but I think people really went to that deck and are using it as a reference point for, “Hey. I’m going to get started. What have other people done? What can I borrow from them that works?” They say you know, “We’ve already been down the path. We’re down the path in doing what we’re doing. I want to validate that what we’re doing actually looks like what other people were doing.” I think that kind of collaborative, free sharing nature of DevOps and the practices that are going on is just part of what’s fueling this innovation wave and the energy behind it.
Jason: Yeah, it really has been a transformation of the industry as a whole in the way that we’re seeing teams really … Some of these DevOps meet-ups, the forums on DevOps, the way that people are actually interested in meeting with peers who are doing things in other companies. It’s created a different environment for this sort of thing where we used to think of innovation as a proprietary thing that happens inside your company. It’s really … It’s almost unstoppable when you think about an industry as a whole kind of embracing open source and embracing the sharing of information to this extent.
Derek: Oh yeah. When I started off my career, and I spent a number of years at HP as a big organization, I can’t imagine sitting down with a bunch of other software development organizations and sharing stuff about kind of the inside secrets and traits about what we’re doing and how we’re approaching it. It’s like, “Why would you ever do that?”
Now there are whole conferences, and DevOps Days, and meetups built upon that premise of wanting to share and no one’s going to steal your secret sauce just from you telling people how to work better. It’s really, I think a refreshing change for those that are participating. They’re benefiting tremendously from that kind of non-proprietary view of the world. It’s certainly from a process standpoint and cultural understanding standpoint that people need to share those lessons learned.
Noel: As we’re talking about borrowing what works and all of the collaboration that’s around DevOps, how do you suggest that organizations look at controlling and deciding what makes it into their DevOps toolchain, and how do you maintain governance over that with so many places to borrow from, and that sort of thing. How do you decide who gets to put things in here? What is that process to maintaining that governance?
Derek: Yeah, that’s also a great question. This is something that I spend a lot of time on, and I’m going to maybe have an answer too specific to Sonatype and the work that I’m doing but I think it’s totally relevant. When we think about kind of what makes it into the DevOps toolchain, a lot of what I’ve gone out and spoken about in the last six months or so, is the open source components that organizations are using to develop their software and their applications. On one side, those components are dramatically improving, expediting development, improving the innovation that we’re doing, accelerating our time to market, but as I mentioned earlier, we’ve seen 31 billion download requests of components coming from 11 million developers. In some cases, it’s a free for all which leads to very little control over what’s coming into an organization.
Where you have any developer able to pull any open source component and put that into an application, they’re pulling these components from unknown suppliers of unknown origins, sometimes of unknown quality or security or integrity. Those components are freely making it into the software supply chain and being orchestrated or assembled through the DevOps toolchain.
When we looked at the volume of all these downloads, what I saw in one of the parts of my analysis was, 1 in 16 components that were downloaded had a known security vulnerability associated with them. For an average development organization, 1 in 16 may or may not sound like a lot, but average development organizations downloading nearly a quarter million components a year and that works out to about 15,000 or so vulnerable components on average being consumed, we can’t imagine any other modern industry on earth operating this way.
Noel: Right.
Derek: You can’t imagine manufacturing a car where 1 in 16 parts is known to be defective. You can’t imagine creating bread or potato chips or anything else where 1 in 16 ingredients that you’re putting in has a know issue, or pharmaceuticals or whatever. Other industries have figured out how to really manage what’s going in their processes, what parts they’re using, and if anything goes wrong with those parts, they can track and trace them throughout the process.
I think we saw with a lot of the security vulnerabilities that came out in 2014/2015, there were a lot of organizations asking, “Did we use that OpenSSL Heartbleed thing? If yes, where did we use that?” There was a lot of, “Well, it’s going to take us like, two weeks, four weeks, six weeks to find maybe where we put those.” In some cases, they put them in places that couldn’t be patched and needed to be completely replaced. I think the “control” factor is something that needs to come more into play. I think we probably have a better chance to manage it now than ever before. I think we’re still at the early stages of really understanding what the benefit is of controlling what gets into the toolchains and how it becomes governed.
Jason: Yeah. That’s great Derek. I mean we’re seeing there’s a whole constellation of what we would call “tools” that could make it into a toolchain. A lot of times there’s too much emphasis on saying a specific set of tools is what you need, but a company does need to standardize them, or something in order to have a working, repeatable process. You know as far as automating the … Here is the reference infrastructure that we’re going to be using. Here is the framework for continuous integration and here’s our framework for delivering automated builds to each successive test environment.
If I can do that as a pattern and make it repeatable then we do need to … A company itself does have to come to an agreement on what is in that toolchain. It’s not a … There’s not a prescriptive set that I would say you have to do, but you do have to say, “I’m trying to automate at every stage of the process and put a chain in place and then we can improve upon it. Each time we set up a new developer, a new team to contribute, we want them to be using as much of that reference toolchain as we can.”
Derek: Yeah. You know in the reference architecture deck that I mentioned previously and for any listeners that want to find it, if you go to slideshare.com, and you just search “DevOps” on that site, it’s the first deck that comes up. I think because it’s been so popular.
One of the things that I noted in that deck was there are certainly a number of standard components in toolchains that people utilize. When you look at those and you see Maven, you see Jenkins, you see Docker, you see Rundeck, Puppet, or Chef, you see Sonarqube, you see Nexus from Sonatype, you see Tomcat, you see GitHub, Gitlab, those are very common components across almost all of those pictures that people have begun to standardize what’s in there.
I think there’s also on the flip side, there are a lot of organizations and you probably have run into this too that think, “Hey. I want to do continuous integration. I’m going to bring in Jenkins.” Just because you have Jenkins in place doesn’t mean you’re doing continuous integration.
Jez Humble preaches about this a lot. He has his checklist of, “Here’s what you’re doing if you’re really doing continuous integration.” Most people aren’t doing that. Inspirationally they want to get there but just because they implement them first doesn’t mean that there’s a lot that they can … Just because they have the tool, doesn’t mean that they’re doing continuous integration.
Noel: When you were talking about the responsibility of not introducing these vulnerabilities as we’re doing all of this learning and collaborating, I got to see Joshua Corman this summer at the DevOps Enterprise Summit in San Francisco. He got up and talked about not just the opportunity that developers have today to create really cool and really powerful stuff, but he really got into the responsibility that developers and operations have now that we are introducing code into automated vehicles, pacemakers and more.
All of these very heavy, interactive pieces of software that we have today—the internet of things. He talked about the responsibility that we have to keep all of that stuff both secure and reliable and got into the whole Rugged DevOps concept. I mean there was just a hush over the whole crowd. It was such a powerful speech by him regarding this responsibility that everyone has and has to take very seriously.
Derek: Yeah. You know, with the amount of software that is going into anything today and the velocity at which we’re able to produce stuff, there is certainly a responsibility for how we produce things. I think we see that emanate in a lot of different ways. For those of us that are a little closer to what goes on in security aspects of things, we see these stories all the time about back in August, you saw the hackers that got into the Jeep Grand Cherokee that they stopped on the highway. Wired reported on this. Luckily, Chrysler fixed those vehicles or put out a recall on those vehicles right away.
We hear about different hackers being able to get into things like pacemakers, or hospital or medical systems that are consuming these … Just to think about the way that we are developing things, you can imagine like they did that proof of concept back on the Grand Cherokee and they said, “Look. We’re able to take control of this particular vehicle.” I am sure that those hackers, even though they were white hat hackers, figured out. There are, I don’t know, half a million, a million who knows, Jeep Grand Cherokee is on the road that had that software in them?
Imagine if they decided, “Hey. Let’s disable everyone of these right now.” The amount of impact that it has is dramatic. Where that software is being used repeatedly among you know different products that we rely on for our own health and security—it is extremely important that all of us takes a responsibility to understand what are we putting into these products that we’re offering to people.
Just as other industries have said, “We’re not going to put unsafe parts into cars or drugs or food services,” or what have you. It’s not just about, “Hey, we’re not going to put bad parts in,” but also understanding, “Hey, in the future, parts can go bad, right? Parts can become known defective.” Whether that’s cilantro that goes in your burrito that has something wrong with it that needs to be recalled, or a drug that has some ingredient that is know to cause some kind of defects, or your car that can be stopped on the freeway—those supply chains and those processes for the organizations, they have taken the responsibility to say, “Not only do I own what’s going into this, but I own the responsibility for fixing it if it goes wrong.” They track and trace everything that goes into their products.
I think in software, we have to take a similar approach to say we have to be responsible not only for what goes into these products but be responsible for … Not only responsible but capable of repairing those things if they go wrong. There’s a popular quote at Sonatype that says, “software ages more like milk than wine.” If you take that impression and say, “Yes, we have to be responsible for not only what goes in but making sure that it stays good over time.” I think that’s critical to certainly national security. It’s critical to our health. It’s critical to our safety. It’s critical to the success of the higher performing organizations in any industry.
To learn more about the ways innovative organizations are modernizing their development, testing, and IT operations workflows, check out the rest of our technical series!
