Who most often leads DevOps initiatives and transformations? Does the order normally come from executive leadership? Developers? IT Operations? While the answer varies from company to company, one department you don’t often hear about leading the charge is QA.
QA director at Audible, Tanya Kravtsov, joins us this week on The Skytap Podcast. We discuss her role in leading DevOps initiatives and how automation done right frees up time for innovation through greater collaboration between devs, testers, ops, and management.
Don’t forget that you can subscribe to our show through iTunes, or check out other episodes on SoundCloud. Thank you for joining us on The Skytap Podcast!
Noel: Who was responsible for initiating a DevOps transformation at Audible, or even the places you worked beforehand, and what was the original reception to that person or that department’s idea?
Tanya: Sure. One company I worked at—which was the first place where I helped build-out DevOps—was producing a data-integration-product and their test-cycle was taking very, very long. There were a small group of engineers, actually only 3 people, who really believed in process improvement. While most developers wanted to do development, there were a couple of people that wanted to do development that would make developers’ lives easier.
Noel: Right.
Tanya: I started working with them. I was working as a QA manager there, but I quickly realized that doing testing, without having things like build-automation, and environment automation in place was really going to impact us. I started working with those guys, and together we drove DevOps practices. We were able to get the upper-management buy-in when they started seeing the differences in the process.
One thing with DevOps is that there are some things you can do very quickly, and it will produce a major effect. Creating a small script that will collect test results from 30 different machines, which used to take hours for one person, and now it takes seconds. We were really able to get managers to invest into these DevOps practices.
Noel: In your session, you also brought up The Phoenix Project, and Skytap is a huge fan of that book. That book was issued as required reading for multiple Skytap departments.
Tanya: That’s good!
Noel: I got the chance to interview Gene Kim, and when I asked him for his definition of DevOps, he said, “DevOps is about the outcomes.” Because if you try to describe what it is, and how to do it, it’s not all done the same way everywhere, there are different approaches to it. I mean, it is about automating a lot of what we talked about, but it really is about the outcomes. You mentioned that the outcome for you is having code that’s ready to go to production as quickly as possible.
Tanya: I mentioned that as one of the possible outcomes, but I don’t think it’s the ultimate outcome. It’s the fact that quality code should go into production, and of course, if you have code ready, you tested it and it’s good to go, you want to put it into production as soon as possible. This is where this practice has come in. It allows developers to get feedback on whether their code is good or not, as soon as possible.
I’ve seen developers get really frustrated and even leave companies because they never saw the products of their work. They basically developed a feature, and then it went into this long testing-cycle, and then it went into UAT. So by the time the feature ends up in production, a developer’s no longer working on it. To really have developers who are satisfied with their work—and testers as well—you do want to see the end result.
Noel: Right.
Tanya: You want to see that product go into production. I think that’s one of the great things the DevOps allows you to do, is to put features through this whole delivery process as fast as possible, but while instilling all of the quality gates along the way.
Noel: Yeah, and it’s shorter releases, too. It’s not like you’re having to wait on a company that maybe did a 12 or 18-month release. It’s these quicker and quicker releases where they’re able to see that stuff a lot faster.
Tanya: Yes.
Noel: That’s great. You described how automation automates those things that you don’t want to do, or don’t need to do manually. What are some of the things that devs and testers can do with that extra time?
Tanya: Right, as far as testers, I don’t think anybody is really going to suffer from not having something to do. Because from what I’ve seen, at this point, is that testers are so bogged down with running regression testing, and writing scripts for new feature testing, is that they really never get through all of the tests that should be run. One thing that DevOps will allow is to actually go through all the tests that you should have done. I think that’s actually one of the biggest things we need to do, because if you’re not testing everything, then we can’t be 100% confident the product works.
Noel: Right.
Tanya: That might not go along with what I mentioned before, that sometimes you have to pick-and-choose, and you do. Unless you got to that point where you are able to run everything, you do have to be selective. Because you don’t want to run your thousands of tests, and have to stop in the middle, and have to execute the other portions.
It’s better that you choose the tests that are most critical to your release. But some of the things that you get time for is actually looking outside of your organization, and that was a part of my talk. You should take a step back, and look at the whole process. So, as soon as testers get the time, or developers get the time, it gives them an ability to start talking to people outside of their organization. This helps them see where they can help to improve processes. Testers can start working with developers to help them do unit testing better. Or, developers can actually sit down with testers and help them write a script for some of their manual process; things like that.
Noel: Yeah, and operations is getting more time, too. Because operations isn’t spending time building environments and fulfilling hardware provisioning demands and requests. They’re free to work with other lines of business as well.
Tanya: This is basically what allows people to become innovative. There are a lot of process improvements that can happen but don’t because everybody is stuck with their day-to-day tasks.
Noel: Right.
Tanya: Being able to automate those manual tasks will actually let people think outside of the box, in order to come up with some creative ways of doing things.
Noel: You also mentioned the dreaded, “It works on my machine” response from a developer that gets handed a bug, and how testers may be able to pinpoint where exactly that bug was found, and maybe how to resolve it faster so that the build is not delayed any longer than it has to be. What are some of the ways to give developers a lot more to work with?
Tanya: Sure, some of the things that I’ve seen work very well in the past, is automating the results analysis. As a result of your test, you get the report which you can actually parse through before you send it to developers. Being able to find old issues that failed on the same machine. Providing developers insight into where things failed, or maybe if any tests failed with the same error message, you can actually provide the analysis that all of those issues might be related.
Tanya: Also, another thing you can do is actually look at the runtime of the test. If something fails, don’t just log the failure—you can dig deeper. Go into the log file and capture what the log file says at that point. Your bug might not just say “user could not log in,” it would actually give the developer helpful information.
Tanya: Things like that can make root-cause-analysis a lot faster, and sometimes can actually help a developer fix it, instead of just spending time analyzing it.
Noel: Last question, I know that DevOps is similar to agile, in that you’re never just “at DevOps.” “You’re good, you don’t need to work any harder.” I was curious as to where you and your teams are currently at. Is it scaling your DevOps initiatives and finding even more things that you can automate? is it getting to production faster than you even currently do? Now that you’ve started it, and it’s in place, where are you hoping to take it from here?
Tanya: Currently, I told you about a lot of things which are already in place. Audible is an Amazon subsidiary, so they’re taking advantage of a lot of tools that Amazon is using. There’s definitely continuous delivery already happening. However, as I mentioned, we’re building this new QA org to make sure that product quality is even higher.
The idea is to have a team that’s going to represent and be the voice of the customer. Making sure that every feature that goes into production has the proper user experience. As I mentioned, being able to do DevOps, but also instill those quality gates in it, this is our primary goal right now. Continuous delivery might already be there, but now we’re actually adding quality into that.
Noel: Right.
Tanya: The name I chose for the meetup I host is “DevOpsQA.” Because, while there’s DevOps, there may not be QA in it. And, as I did at the previous companies where I worked, I’m making sure we put QA into DevOps.
We hope you enjoyed this week’s episode; click here to access all of our previous episodes of The Skytap Podcast right here from our blog!
