We sat down with Paul Farrall, Skytap’s new VP of Operations, and the president-elect of SIM Seattle, to discuss his successes and failures within his own DevOps initiatives. His experience helps prove the importance of trust between teams. SIM Seattle will host the 2015 Seattle Technology Leadership Summit on May 20th.
Noel: Hi Paul! Let’s dive right in. Where were you before Skytap, and what’s something from your background that you’re exceptionally proud of?
Paul: I was born and raised in Delaware, and I originally came to Seattle to work on my PhD in chemistry/physics at the University of Washington, but jumped ship to work in the startup technology world.
For the past 15 years I have been focused on web operations at a variety of organizations from large companies like AT&T to startups like Getthere.com, and Intelius. Most recently, I was VP of Operations for Big Fish Games here in Seattle where I had overall responsibility for web operations, infrastructure, Corp IT, information security, and release engineering teams.
As for what I’m really proud of, I could list a bunch of technical accomplishments or projects I’ve delivered over the years, but my greatest satisfaction and pride has come from building and leading world class Ops teams. I derive great satisfaction from hacking code, but the deeper satisfaction that comes from rallying a team around a common goal is on a whole other level.
Last week Paul participated participated on a Seattle SIM panel discussion titled, “Cloud Smackdown 2015- Where should you put your infrastructure.” Video from Laura Naumann on Vimeo.
Noel: That’s very cool! In looking at your background, I noticed that while you were at Big Fish Games, you led the implementation of the “Big Fish DevOps program.” I’d love to know more about that. When the implementation began, what went really well, what challenges were the hardest to overcome, and what was one or more learning points or best practices that you would recommend to others looking to implement their own DevOps initiative?
Paul: Implementing DevOps at Big Fish was intense but fun. The hardest challenges to overcome were cultural. In fact, my first attempts to introduce DevOps at Big Fish failed miserably due to cultural misunderstandings and lack of trust between development and operations organizations.
Here is a simple (though embarrassing) story to illustrate the point. At Big Fish, application development teams equated DevOps with developers carrying pagers, an idea they were highly resistant to. Every time I brought up a particular DevOps methodology for discussion, they thought I was engaged in a secret plot to make them carry pagers. I never had any intention of asking developers to carry pagers.
The term “DevOps” covers a large range of continuous improvement techniques. Some organizations have developers carry pagers and it works for them, but this is not a prerequisite for DevOps. I had no idea they were thinking this because they were afraid to bring it up for discussion. So we failed because the trust level between Dev and Ops teams was too low for any DevOps methods to succeed. We couldn’t even tell each other what we were thinking.
This is a really important point and the foundation for DevOps. Your DevOps initiatives will fail without an adequate level of trust and communication between development and operations organizations.
Noel: I couldn’t agree more. I’m assuming the trust improved over time?
Paul: Our next attempts at DevOps were more successful. Three things had changed: (1) Some time had passed and we had improved trust and collaboration between dev and ops. (2) I somewhat sneakily stopped using the term DevOps and instead just talked about the specific continuous-improvement methodologies we wanted to implement. (3) When deciding which methods to implement, I specifically chose ones that would benefit development teams (this helped build trust).
Noel: That’s interesting about having to be “sneaky” about not calling it DevOps anymore. I think we’re seeing that more and more, purely based on the variety of ways people interpret the term. What else worked and helped the program succeed?
Paul: One very simple DevOps technique we had a lot of success with was incorporating operations and InfoSec input at the very beginning of application design. This was a little unusual for development teams at first, but they quickly saw how obtaining input from operations and InfoSec before a product was built resulted in better quality and less headaches for everyone.
A larger project we undertook was building a Continuous Delivery framework that allowed game development teams to publish their code directly to production without direct involvement from operations. This was a big win for everyone. Game development teams were happy because they could push production code as fast as they desired without operations being a bottleneck. The operations team was happy because they were no longer in the spotlight as a bottleneck and designing and building release tools was more satisfying work than manually pushing code to production. Business stakeholders were happy because game features were getting to production faster.
It felt a little weird at first letting developers push code straight to production, but the operations team built and maintained the release tools so all necessary controls, safeguards, audit trails, etc. were built into these tools and we quickly got used to having developers push the release button.
Extending on this tool building and self-service theme, the operations team later built a private cloud to allow people to spin up their own infrastructure, but in the interest of space I’ll save that story for another day.
Noel: I would love to hear that! So, what will your new role and responsibilities as Skytap’s new VP of Operations include?
I have overall responsibility for production operations at Skytap’s five (soon to be six!) production datacenters. I also have responsibility for internal corporate IT services at Skytap.
Noel: We’ve added so many awesome, talented, and industry-passionate people early this year; how exciting of a time is it to join a rapidly accelerating cloud company in a city as awesome as Seattle, and what would you say attracted you most to Skytap?
Paul: Yes, it is definitely an exciting time to join Skytap. I am very impressed with the caliber of people I have met; we have some really smart people working here.
Two things first attracted me to Skytap: (1) Environments-as-a-Service. Fast, reliable provisioning of test environments is so frequently the rate-limiting bottleneck for code flowing into production that it is encoded into the core principles of DevOps. Skytap delivers a cloud EaaS product to address this problem. Cloud + DevOps—How much more exciting can you get than that? (2) The positive get-stuff-done culture. This was noticeable from my very first conversations with Skytap employees. Before I decided to come work at Skytap, I asked around and found that everyone who was familiar with Skytap had nothing but good things to say about the company and the people who worked there.
Noel: It is absolutely a great time to be here! What are you looking forward to helping Skytap accomplish in 2015 and beyond?
Paul: As Skytap continues its rapid growth trajectory, I will be focused on two primary objectives: (1) Growing capacity in our five, soon to be six, datacenters to meet customer demand. (2) Continuing to improve upon our operational excellence to ensure that we keep delivering a great customer experience as the company continues to grow.
Paul Farrall is the president-elect for the Seattle chapter of SIM, and on May 20th, SIM Seattle will host the 2015 Seattle Technology Leadership Summit, where Skytap’s CTO and VP of Engineering, Brad Schick, will present his breakout session titled, “The Evolution of Complex Enterprise Applications.”
