The rise of the popularity and functionality of mobile apps and wearables has created tremendous growth, but for one industry in particular, there have been an equal amount of disruption and challenges along the way. The hospitality industry provides memorable and rewarding experiences for customers through the use of mobile apps and wearables, but a dependence on integrating with core systems of record, legacy systems of various vintage, and third-party apps make delivering these experience a monumental challenge.
On this week’s episode of The Skytap Podcast, we chatted with QA evangelist Dave Higgins about how Hotels.com, is working toward continuous delivery across the company’s 40-plus globally distributed development and test teams. Dave gave some fantastic insight into the organization of Hotels.com’s software delivery teams, and how automation is helping them thrive in a highly competitive space.
For those who cannot listen right now, a transcription is provided below, and don’t forget, you can also access, download, and subscribe to The Skytap Podcast on iTunes for your on-demand listening enjoyment!
Noel: Let’s have you talk about Hotels.com a little bit, and how you’re set up there. I know you said that you were really proud of the teams there; I’d love to know more about the way they’re organized.
Dave: Well, not so much a challenge as such, but it’s the sheer scale of what we do. We’ve got about 40 development teams. Half of them are based in London. All of the project management tech leadership is in London as well. We outsource almost half of our work to our teams in Hungary, and we work with EPAM Systems as an outsource partner. There are teams in Budapest, in Szeged, and Debrecen in Hungary. Then we’ve got small teams in Rome as well. We’re spread across several sites. About 40 teams all developing their own areas of the site, which then comes together into an amalgamated release. We have release testing, which tests the end-to-end, then it gets pushed out into production.
One of the big challenges we’re facing this year, by the end of this year, we’ve set ourselves a target to have all of our teams doing continuous delivery. 40-plus teams all delivering to one final application whenever they feel like it.
Noel: That’s a big goal.
Dave: It’s going to be a laugh a minute! It’s going to be interesting, but we’re on track. The pipeline is set up and established. There are teams using it already. It’s just the case of getting adoption sorted out and bringing everyone up to speed, getting everyone doing the same thing. We’ve got a good DevOps structure in place as well, which help things along. The most important thing that we’ve got is we have a very flat structure. Anyone can go and talk to anyone about anything. In that respect, we’ve got a very good culture.
For instance, I’d have no problem going to our CTO and saying, “This isn’t quite right. Perhaps we can change things this way.” He’d have no problem with me doing that. He’d take it onboard and then filter it down. I wouldn’t go that high immediately of course, but I have no problem going to anyone and asking anything, and that’s the same across the board.
Noel: That’s great. Going to continuous delivery, with that being such a big goal, I once saw a session here at one of the STAR conferences where a guy, I think he was from Capital One, said that he always advised people to try to get to continuous delivery. Because even if you just get close, that’s better than being far from it.
Dave: Absolutely.
Noel: To get there is fantastic, but even getting close can still have huge rewards.
Dave: Absolutely. I think the plan is that once we’ve got everyone doing continuous delivery, and everything is ready, the plan is to be able to go from concept to prototype in an hour. That’s our over-reaching goal. We’ve got a very good automation architect on-board, Sven Van Laar (Hi Sven!) who has just come in and revolutionized what we’re doing with automation. He essentially came in and tore up what we had as far as automation infrastructure and he started working with Selenium WebDriver and Java and has really produced a footprint on the whole thing and has sped everything up immensely. He’s the one who came up with that lofty goal of getting everything in an hour.
Noel: You talked about having a good DevOps structure in place, and what you’re doing with automation. I love when people describe automation as being great for automating the things you don’t want to do so that you can do the things that you do want to do.
Dave: Yes. That’s the real key of automation—using it properly. Automation should be about doing routine and mundane stuff, the repetitive stuff. If you’ve got manual testers who are just working from a checklist top to bottom, that’s a complete waste of manual testing. Automation should be covering that. That then frees up the manual testers, the ones who should have the real creative minds. That frees them up to then go and do exploratory testing and to really dig into the corners of the application and search out the stuff that automation won’t. The two work hand-in-hand. Automation is never going to replace manual testing.
Noel: Exactly. You talked about how with your team being as large as it is and growing even bigger that that can create challenges for your Ops teams.
Dave: Yes.
Noel: I imagine if you have that many teams all putting in new code that frequently. I assume that’s one of the challenges for them, but what are some of the others?
Dave: Well, the big problem our Ops team had until recently was that we’ve had an awful lot of growth in the last five years. These 40 teams that we now have—five years ago they didn’t exist. We had a couple of development teams in London, some teams in Hungary. The majority of Dev and test work was done in Hungary. The London team was only established in the last five years.
The big challenge we had is that we’ve grown so rapidly in such a short space of time, but our operations teams didn’t. We still have the same number of people and the same resources now trying to keep up with 40 teams who are releasing stuff. It’s only now that we’ve started investing in operations and getting new folks on board and just really expanding that area so that we can properly support the pipeline delivery and that sort of thing.
Noel: Looking at the hospitality industry in general, for Hotels.com, I assume there are a number of third parties that Hotels.com integrates with. What are some of the challenges of maintaining a release schedule that works with all those integrations and their updates and releases, and giving everyone a system that integrates smoothly with everything else?
Dave: I think probably a good example of that would be recently for a lot of our SEM and big data stuff, we rely on Hadoop. I don’t know how widespread it was, but Hadoop recently had a long outage, which essentially left our teams stuck for a couple of sprints with not being able to do anything. Obviously, we’ve got that dependency. We do have service agreements, and we do still chase up and try to get things done as quickly as possible, but there are times when it’s just not possible.
We have teams, not necessarily sitting idle because there’s always something to do, but it does mean we’re not getting things over the finish line. That’s a challenge and it can be demotivating as well, so it’s a case of keeping people focused and saying, “Look we can still develop. There are still things that we can do, so let’s focus on those and then we will do our best to catch up and get back on track once everything is back up and running.”
Noel: I know with Skytap, we’re seeing a lot of these traditional or legacy apps as giant, monolithic—whatever you want to call them. People are now figuring out ways to modernize those and innovate on top of those faster than they could in the past. I would imagine that for creating, let’s say the Hotels.com mobile app, by being able to have those legacy systems upgraded or “modernized” faster, that helps you modernize apps on your end as well.
Dave: Yes, it does. Up until about three years ago, Hotels.com was a monolith. It was the Hermes Web App, it was called. The whole thing was a monolith. It was around that time we then decided to split the applications, and that’s where the development team started splitting up. It broke down into individual apps for shopping and search, and booking and landing pages. Whatever it is, it’s been broken down.
There are still a few bits that are integrated and it’s very hard to decouple. But, breaking things down, it makes it a lot easier to do Agile, so everything becomes more iterative and smaller releases by smaller teams. It just made the whole thing go a lot faster.
Noel: You guys are using agile in what ways?
Dave: The way we use agile is… it’s different for every team. We’re not prescriptive in that we say, “You will use agile and you will use it in this way.”
Noel: Right.
Dave: Every team works in their own way. They’ve got their own priorities, their own cadence that they work at. It’s probably wrong to say that none of our teams use agile in the same way, but everyone’s got their own little twist on it, so some people use Scrum, some people use Kanban, some people use hybrids. Really, it’s horses for courses. It’s having the trust in the teams that they know what’s best for what they’re doing.
The one thing we’re very big on is making sure that when we hire people, we’re getting the right people in the first place. When we do that, we then have that level of trust in that they’ve got the knowledge and the experience in what they’re doing to be able to say, “Actually I think we should do things just this way and it will make things a lot better.” Management isn’t prescriptive in saying, “But you must do it this way.” We hire people who know what they’re doing, so we let them do it.
Noel: That’s awesome. What makes them the right people? I would assume another thing is understanding the “DevOps mindset” and this new view of accountability—everyone being responsible for speed and quality.
Dave: Yes, absolutely. We’re very big on that. I’ve never known a company that has such a rigorous interview process. We had a new development manager come in recently and I know that he went through a phone screen with HR, a phone screen with me, because I, for some reason, got to interview a Dev manager. He did a series of face-to-face interviews with about four different people. I think he might have done a second lot with directors as well. I think by the time he was done, he’d been interviewed by about seven people. There has to be a consensus.
Noel: Right.
Dave: In that respect, we’re strict in our interview criteria.
Noel: I also wanted to ask you, talking about innovation and experimentation, with booking a lot of hotels and flights, these mobile sites are able to do more and more things. Integrating with loyalty programs, choosing your own seat, unlocking your own hotel room door—all of these kinds of things that can now be done. What is some of the fun, innovative stuff that people just didn’t have a couple years ago that you all are working on now?
Dave: Certainly the native mobile apps have come on an absolute bundle since we first launched them. All sorts of cool stuff like Passbook integration, so that you can get a copy of your reservations sent straight to Passbook, and to Apple Watch as well.
Once the technology is there and once wireless technology gets into hotels, the idea is to be able to open your hotel room door with your watch, but we’re still somewhere away from that just because the hotels themselves don’t necessary have the technology to do it.
It’s something that’s in the back of our mind and something we want to be able to offer as something seamless, where you can check-in from your watch based on your location. Then go to your room and beep the watch, fire NFC, and have the whole thing being seamless. It’s in the back of our minds, but we’re slightly ahead of the technology on that one.
Noel: Nice! Do you see features like these coming from customer expectations and requests, or are hospitality companies largely providing things to customers that they didn’t even know were possible?
Dave: To some degree we work on the feedback from customers, but a lot of the time we have analytics teams and marketing teams and that sort of thing who will then come to us and say, “Wouldn’t it be cool if we did this?” Even as technology teams, we do the same thing. We think, “You know what? It would be so brilliant to do this. No one else is doing this. This will be fantastic.” Innovation comes from everywhere.
Noel: Right.
Dave: Again, with the flat structure, anyone can bring it.
Noel: That’s right. My last question would be, with you being responsible for QA, what’s a current quality goal or an ongoing quality goal of yours? Is it decreasing the number of bugs in production? Is it just overall quality? What’s a benchmark that you’re being held to?
Dave: A benchmark we’re being held to? We always have a bug limit, which I think most folks do. Although it’s counted as one of our over-reaching goals, it’s not really used as an assessment metric, I suppose. We don’t assess people on it. We do give people a nudge if they go over their individual team limit just for the sake of keeping the quality high. As for metrics that we use, we do have an over-reaching goal. We changed it slightly from last year, so last year we had a goal in our overall tech goals to automate 80% of our testing.
It’s a noble ambition, but we were slightly off base because we’ve got teams who just can’t do that amount of automation. The nature of their work means that they do a lot of manual stuff, so it won’t be possible for them to automate 80% of their testing. We’ve changed it slightly this year so that the goal is every team must automate 80% of the tests that they are able to automate, which is a much fairer goal.
Noel: That’s true. I can see where there would be cases where that number worked for some teams, but not others. Again, as long as you’re automating the things that decrease business value by doing them manually, you’re on the right path.
Dave: Absolutely. As I said, too, automation is key to freeing up the creative minds of the testers. They’re the ones who really know the system under tests better than anyone, so they’re the ones who can really do deep testing and get stuck into things.
