Ticketmaster Presence wins “Best in Sports Technology” at Sports Business Awards

Exciting news. Ticketmaster Presence was just awarded “Best in Sports Technology” at Sports Business Journal’s 11th Annual Sports Business Awards!

Ticketmaster Presence is the next generation venue access control and fan engagement platform that replaces paper tickets with digital passes. Presence streamlines live event venue operations, provides real-time insights and analytics to venues and teams, enhances security, protects against fraud, and allows for a personalized fan experience. Ticketmaster is also developing and exploring new features for Presence, including facial recognition technology through Blink Identity.

Presence is also a cornerstone of Ticketmaster’s official ticketing partnership with the National Football League. And since hitting the market just over a year ago, more than 100 venues have implemented Presence, and more than 13 million fans have entered venues using the technology.

We’re honored to be named the “Best in Sports Technology” and know that Ticketmaster Presence and digital tickets will continue to change the way fans experience live events. And it’s all happening in the palm of your hand with your mobile device.

It’s the kind of technology that venues, teams, artists and fans have been waiting for.

Ticketmaster Launches an Action on Google Assistant


At Google I/O 2017, Google announced that third-party Actions with transactional capabilities would become available via Google Assistant on both Android and iOS devices.

The day has arrived.

What is Google Assistant?

Google Assistant is a virtual personal assistant that can engage users in a two-way conversation to help them get things done. An Action on Google is a way for developers to bring their products and services to Google Assistant.

Google Assistant originally debuted on the Google Home device, but is now available on Android devices that are version 6.0 (Marshmallow) and higher, iOS devices with the Google Assistant app installed, Android TV, and Android Wear. Furthermore, with the recent release of the Google Assistant SDK and the growth of the Internet of Things, the Assistant is also available to many more connected devices.

On Android devices, Google Assistant can be invoked by holding the home button or saying “Ok, Google.” Then, without installing anything on the device, a user can access your Action by saying “Talk to your_action_name.

“Talk to Ticketmaster”

The accuracy of speech recognition and the efficiency of a voice search intrigued us. What if we could help fans find events with simple phrases?

“Find a jazz show in New York.”

“What is happening this weekend?”

“Where is Jay-Z touring?”

Early this year, we spoke to Google about our vision and had the privilege of being invited to an Early Access Program to develop an Action on Google with transactional and multimodal capabilities. We were so excited about the invitation that we began development immediately. Not only could we help fans find events via speech, but we could also present an interface to help them visualize their options and provide a mechanism for quick, seamless purchases.

The recent release of Ticketmaster’s Action on Google Assistant marks the result of our partnership. Now a fan can invoke the Ticketmaster Action by saying “Talk to Ticketmaster” and be led through a clear, crisp experience to find an event and purchase tickets.

How does it work?


Google has integrated API.AI (an intuitive platform that enables the creation of natural, rich conversational experiences) into the Actions on Google development process. API.AI takes the user’s speech or text and breaks it down into the Intents that the Action developer defines. An Intent is simply a way to express what action should be taken based on what the user said. The Intent that API.AI creates can either be handled in the API.AI agent itself or sent to your webhook (via JSON) for fulfillment. The webhook can determine how to handle the Intent from there. For our specific case, we access the Ticketmaster Open Platform APIs to address the fan’s request. Once the webhook determines how to handle the Intent, the speech, text, or graphics data is sent back to API.AI, which formats the response and presents it to the user.

What are multimodal responses?

The multimodal feature provides an audiovisual component to your Action. The recently-released capability allows you to respond to your users using images, cards, lists, carousels, and suggestion chips. This flexibility means you choose the best way to present a user with options. Below are some examples of these UI elements. Lists, carousels, and suggestion chips are all selectable via tapping the element or speaking or typing the text that is on the element.

Example: multimodal elements available for Actions. The user is presented with a tappable list of options (left) or a horizontally scrolling carousel (right).

What is the Transactions API?

The Transactions API lets you accept purchases and reservations with your Action. The API provides a structure to help the user quickly build a shopping cart, supply a delivery address (if applicable), establish the payment method, check out, and confirm a purchase. For our specific case, we have chosen to email the fan their tickets.

Example: Checkout (left) and confirmation (right) steps in the Transactions API.

The integration of different payment methods is perhaps the greatest feature of the Transactions API. Google has provided a mechanism to either link your Action with an existing Google account or to link your existing web application service via OAuth 2.0. Using this mechanism, vendors can either use previously-stored payment methods or quickly integrate other widely-accepted payment processors such as Stripe, Braintree, or Vantiv.


Bonus track: Actions simplified for other developers

While Google provides a Node.js SDK to facilitate creating the multimodal and transactional experience, we wanted to create our Action using Kotlin. Since the Kotlin language permits top-level functions (similar to JavaScript) and has robust testing library support (Spek), using Kotlin (over Java) meant we could more easily port Google’s Node.js SDK. Furthermore, Kotlin compiles down to Java byte code so any other Java developers could use it as well. Our own Senior Android Engineer, Patrick Jackson, has taken the JSON spec and created an open-source Kotlin SDK. We encourage other developers to use and contribute to it freely.

The future of ticket purchasing

We are excited to release the Ticketmaster Action on Google Assistant. With the increasing accuracy of natural language processing tools, we believe virtual assistants are the future of human-computer interaction. This radical innovation is another step in the long history of live event ticket purchases: phone calls to the box office, internet searches on the home computer, mobile app purchases, and now, quick, natural conversations with a personal assistant. My favorite group, Straight No Chaser, comes to town every November, and I can’t wait to buy my tickets using Ticketmaster’s Action on Google Assistant!

Ticketmaster Presence: When tickets go digital, event going gets better

Live entertainment has enamored fans for thousands of years. Iconic events, that captured the imagination, shaped cultures, defined eras and won hearts. But throughout all this time, one thing has remained the same.

The ticket.

But that’s now a thing of the past, because with new tools and approaches, the future of ticketing is here today.

At Ticketmaster, we’re always asking ourselves ‘how can we make live events better for everyone involved?’ Based on decades of experience, we saw a way to improve ticketing as a whole by replacing the paper ticket with digital technology. That understanding drove the creation of our newest product, Ticketmaster Presence, the next generation venue access control and fan engagement platform.

Presence replaces traditional paper tickets with digital passes. It’s built on Ticketmaster’s software platform and uses proximity-based digital technology to enable an easy to use ‘tap and go’ venue entry system. Venues install Presence, including mobile hardware and revamped software for real time map-based attendance reporting, event set-up, and enhanced ticket management.

Fans simply use their smart device, like a phone or a watch, or a wallet-sized smart ticket with a built in unique ID chip, to enter a venue. Ticket management can be embedded directly into a venue, team, or artist’s mobile app, so fans can easily view, transfer – using text, email, Facebook Messenger, and more – and post tickets for sale directly from an app.

The result is a streamlined, safer, and more personalized and enjoyable live event experience for all.

But how can a digital ticket possibly do all that?

What’s unique about Presence is that it replaces something outdated and inherently anonymous – a laser reader and a paper ticket – with cutting-edge technology that can be personalized throughout the entire lifecycle of a ticket.

That means things like entry information, in-venue updates, upgrade offers, and any event changes can be communicated directly to the fan attending the show. In addition to providing a more engaging experience for fans, venues can increase security and decrease fraud by better understanding both who bought the ticket and who is in the seat at an event.

Our partners at Orlando City Stadium nearly quadrupled the number of identified new fans per game and have seen their instances of ticketing fraud go from 120+ cases per game to zero after implementing Presence into their stadium.

To date, over two million fans have entered venues using Ticketmaster Presence. It’s currently live in 33 venues across North America, and will roll out to some of the most marquee concert and sports venues in our ecosystem by the end of the year.

With live event fans snapping up 15 tickets every second on Ticketmaster, we know the popularity of live events continues to grow. That growth requires sophisticated technology to meet the increasing demands of both fans and the entire live event industry. I’m proud to say that Ticketmaster has successfully reimagined and rebuilt ticketing from the digital ground up, making that incredible live event that only happens once, better than ever.

Ticketmaster Demonstrates Cutting Edge Android Instant Apps Technology at Google I/O


It was a typical pleasant bright April day in Hollywood, CA but something  was in the air. Working with the very latest Instant App tools provided by Google, I was delighted to see the following message flash across my dark console, a first of its kind success for our Android development team:

“Successfully pushed instant apps to instantDev!”

Only a few of us who had worked on the Android Instant Apps project were aware that we had just pushed to the developer track, the first step in getting our Instant Apps to the Google Play Store. And only a few weeks later we had a working version ready to be whitelisted for production. 

Working with Google as part of the Instant Apps Early Access Program(EAP) was an incredible experience for our team. It’s always challenging, and sometimes even a bit scary to be at the very cutting edge of technology. At Ticketmaster we have always believed that “game attracts game”, so it was with great pleasure that our team took on this challenging opportunity.  Starting from a boot camp at Mountain View, through to our demo at Google I/O, it was certainly not as “Instant” as the name may imply.

If you work at a large organization, re-architecting your existing application to support Instant Apps, like any new technology, takes a bit of imagination and perseverance. We had to lean on other engineering teams like Google and Branch to help get to our desired experience. In the end it was well worth it. We are happy to have stayed true to our initial vision of using our Native OpenGL seatmap to create a truly unique Instant Apps experience that will provide new ways to attract fans with a better mobile experience. By pushing our vision to Google, we have helped to clear the roadblock for other engineers eager to use Native Libraries (NDK) such as gaming and camera applications in Instant Apps, and made a small milestone in what we believe will be a bright and exciting future for the Android platform.  

To help our fellow native Android developers, we are Giving Away Our Top 5 Secrets to Getting Android Instant Apps Work Started at Your Organization

Tackle Technical Debt.

        Find a way to tackle technical debt as part of your Instant Apps refactor. Tech debt has been a large part of my work at Ticketmaster, which I expanded on as a metaphor in a blog post back in 2015.  In our case, our Instant Apps architecture was able to speed our migration to the Model View Presenter (MVP) architecture pattern while also helping to move away from a deprecated networking stack, and to push forward in development against the new Ticketmaster Open Platform API’s. Developers don’t want to do throw-away work, so building something that will stand the test of time is a goal everyone can get behind, as well as providing new areas of growth.


Be Excited to Show and Tell

Enthusiasm is contagious, and Instant Apps make for an incredible demo.  Get comfortable demoing already available Instant Apps functionality in meetings and in individual interactions. Find the best way to show how quickly you can take a user from a deep link to purchase completion, and what a difference it will make to reduce friction. Not everyone may understand why this is so gaming-changing, it’s true that links on the web work instantly already… which leads us to…

Have Pride in the Native Android Platform

        Instant Apps show the full benefit of the native Android platform. It helps to highlight the latest conveniences like SmartLock and Android Pay.  Become an advocate for the Android platform by highlighting key advantages over your mobile website like material design effects and reduced purchase friction.

Links and tracking are the key

        Initially we had a hard time understanding what data we needed to contain within our Instant Apps links, and how best to configure them, both in the Ticketmaster instant app and our full app. Teaming with partners like Branch made it possible for us to include extra information in the links that we would have had trouble supporting, including adding support for non-Android platforms. We used Fabric for easy analytics tracking in real time. It’s important to work out the details of your instant app links as soon as possible so that you can focus engineering resources on building out the key functionality in Instant Apps.

Aim for a Best in Class Fan Experience

        Don’t be afraid of aiming high. Native platform advancements have the ability to blow conventional websites out of the water. Although initially Google was not able to support our native seat map, they eventually were compelled to support our use case in their initial public SDK since we were able to work with them.  It’s always a good sign when your company is “pushing” Google, one of the most advanced software companies in the world.  We will continue working with the Ticketmaster Product team to make this into a great fan experience now that we have resolved the engineering challenges to support our seat map technology. 

Last but not least, we would like to thank all the people at Ticketmaster, Google and Branch who helped us develop our Instant Apps Demo for Google I/O.  Kanak Siwach and Aadithya Bangalore Keshavamurthy were both especially helpful in pushing development of our Instant Apps in their 10% time. Henry Ciruelas(@hdigga) is a great leader and manager and our entire QA team are the best in LA tech.  We are looking forward to working with Google and the rest of our team to improve and advance this exciting technology.  We know our fans deserve the best-in-class mobile purchase experience and believe that Instant Apps will help us rock the tech world!

To learn more about Instant Apps at Ticketmaster, please reach out to me on Twitter @JeffKelsey

Tech Maturity is now available on GitHub

We’re excited to announce that the Tech Maturity model we use internally is now available on GitHub!

At Ticketmaster, we use Tech Maturity to identify growth opportunities, eliminate waste, set clearly defined targets, and measure progress all while we work toward the ultimate goal of continuous delivery.

The model charts a clear path that can be completed in stages and allows flexibility for progressing through five key dimensions of software development: Code, Build & Test, Release, Operate, and Optimize.

By using Tech Maturity, you can:

  • Quantify the maturity of your products
  • Visually compare individual product maturity to company-wide trends
  • Identify patterns for removing friction within your SDLC
  • Track and measure improvements over time

Our model does not prescribe solutions. Rather, it offers standards with an aim to give teams a clear path towards efficient product development at scale.

To learn more about why we created Tech Maturity, read this blog post.

Checkout the project on GitHub to get started or watch this brief video tour:

Get Involved

Tech Maturity is a part-time project for a few of us at Ticketmaster. If you’d like to help expand upon what we started, we couldn’t be more excited about that and would love to hear from you!

Sherry Taylor | @seakitteh
Vignesh Jayabalan | @vigneshjybln

Follow @tmTech for other updates on what we’re up to! 🚀

Interview with our Test Manager of the Year

Isabelle Magnusson leads the Quality Assurance team in Ticketmaster’s Gothenburg office in Sweden, and recently won Test Manager of the Year at the 2016 European Software Testing Awards. We caught up with her to find out a bit more about her success:isabelle

  1.       How does winning this award make you feel?

I feel very proud and a bit surprised to be honest.

  1.       What is it that you love about being a test manager at Ticketmaster?

I’ve been given a lot of freedom to do what I believe in from the start. My team is always striving for improvement, but will still keep challenging new processes. I have a team of testers with very different backgrounds that makes my role as manager very attractive.

  1.       How long have you been part of the Ticketmaster family?

5 years

  1.       How has Quality Assurance and software testing in general changed from when you first started?

I started my career as a trainee at a consultant company where I practiced all kind of test techniques and processes in a variety of organisations. Back then the most important part of my CV would be the ISTQB certificate. Today certificates like that are not very attractive.

  1.       What does the future of QA and software testing look like from your point of view?

I don’t believe in QA “departments”. The tester is a part of the development team and the quality assurance is not not just an activity for the tester, but rather a whole team approach. The tester in the team is still the test expert and can coach and drive the test strategy though. The opinions regarding manual and automated testing differs a bit. I believe we will not be able to automate everything but the role of the manual tester will not just be about following step by step test cases. Instead it will focus on validation from the start of a project.  

  1.       What does a day in the life of Isabelle Magnusson look like?

I scroll through Slack and mails from bed. The benefits of working with several time-zones, someone is always working, right?  After leaving my daughter, Juni at preschool, I take the bus to the office where I have my breakfast while catching up with the team. I spend quite some time coordinating bugs reported from the markets and aligning with product managers and project managers. I’m coaching the testers in their job in the development process, representing QA in different contexts. I still like to be a part of the development and step in where we are low in resources when it’s needed. We are currently recruiting a whole new team so I also spend a lot of time in interviews and reviewing CVs. After office hours I try to spend as much  time as possible with my family.

  1.       Please describe what you feel is your biggest career achievement to date:

Being a part of moving the agile process at Ticketmaster forward have been a great journey. I’m really proud of how far we as a team have come.

  1.       Please describe the vibe in the Ticketmaster Gothenburg office:

It’s a very friendly atmosphere; I consider my colleagues my extended family. We have a lot of fun and we work hard to achieve what we have committed to.

  1.       What’s it like to work at Ticketmaster?

It’s a great place to work. Being a global company the opportunities are endless.

  1.   What do you love about working at Ticketmaster?

It is a large organisation but I still feel that each individual counts. Ticketmaster invests a lot in people. I love having the freedom to do what I think is best and the responsibility that comes with it.

  1.   You recently participated in the Ticketmaster International Hackathon. How was the experience?

It was a lot of fun. I was collaborated with colleagues from Sweden and Canada. It was a great opportunity to innovate with new features and methods.

  1.   What was the first live event you went to?

Tracy Chapman

  1.   If you could go to any live event, which one would it be?

I love Christmas. I’ve been listening to Christmas songs since early November and my favourite artist is currently “She and him”. I would love to see a small, intimate live event where they perform songs from their Christmas albums.

  1.   What’s your passion?

Except testing? I’m a foodie. I love to find new restaurants and have their tasting menu. I plan my vacations based on what restaurant I want to try out. Unfortunately I’m not a very frequent customer since I had kids, so currently I have to settle with experimenting in my own kitchen. I’m planning for my next stop to be Fävikin Magasinet in Åre though.

  1.   What’s one thing that no one at work knows about you?

Hmm, I think they know most of me, I’m like an open book.

Getting to Innovation Faster

📌 UPDATE: Tech Maturity is now available on GitHub

Original post below:

Dealing with growing pains is a good problem to have! Over the years, Ticketmaster has evolved from a closed ticketing website to a technology company with an open platform powering products offered by major brands like Facebook and Costco. It has been an enormous challenge to remain agile while scaling, and we are excited to share how we have met the demands of fast growth.

Soon, we will be releasing an open source portal that measures “Tech Maturity” so companies of any size can also realize the benefits, e.g.:

  • Spend more time innovating, and less time being “busy” with repetitive, manual tasks
  • Use a common set of standards to drive business results
  • Increase visibility for better decision making
  • Gravitate towards consistency across teams and products
  • Use standard components over custom-built solutions
  • Invest in fixing root causes and not just the symptoms

View / Download Tech Maturity Model

With the goal to drive innovation, we aim to free ourselves from repetitive busy work by removing friction so that we can focus more on what we really want to do: build awesome products. I’m sure that most developers, whether working at a small startup or large enterprise, can relate to the lightning speed of product development and the pressures that come with it. At Ticketmaster we have hundreds of products in our portfolio, and at times our SDLC has had developers chasing their tails to meet product and quality specifications by a deadline. There is rarely a moment to reflect and clean up shop before the next project comes along.

To improve efficiencies across the board, we took a giant step back to see how to remove friction and help teams work smarter. We found that the existing maturity models didn’t address the pain points felt by most of our teams on a tangible, practical level, so we created our own! We started with a punch list of barriers preventing teams from achieving continuous delivery and that evolved into the model we use today.

Our Tech Maturity model values automation, repeatability, and visibility over manual, repetitive tasks.  It minimizes the risk of paying “tax” later in the form of fighting fires, longer release cycles, and limited visibility into the health and performance of a product. After applying the model to just a few products, a pattern quickly emerged highlighting a trend in low maturity in repeatable deploys, test suite coverage, and monitoring & alerting. We began quantifying the results into a “product quality score” and correlated the data with outages. This resonated with our product leadership, and teams found it much easier to prioritize non-feature work.

Example of how we visualize product quality at the application level

How the Tech Maturity Model Drives Competitive Growth and Continuous Delivery

Tech Maturity helps us identify growth opportunities to eliminate waste, set clearly defined targets, and measure progress all while we work toward the ultimate goal of continuous delivery.

The model charts a clear path that can be completed in stages and allows flexibility for progressing through five key dimensions of software development: Code, Build & Test, Release, Operate, and Optimize.

For Ticketmaster’s move to the public cloud, the model served as a “cloud readiness” gauge that quantifies how close a product is to being ready for migration. This was achieved by establishing targets for a subset of capabilities that we believe define the minimum requirements for any product or service in the public cloud. This gives teams who own legacy products a clear goal to work toward that they can easily track. It also allows the company to operate in a decentralized, self-service way so that teams can run with their migrations without delay.

You can’t tell if you’re winning without a scoreboard, so we created a portal to gather, aggregate, and display patterns from the data assembled and made it visible to everyone in the company. Strategically, Tech Maturity provides a key indicator of our performance so that we can continually make value-driven improvements.

Example of how data is aggregated across capabilities to spot trends across products
We make these improvements at all levels within the company. Teams use their “product quality score” to prioritize work while leadership studies the aggregated data to identify patterns that reveal where the company can make strategic investments. These investments are geared towards reducing product development cycle time and leveling up all products across the board to increase our technological edge.

Open Source Tech Maturity Portal Available Soon!

The open sourced version of the Tech Maturity Portal will be released to GitHub.

With our Tech Maturity Portal, you can:

  • Quantify the maturity of your products
  • Visually compare individual product maturity to company-wide trends
  • Identify patterns for removing friction within your SDLC
  • Track and measure improvements over time

The best thing about our model is that it does not prescribe solutions. Rather, it offers standards with an aim to give teams a clear path towards efficient product development at scale. The model can be easily tailored to meet your specific needs. Ticketmaster has technology ranging from the 1970’s to today, and we have been able to successfully apply this model to products ranging from VAX code to JavaScript libraries. It’s a great vehicle for sharing and rallying around a common vision.

We’re excited to see the response from the community, and are thrilled to announce that we are collaborating with the Cloud Native Compute Foundation to demonstrate how technologies like Kubernetes provide a boost in Tech Maturity when used effectively.

To receive updates on Tech Maturity and the release date for the portal, sign up here.

Have feedback or ideas on removing friction in the SDLC? Let’s discuss!

Sherry Taylor | @seakitteh

5 Reasons Why You’ll Love Our Devjams

Back in January, we held our very first developer outreach and mini hackathon event, aka devjam, in Scottsdale, AZ, to a sold-out audience. Since then, we’ve had three of these one-day events in Durham, NC, Québec City, Canada, and Austin, TX, with one coming up in Los Angeles in June.

So far, the response has been very positive. The feedback received has helped us build better APIs and a better developer experience. The devjams’ Net Promoter Score (NPS) is 72, which is pretty high. So thank you to everyone involved in this success 🙂


Devjams are our way to, in the parlance of Steve Blanks, get out of the building and engage our customers. Since my team’s mission is to create APIs developers love to make our platform accessible, devjams are a great and fun way to engage our customers, the developers using those APIs.


Whether you’re a developer or someone interested in Ticketmaster tech in general, here are 5 reasons why we believe you too will love being at one of our devjams:

#5 Meet the team

Devjams are a great opportunity to meet the people of Ticketmaster who are opening up the platform, building APIs and shipping products. We believe that the best innovations happen when people connect at a human level, and devjams are a fantastic setting for that. Once you meet the team, you will feel the stoke!


#4 Join the team

It takes a special kind of talent to spend their Saturday with a bunch of strangers to talk APIs and product innovation. That’s the kind of talent we’re interested in and we have various opportunities where they can contribute. Who knows, we might be working together soon on some awesome products 😉


#3 Get inspired

Serendipitous “eureka” moments are a conversation or a keystroke away. It happens every single time without fail. You’ll find inspiration in the demos others give and in the cross-functional collaboration throughout the day. Given how intimate our events are (30-70 people), it’s easy to have meaningful conversations that will set your imagination free.


#2 Give feedback

As mentioned earlier, we focus very heavily on the developer experience, or DX. You’re the customer and we’re here to listen to you. We want your feedback. We want your input. As a developer myself, I love when companies do that because it shows they’re invested in me. We’re definitely invested in you, so come on out and share your thoughts with us!


#1 Have FUN!

By far the best reason for coming out to one of our devjams! We’re a fun bunch and like to connect with others. Our events are informal, laid back, and are as much about enjoying oneself as they are about coding and APIs. There will be plenty of food and beer for everyone. Just bring your jolly self and have a great time!


We hope to see you very soon at one of our devjams this year. You can also follow us on Twitter to stay abreast on our exciting journey to open up the platform at Ticketmaster.

We would love to hear from you directly. Please leave a comment below or reach out to me on Twitter.

Let’s build amazing products together! 🙂

How Agile Inceptions Are Changing Ticketmaster’s Team Culture

The Ticketmaster Way

An agile inception is a collaborative discovery workshop to visually get all stakeholders and team members to a common understanding prior to the start of a project.  The first time I observed my first agile inception I watched 3 transformations occur.  API developers’ eyes widened as they realized for the first time how their “piece” fit into the larger ecosystem and how different applications used it; developers were visibly excited to understand the business problem they were trying to solve and the team rallied together to then solve the problem and ideate based on a common understanding. Product, engineering, UX, program and our stakeholders were bound together by a common mission.   It was one of the most inspiring things I have ever witnessed a team go through. A year and a half later, the Ticketmaster coaching organization regularly facilitates inceptions as a way to kick off a new project or team and the inception process is now part of our technology culture. “The Ticketmaster Way” became a term used to describe how our culture was changing and we began to experiment with new processes and techniques to deliver ideas to our fans and clients.

How we got here

Like many organizations, Scrum gave us a base and common understanding around agile principles and ceremonies.  It became a very dogmatic approach and much of the principals were driven by the project management organization however, the value was not fully understood throughout the team.  Although, we were doing agile things, we were not agile minded.  It was time to change (yes, this was hard!)  At the start of our transformation, most of our teams were comprised of all one specific skillset on one part of the system or site.  We piloted the idea of creating a cross-functional product focused pod. This group of people were assembled as a team and had varied understandings of systems, architecture and applications.  It was then that we piloted our first inception.   The entire team moved to a separate building which gave them ample opportunity to form autonomously.  Ticketmaster’s first inception lasted 2 weeks.   During this time, we also introduced other agile frameworks into the team including Kanban, Lean and some Design Thinking workshops.


The Inception

We have since refined our process considerably.  After restructuring most of our teams into pods and over 40 inceptions later, we have found that 2-3 days at the beginning of the project to get to a common understanding is the right amount of time.  Inceptions are critical to help the team spend much less time in the forming and storming stages of team development and shift to norming and performing much quicker (therefore, eliminating waste!)

Figure 1. The Tuckman Model of Group Development

Inceptions are not only Ticketmaster’s way of kicking off a new project or team, it most importantly aligns the team around a concise mission with measurable business value – outcomes vs. outputs.  In fact, we no longer call projects…”projects”. We deliver on Promises of Value or (POV’s).

Who comes to the inception and what is the outcome?

We bring all team members, key stakeholders and customers into the same room (yes, we fly remote members out for this). The inception is an investment as part of the POV.  We keep the tone neutral by ensuring that it is facilitated by 2 members outside of the delivery team.  We break up the inception into 3 key areas. “The Strategy”, “The Work”, and “The Team”.  Although “The Work” and “The Team” are equally important, strategy is most often done at the product or business level and the team rallies around the work. Think of the team as “The Shark Tank”.  The team needs to be fully bought in and invested in the product that they are building to be inspired and motivated.  Because of this I am going to focus on “The Strategy” portion of the inception in this blog post.

Figure 2: Illustration via
Jeff Patton & Luke Barrett for Thoughtworks who re-created the cartoon from an unknown origin.

The Strategy – The Mission

The mission is not about telling the team what they will deliver. It is about understanding the problem that they are trying to solve. Product and design help the team to align on some materials prepared prior to the inception.  These materials help the team understand how the product will fit in to the ecosystem, who our competitors are, target users, the user journey, business drivers and the investment. UX will then will help to put some visual context, early prototypes, personas and design around what we are trying to solve. Finally, the mission statement is the most powerful piece. Why is the mission statement so important? It is what the team believes in and clearly states the problem that the team will solve. Product may come to the table with a mission statement already written, however, I typically have the team rewrite the mission so it is in their voice.  They own it. It needs to be something that any person on the team understands, is able to state and is passionate about. This is where it all starts. In writing the mission statement, I actually prefer a hypothesis statement format over an elevator statement. It clearly states what the team will deliver, the value it will bring to the business and how they know they will be successful.  At the end of the day, we really are building a hypothesis.  We have yet to prove its value.

We will deliver _______________________________

Which will enable the business to ________________

We will know we are successful when _____________

We are fans too! Bring the team members into the problem

The teams at Ticketmaster are passionate. Employees go to events; they experience shows and connect with the fan. We are fans! This passion becomes a critical piece in binding the team to the mission.  We begin the inception by allowing the team to fully experience the problem that our end-users are facing and empathize with them. We found that creating personas for the team was not enough, but having the team fully immersed in what the fan or client was experiencing sparked instant emotion, processing, and then ideation from the team. For example, in one inception we had the team call customer service and make changes to an order. We recognize that this is an experience that can be much better today, so what better way to have the team understand the pain then to go through it! This is the hand-off. It is at this point the team fully understands the WHY. They are not being told it will save the company x amount of dollars, or that it will improve the bottom line. They understand the pain and can see that this is a problem that can be solved.

When possible, we will bring our client or customer into the inception and have them demo directly to the team in how the product is being used and the many steps they have to go through to find a solution.

Measure Success

How will we know that what the team has been working on has been successful? Output is not as critical as outcomes. Just because we deliver something does not mean that it is meaningful to anyone. This final step is critical to motivate the team and help them feel successful in their journey. Are they going down the right path? Should they pivot? The team, just like the business, wants to know from the customer themselves. What do these measurements look like?

I often hear in these workshops that “we want to improve our Net Promoter Score or improve conversion”. But it’s not clear how this product or feature will move the needle in the many products we are delivering throughout the organization. It needs to be tangible. I usually ask 3 basic questions:

  1.  How do you know that you’ve achieved your mission for your slice or POV?
  2.  What does that look like and how will that help you make decisions or validate what to work on next?
  3.  When can the team celebrate their next win?

I like to ask when the team can celebrate their next win because most often than not, these are measurements that the team can fully get their head around – “We integrated with X and validated that the system is fully functional”. “We tested with internal users and validated that the user interacted with application as expected”…

It helps to break down the victory into much smaller wins which keeps the team motivated and excited.

Ticketmaster’s culture is changing. At the core of this change are the people who are passionate about solving hard problems. Given the context, room to innovate, alignment and a way to measure their success the teams become the change agents.  Invest in an inception, it will allow your teams to align early and deliver value quickly.  The key to kicking off a successful inception is by bringing the team into the strategy up front as possible.

  1. Get the team to collaboratively write (or rewrite) the mission statement.
  2. Demo the problem to the team.  Get every member on the team to feel the pain and then give this demo again for any new team members that come on-board.
  3. Understand the team’s success metrics and then measure success at the end of each slice/mvp.

Teams that are motivated feel empowered and driven.  Empowered teams produce amazing results.


Our journey to shorter release cycles

The team that is responsible for the development of the ticketing websites of Ticketmaster International has been around for many years.

Where we came from

When we started we had a rather undefined project based process where we released new versions of the site when a project was ready. It was generally many months between each new version. Sometimes over a year. We were using a waterfall approach.

The business made project requests and handed over their requirements to the product team. They then analysed the project and wrote a large specification that they handed over to the visual design team. When designs were done the engineering team analysed the specification and made a technical design. After that, we broke the project down in smaller work items and worked until we were done. Then testers got to work and reported bugs for anything that didn’t follow the specifications. After a few loops of bug fixing and testing again, the new version was tested again for regression issues.

Then the new version was handed over to the operational team that deployed it to a staging environment where the business could see the result. It was common that the end result did not meet the expectations of the stakeholders. All the handovers in the process meant that the original intent was lost.


Where we come from (Image courtesy of iquestgroup.com)

One step at a time

Our first step to improving this was to bring the testers into the same team as the developers. That meant they could test right after something was done. The next step was to regularly demo what we had done for the product team and the business. That meant we got feedback earlier if we were not delivering what was asked for.

We started to do sprints of originally 3 weeks with a demo after each sprint. Then we started to bring product managers into our daily meetings to be closer to the development. Then we made the engineering team more involved in defining the work. We broke the project down to stories that would bring value to the end user. The development team was actively helping product managers breaking down the project to user stories.

We shortened our sprints to two weeks. We also started doing test driven development. That is a way to help driving good technical design by doing automatic tests even before the functionality is developed. We also wrote higher level tests that tested the system from the outside. We decided that a new release would happen every 8 weeks.

After doing that for a while, we moved down to a 5-week cycle. We still had low coverage of automatic tests so we had to test the product manually for regressions at the end of each release. We got stuck in this state for quite some time but were happy with the way things worked. After all, we had come a long way from how we used to work.

We took further steps to work better together. We moved people to the same office, we started to create tools for improving our efficiency of deploying and generally automated as many manual tasks as possible. Coverage of automatic test grew over time.

Our most recent step

We wanted to take a further step to reduce our cycle. This time, the goal was to release to production after every sprint. It had a more profound impact than the changes we did before. All stakeholders would be affected as everyone would need to do their part more often. We decided to ask all stakeholders what such a change would mean for them and what changes were needed to enable a 2-week release cycle. We started tackling the identified issues.

Of course, our main priority in this change is to maintain the high-quality standards we have previously set for ourselves. We measure the rate at which high priority bugs are reported from production and how often and for how long business disruptions occur.  

One leap of faith we have to do when releasing after every sprint is that we removed the regression testing step that we used to have at the end of the cycle. That’s not to mean that we don’t test for regressions. Every story has manual exploratory testing included. All our automatic tests also catch regression issues and they are fixed within minutes of being introduced.  We also need to deploy the new version of code without disrupting sales. The local staff in each country that use our product must be able to translate any new text to the local languages before production deployment.

What we learned

We now have some 2-week releases under our belt and things have gone well. It has uncovered lots of inefficiencies that were hidden before and made everyone rethink their ways of working. Doing something that feels wasteful can be tolerated when you do it only every fifth week. But doing it every other week gets you thinking of ways to improve efficiency much more. It has also made us take the incremental approach much more serious.

I’m very proud of what our team has accomplished so far. It has taken hard work and dedication to reach the point where we are now.  The next steps would be to come to the point where we release code as soon as something is done no matter how small the change.


What Ticketmaster is doing about professionalising software careers

“My wife is doing her accountancy exams so is busy tonight, so I’m staying in to look after the kids” is something you expect to hear in every day conversation. But what about “My partner is doing his software engineering training tonight so I’m unable to come out for a beer?” Why is this still unusual? Are software developers still a product of their bedroom and dorm room past, hacking things together and so often saying ‘yes’ when no other professional would in a similar situation?

Software development, let’s face it, is an immature industry. We build software, but not many of our parents generation did. What we often fail to realise is the number of different roles we play. When we started out in software, we most likely will have felt the excitement of building something, and seeing it work. But to get it to work properly, we likely had to do some testing. As our initial projects mushroom, we start to learn various constraints – so without realising it, we start thinking about the design. Then when we have something that other people start using, they might have found a problem with it. So we need to do some investigative work. Then we find that there are additional requirements which can’t be met by the current design – so we start to think about the architecture.

What we haven’t realised is that we’re doing all the parts of what is known as the Software Development Lifecycle (SDLC): Requirements, Design, Development, Testing & Support.

Let’s think about other roles we play in our life. We may be parents, sons & daughters, friends and neighbours. But what about surfers, fans, climbers, shoppers and garage organisers? What we learn from looking at our lives that way, is that if we don’t plan for these roles, or put any time into them, then there are most likely negative consequences. If as a parent, we put no time or effort into parenting, or trying to be better as a parent, then that’s likely going to have a detrimental effect on us and our children. If as a garage organiser we put no time or effort planning or being a garage organiser, then our garage will remain a mess, and it’ll be impossible to find things when we need them.


So the same goes for us as software engineers. We need to plan, prepare and learn how to master the different roles we play, or there will be negative consequences. Software support is often seen as less glamorous than developing, but as engineers it requires sharpening our detective skills to be able to find a metaphorical needle in a haystack. This requires a completely different set of skills to development. So in order to take ourselves more professionally, we need to gain an awareness of these different roles, and gain maturity in developing our skills in these different areas.

Here at Ticketmaster we’ve been Agile for over 6 years now, and we’ve come to the realisation that with that level of maturity, multi-disciplined skills are required in order for development teams to continue to grow in a healthy way. When we reviewed our career progression processes across multiple engineering locations, we also found a lack of consistency across job roles and criteria for career progression. We came to the conclusion that without a transparent process that was consistent across all our teams, it was hard to set expectations and provide our engineers with the optimum experience for growing how they wanted to in their chosen profession.

So what we’ve done at Ticketmaster to address these issues is create a career map – a flexible set of pathways built from what we do as engineers day to day, but that provides both a range of options for career progression along with outline steps of how to get there. This provides mastery of any chosen career progression route, with a clear vision of the outcome that you’re looking for as an engineer.

How did we do this? We used a number of industry standard resources, building on the SWECOM model created by the IEEE, and using influences from other industry models such as SFIA. This has enabled us to build a three part progression guide:

progression guide

  1. SDLC skills – these are built up from the SWECOM model and outline 5 maturity steps in each professional skill, built from the SDLC.
  2. Technical skills – these have been built up using a platform and technology neutral approach, again providing a 5 step program of technical skill for all areas pertinent to Ticketmaster’s engineering strategy.
  3. Behavioural skills – we have outlined four areas for awareness and growing maturity that we believe make the biggest difference to our success as a people first organisation: Team work, Innovation, Results orientation, Professionalism. We can’t and don’t want to mandate behaviour, so we provide lots of examples of what maturity looks like at each level.

So armed with a clear map of what is required to progress, engineers are empowered to make this journey with the support of their colleagues. How do we do that? We used the guiding principles from Drive by Daniel Pink.


It was really important to us that we avoid this scenario, which you may be familiar with:dt_c131214

Again, we considered the engineering profession from the viewpoint of different roles, and saw the role of the manager as a coach, like Boris Becker working with Novak Djokovic. We want to create an environment in which engineers can excel, rather than be micro-managed or stressed by unnecessary distractions. So we trained all our managers to operate with a coaching model in mind at all times, supporting and only intervening from the sidelines when required.


We’ve empowered our engineering teams by giving them the tools and resources required to progress their careers – and them giving them responsibility for progression. The progression guide makes it clear what is required to progress to the next role. The guide was put together through numerous rounds of consultation, to ensure everyone had a say in how it was constructed. In order to satisfy the requirements of the next role, we have introduced a peer review and assessment process, whereby individuals submit their evidence to show they have mastered the professional, technical and behavioural skills to the appropriate level. What we learnt from engineering led organisations such as Google as well as other industries is not to have a single peer reviewing, but to have a minimum of three to ensure bias is minimised and the process is as objective, fair and transparent as possible.

We’ve rolled out training across all our engineering teams on how the career map process works, and run workshops on how to create suitable objectives in order to reach the requirements for each role. At each stage we have looked to put responsibility for progression with the engineer, but provided suitable support structures to enable and facilitate progression. This includes an online tool specifically created to help manage progress, providing a simple view for instantly seeing what steps are required and a convenient way to store evidence.


Software engineering is constantly exploring unchartered territory. That’s why agile has won out over waterfall as a methodology because iterative exploration and learning is almost always better than educated guesswork. Like software projects, software careers are also subject to a high level of flux because of the nature of the industry. Established skills can be rendered redundant overnight by the introduction of a new technology, product or service (e.g. the iPhone, or Uber). By providing engineers with multiple options in terms of charting their career path, we believe that we can avoid exposure to the threat of redundancy of specific technologies by building opportunities for career breadth and depth.

This enables engineers to not only change as the industry does, but also to facilitate proactivity rather than being constantly reactionary, which is far more stressful. Undue stress does not contribute to a healthy working environment, which is what we’re constantly striving for.

But we’re also looking to professionalise our industry, and make software engineering an even more exciting destination than it was a generation ago, when it was really in its infancy. By building our approach on existing standards and the best disciplines from other industries, we’re aiming to show that we’re part of an extremely vibrant but also professional industry which will endure for millennia to come.

Ticketmaster win big at the Techworld Techie awards

A huge night for Ticketmaster last night taking four prizes at the Techworld Techie awards! The Ticketmaster tech team were a very happy bunch taking the award for Most Innovative Team, Rockstar Developer of the Year, Best Place for Developers to Work, and the big one…the Techies 2016 Grand Prix award.

“This is great recognition for great people working for a great company” said Gerry McDonnell, SVP of Technology for Ticketmaster International who was especially proud of his team and their achievements.

On taking the prize for most innovative team, Adam Gustavsson said “Our team story is about a journey, a journey that started over four years ago in an attempt to answer a simple question: how can we move faster? That journey has taken us from a team that released once or twice a year to one that releases every other week, with even shorter cycles in sight. But more importantly, it is a story demonstrating that seismic change can be driven from the bottom up, even within large organisations.”

Nicolo Taddei was overjoyed on receiving the award for Rockstar Developer of the Year, “I personally believe that the biggest impact didn’t come directly from my software, but more from my personality on the team of engineers within Ticketmaster. With a positive mind set I enabled myself and my colleagues to take more action and calculated risks to improve our developer experience, code quality and innovation which have a real and positive impact on our clients and customers.”

We’re super proud of winning the best place for developers to work too. We love live entertainment, we live and breathe it. Everyone at Ticketmaster is a fan, hence our mission to bring out the fan in all! The move to our new offices in Angel three years ago came as part of a pivot in our company strategy, reinventing ourselves as a tech leader and the owner of the largest ticketing ecosystem in the world. “We look at ourselves as a technology company. A few years ago we didn’t do that. We had to redefine ourselves culturally”, Mark Yovich, President of Ticketmaster International said. With this award, it’s fair to say that this transformation is complete, as we now look to build on the solid base we have to provide fans and clients with the best possible experiences.

A big thanks to everyone at Techworld for the recognition and organising such a fun and inspiring evening, and hats off to Ticketmaster for sweeping the Techies 2016!

Black Friday – Celebrating savings through Kaizen

Whilst the internet whipped itself up into a frenzy about discounted televisions, here at Ticketmaster International we were celebrating savings of a different nature. For a few years, we have been using Kaizen techniques to improve efficiency and quality. “Kaizen”, a term borrowed from lean manufacturing, can be translated from Japanese to mean “good change”: it is a continuous improvement technique which recognises that the individual doing the job is the expert on that job and encourages individuals to make small, incremental changes that are within their power to implement.

Kaizen has been embedded in our business in different ways. For our developers who already use agile techniques, including scrum and Kanban, Kaizen was already at the heart of their operation. Regular retrospective sessions are used to identify, shape and get commitment to incremental changes to development and team behaviour. But whilst Kaizen is familiar to agile developers, it was an alien term to people elsewhere in the business. We initiated a programme to encourage wider adoption of Kaizen and empowered individuals to improve the processes they worked on a daily basis. A reward scheme was initiated and a shortlist of Kaizen implementations is reviewed regularly by the Exec team and celebrated through a number of internal communication channels.

Recently we have seen teams such as Ticketmaster Denmark and our contact centre team in the UK taking the Kaizen programme and making it their own. The contact centre team have started an initiative focussing on incremental changes that improve the customer journey. In Denmark, a Kaizen working group has been formed, not only to encourage Kaizen in their business, but to look at the Kaizen improvements that have been made elsewhere and see how they can be adapted and reused.

Black Friday is a US shopping tradition that is being exported through major online retailers such as Amazon. So with all hype about savings all around us, I asked people from our international business to share their Kaizen time savings and improvements. Here are some of their stories:

  • “We simplified the process for customer services engaging with event promoters. By cutting out intermediary steps they calculate they have saved 6 hours per month.”
  • “Our team have been doing some work to reduce the solution size, which has had a massive impact on [software] build times. So far we have reduced the build time of the solution from around 7 minutes to 2.8 minutes which is a saving of nearly 5 hours per day across the team.”
  • “By standardising print codes in Ireland we reduced duplication and conflict, saving 62 hours per month”
  • “One of my colleagues wrote me a script to automate the collection of KPI data. Hugely helpful and saves me at least 1 day each month.”
  • “We improved the guidance for visitors to the Copenhagen Office which led to a £255 reduction in metro fines and delays in the first 9 months.”

It is great to see Kaizen continuous improvement becoming more mainstream in our business and the benefits of small, incremental change being celebrated by such a diverse group. We will continue to evangelise Kaizen and provide coaching to teams who are interested in adopting it as well as other lean techniques and look forward to sharing more success stories with you on Black Friday next year!

Third-Party Components – Hidden Technical Debt

I was recently reminded of something I learned many years ago before coming to Ticketmaster from people much smarter and more experienced than myself. Back then I was pushing to introduce a set of third-party libraries to help lay the groundwork for a replacement for our flagship product, a mainframe based mail and groupware system. The logic, I thought, was flawless: The libraries would give us cross-platform support for a number of key functional areas including network communication, database access for many different database systems, file system, threading, you name it. Writing cross-platform code is pretty straight-forward until you have to touch the metal, and then it can be…challenging. Why re-invent the wheel, I thought, when somebody else had already invented some very nice wheels?

The company selling the libraries – yes, there was a time before Github and the explosion of open source libraries – was successful, well respected, produced quality libraries and offered great support. I did my research, readied my arguments and presented it all to management and senior developers. They were, in a word, underwhelmed. When I asked why they didn’t think it was a good idea I got the simple answer, “We’ve had nothing but bad experiences with these types of things”.

I was disappointed but there was a lot of work to do so I just let it go. But it did stick with me. I mean, why would seemingly smart and experienced developers turn their noses up at re-usable components solving common problems? Over the years however I started to understand their reluctance. Nothing truly catastrophic, mind you, just a lot of time spent wrestling with the devil in the details. And that is what I was reminded of the other day at Ticketmaster.

A Simple Job

The job seemed simple enough: Upgrade several open source components we use, all from the same group, from version 2.5 to 2.6. Certainly there couldn’t be any major changes, and the previous upgrade went smoothly enough. What could possibly go wrong? So we upgraded the components, ran the tests and BLAM, the first sign of trouble: a bunch of our tests were broken. Well, not just the tests. Our app was broken. In the end, it took a couple of people a couple of days to work through all of the issues discovered. And while QA always intended to perform a smoke test after the upgrade, testing was much more extensive than planned because of the issues during the upgrade.

This story would have ended happily enough except our app, a web-based e-commerce site, came out in production and BLAM, two showstopper bugs that required a rollback and immediate fixes. And both could be tied directly to changes in the third-party components we had just upgraded. This is not to say that it was bugs in the components that caused the problem. Rather, changes in the component code combined with our existing or new code lead to unintended, and more importantly, undetected side effects.

The Devil IS the Details

In the one case, the behavior of one component method had changed. Combined with some new, and seemingly unrelated changes in our code, the side effect showed itself in a very specific scenario with the result that a large group of site visitors would be unable to buy things on the site without first encountering an error. In the second case, a deprecated method for initializing a widely used component had to be updated to use newer and less clear methods. In this case, we simply implemented the new method wrong with a small, but very important side effect: we were passing the proxy server ip address to backend systems instead of the client’s ip address where the actual client ip address is an important part of the anti-fraud system.

So what’s the lesson of all this? Well some would say more tests are the answer. And they’d be wrong. In the first case, the error appeared in only one very specific scenario with a very specific set of pre-conditions. It was triggered by changes in our code, which we knew about, that interacted with changes in the behavior of the third party component, which we didn’t know about. Couple this with the previously unknown set of preconditions to trigger the error and you see that nobody could have foreseen the potential error and written a test to cover it.

In the second case, where we implemented the new method incorrectly, we had a test covering it. The problem was that the test was wrong. And this was owing to a tiny detail in the implementation of one of the component’s internal methods. And for the test-first proponents out there, yes, the test failed, was implemented and then passed. Problem is it was a bug in the test that made it pass.

To me the lesson is pretty simple: Think long and hard before pulling third-party stuff into your code-base. Don’t be blinded by “how easy things are to integrate” or “look at all the cool stuff we get” or even “everybody else is using it”. You really need to understand what you are getting yourself into and have a solid plan for how to maintain what has now become part of your code base. Because in the end, this is technical debt that you will be living with for quite a while.

Implementing a DevOps Strategy across multiple locations & product teams

Over the last 18 months, a change has begun within the Ticketmaster International Team. Barriers are being broken down between the engineering and operational teams, our different product delivery teams are being aligned and knowledge sharing across teams is happening more and more. What’s changed? We developed a strategy based around DevOps to create a leaner higher performing organisation and our journey is underway.

As with many large mature international companies our organisation is probably not unique; our Product delivery & TechOps teams are distributed across 5 geographical locations: Belgrade (Serbia), Gothenburg (Sweden), London (UK), Quebec (Canada) and Stoke (UK). Across these teams we manage about 15 different platforms. Our challenge was to create a DevOps strategy and implement change in a flexible manor is across all delivery teams.

With any distributed organisation we have suffered from communication barriers, although tooling such as Skype, Slack, Zoom, are all helping to break down the barriers. However, more fundamental issues existed such as terminology, multiple tools being used for the same job, skills and abilities differences between locations, and silos. A further example of silos was with our TechOps team being a separated centralised group, with different goals to the engineering team. When different groups that need to work together are not aligned and have different goals this can cause friction. In this case, because the way we’ve been organised, the multiple concurrent requests coming into TechOps from the various engineering teams has caused problems in their ability to service all teams at the same time which causes delays.

The differences in tooling and processes have created a barrier that slows us all down. We needed a new approach and developing a DevOps strategy has been one of the answers for us.

Our DevOps Strategy

In developing our DevOps strategy we wanted all teams to speak the same language, and have a shared understanding and skills. We wanted to break down the silos that had been built over time, bringing teams closer together and aligning resources to delivering products, so that we can be more agile, nibble, developing and releasing high quality products quickly, efficiently and reliably. Echoing the Agile manifesto principles:

Our highest priority is to satisfy the customer through early and continuous delivery of valuable software – Principle #1

Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale – Principle #3

Coalescing our ambitions and desires, mindful of the agile manifesto principles we defined 4 main objectives for our DevOps strategy:

  • Maximise for delivering business value.
  • Maximise the efficiency and quality of the development process.
  • Maximise the reliability of applications and environments.
  • Maximise for service delivery.

With these objectives we started to define requirements to achieve them. Quickly we ran into a mountain of requirements and with that a prioritisation nightmare: how to prioritise the requirements across 5 global locations and 15+ delivery teams, each with different needs.

The Maturity Models

After several rounds of attempting to prioritise in a sensible way we began to arrange the requirements into themes and with that a Maturity Model evolved; one maturity model for each objective.

Maximise for delivering business value. This goal is centred on continuous delivery, creating fast runways down which we can launch our applications.


Maximise the efficiency and quality of the development process. This goal is centred on continuous integration, creating the environment to launch a battery of automated tests and gain fast feedback to be able to evolve code.


Maximise the reliability of applications and environments. This goal is centred on instrumentation, creating the visibility into the inner workings of our applications for root cause analysis and fault tolerance.


Maximise for service delivery. This goal is centred on organisational change, creating alignment of cross-functional teams responsible for delivering software.


The Maturity Models are great; they provide a vision of what our strategy is. Defining the steps required to progress through to achieving advanced levels of DevOps, we can set long term and short term targets on different themes or levels to be reached.  They’re modular so we can change the strategy if improved technology or processes become apparent, and fill in gaps where some exist.

Flexible Planning

The nice thing about the maturity models is the flexibility they provide. They are maps that can guide you from a low maturity to a high maturity level of DevOps.  If you imagine how you would use maps to plan a route from A to B, depending on various conditions, such as day of week, time of day, potential traffic density, road speed, road works, etc the routes chosen will be most appropriate given a set of circumstances.


The DevOps maturity models are true roadmaps, as opposed to a linear list of requirements, allowing each individual delivery team to navigate their own path dependent on their context based on what is most important to them or what concerns they have at any point in time.  Furthering this flexibility, the Maturity Models allow teams to change their routes and reprioritise their plans in consort with business changes and needs.

When individual teams select and complete a portion of the maturity model no other team has yet reached comes with an additional benefit. The problems solved by those teams can be shared with all other teams allowing them to achieve that same work faster avoiding the potential pitfalls that would have been learnt by the early adopting team.

Even though all product delivery teams have the flexibility to select their own routes to achieving our DevOps objectives, ultimately everyone ends up at the same location. So the maturity models enable various programs of work to be planned across different teams with very different needs and abilities.


As good as our maturity models are they weren’t able to solve a couple of issues which still existed: we’re using multiple tools to do the same jobs and we speak different languages because we use different terminology for the same things. To solve this prior to kicking off our strategy we set up focused working groups to define and agree a set of standards for tooling, definition of terms (e.g. naming conventions), best practices (e.g. code reviews) and core specifications (e.g. Logging, Heartbeats & Health checks).

Our Core Tooling

  •         Git – Source Control
  •         GitLab – Git Management & Code Reviews
  •         Jenkins – Application Builds
  •         SonarQube – Code Quality Reporting
  •         Sonatype Nexus – Package Management
  •         Rundeck – Operational Support
  •         Octopus Deploy – Deployment (Windows only)
  •         Chef – Configuration Management

Standardising our tooling and specifications for implementing instrumentation meant we could reduce support overheads, share knowledge and solve problems once. Guidelines and best practices meant we’re working in the same ways and all have shared understanding. Definition of Terms meant we could all speak the same language and avoid confusion.

With the maturity models and standards we have created a shared vision and enabled flexibility for each product delivery team to plan what they want to work on. We have created a framework that enables all product delivery teams start working on achieving the DevOps objectives in parallel but focusing on what’s important to their needs at any given point in time.

Sold-out Event Kicks Off Developer Outreach at Ticketmaster

As we prepare to roll out a ton of new technology and deliver APIs that developers love this year, we’ve started to engage the developer community to get their feedback on our redesigned developer portal and the updates to our APIs before we launch them.

Last Saturday, we held our very first event in Scottsdale, AZ, to a sold-out audience of enthusiastic geeks. The event helped set the tone for all future engagements with the developer community. It also clearly showed how external developer feedback is critical to the health and stability of a platform and how we need to hold more of them often.


Over 75% of registered participants showed up. The market average of free events like this is about 40%. I think we did well 🙂

The turnout showed how much the community is interested in what we have to offer. Some of the attendees flew in from Virginia, NY, Washington and California. The energy and the feedback we received were far more than I had hoped for.

Also, five API demos were given at the end of the day, which was great to see 🙂

jodymulkey_2016-Jan-16 (1)

The Results

The verbatim we received from developers was overwhelmingly positive with lots of feedback on what we could do better. Here’s some examples of what they said:

The whole platform is very clean and simple. It’s very easy for any developer to jump right into it.
all good…took a few minutes to get oriented to the various parts.
I found myself flipping between the interactive demo and the static documentation a lot. It would be helpful if I didn’t need to do that.
API response docs should be present
So far I can see it is very easy to use, with clean documentation and quick starter.

I’m a biz dev guy…the fact that I can understand any of it makes it pretty impressive from my pov.

The API needs to consistently deal with images for non-events. We also need a way to get a link to an event itself, not its attraction(s). Also, it needs to have a way to identify the base URI for relative URLs.

A few common themes emerged:

  • API bugs
  • Data inconsistencies
  • Documentation completeness
  • Need for tools and SDKs

We gathered detailed feedback throughout the day and the team is currently addressing them. The state and quality of the platform will be a whole lot better by the time we hold our next event in Los Angeles in February (stay tuned).

Open Platform NPS = 65

Overall, our initial Net Promotor Score, or NPS, is 65. This is a pretty high score (Amazon’s is 65) and shows a lot of goodwill from the developer community. Our goal is to keep our NPS above 70. Now that would be amazing! 🙂

Aside from the written feedback, we also asked developers to rate us on various aspects of the platform. Here’s how we fared:


Stay Connected

Follow us on Twitter and subscribe to our Medium Publication to be the first to learn when we launch the developer portal and make our APIs publicly available. Exciting times ahead for the Ticketmaster Open Platform 🙂

Tests and Comic Strips: How Dilbert Explains a Philosophy of Testing

At Ticketmaster, we’re committed to testing. Automated tests improve our stability, helping us sell tickets to more fans and add new features. As a developer, I have an additional motivation to write tests well: tests illustrate the intended behavior of my code. Besides achieving good code coverage, an articulate test quickly teaches another person what my code does. Here are a few cues that I’ve settled on, inspired by comic strips.

Testing speeds things up.


DILBERT © 2015 Scott Adams. Used By permission of UNIVERSAL UCLICK. All rights reserved.

Keep it short

Three frames, four lines of dialogue: that’s limited space for pictures and words. Readers will skip over a fifty-line test block with glazed, lizard eyes. Do you want your readers to pay attention? Keep it tight, then.


DILBERT © 2015 Scott Adams. Used By permission of UNIVERSAL UCLICK. All rights reserved.

Stay focused

Test one feature at a time.


DILBERT © 2014 Scott Adams. Used By permission of UNIVERSAL UCLICK. All rights reserved.

Do you love Game of Thrones? Complex subplots, amirite? Intertwining narratives are fun, but testing many things at once is just confusing. Instead of writing one script that covers all possible cases, I separate each behavior into its own test. One strip, one joke; one test, one case. Isn’t that a Marley song?

Be self-contained

The Boss wants data-driven product releases.


DILBERT © 1994 Scott Adams. Used By permission of UNIVERSAL UCLICK. All rights reserved.

Do you recognize Dilbert and the Boss? Of course! Did you forget what happened before this strip from 1994? Maybe a fanboy remembers, but everything you need to know to get the joke is in these frames. I describe this as episodic style. The tests form a series of short narratives with familiar actors, but the events of one test do not impact the outcome of another. When tests rely on the same data, I generate fresh objects before each test. Tweaking the data for a particular test won’t pollute the tests that come after it, and my readers won’t need to track state as they scroll down the page.

Clear narrative

Have you noticed the similarity between Given-When-Then stanzas and the Arrange-Act-Assert pattern? These templates both resemble story structure because people remember narratives. While writing a test, I map the logic into exposition, rising action, and resolution. If any phase is longer than a line or two, I use comments or wrapper functions to delineate the boundaries. But I can’t hide too much detail—the punch line loses its sting if you don’t know the Boss practices feng shui martial arts.

Don’t leave your app defenseless.


DILBERT © 2015 Scott Adams. Used By permission of UNIVERSAL UCLICK. All rights reserved.

Tie it all together

There are many elements to a comic strip: the dialogue, the characters, the scenery. These layers all converge on a single message. Do you see Rodney’s wrinkled tie? This subtle detail emphasizes just how unsafe the Boss’ new product is. Without pictures, variable names must evoke imagery that reinforces the behavior being tested.

Poor Rodney—no wonder his tie is wrinkled.


DILBERT © 2010 Scott Adams. Used By permission of UNIVERSAL UCLICK. All rights reserved.

That’s all. I write short tests that focus on a single behavior. I name variables and functions carefully, and isolate state within each test. I ensure that the sequence of actions conveys a narrative structure. These elements of style focus the reader’s attention on my code’s behavior and foster clear understanding in a memorable way. I hope you find these cues helpful.

Love it? Hate it? Know any good jokes? Tell me what you think by dropping a note to matt@ticketmaster.com.


Using Multiple Languages in Your Development Environment

Many modern software engineers now work on multiple projects over the course of their career, each with different requirements.  These requirements often cause us to consider different tools and even languages to get our work done faster and more efficiently.  The Commerce Order Processing and Order Details team decided to take a different approach and decided to integrate multiple languages into its development environment, using Java and the JVM as a base.

The Transition to Groovy

Java is a great language.  It’s well supported, it’s standard in many large corporate enterprises, it’s taught in virtually every single CS/CE program so you have a large pool of talent to draw from.  It allows you to create highly structured programs, and allows code reuse to a large extent.

But sometimes Java is frustratingly inflexible.  Some code that should be dead simple is very complicated by virtue of forcing you down a particular philosophy.  Worse, experienced Java developers tend to create designs that encourage complication and lots of structure.  For a team that wants to be more flexible and agile, this is not good.  Java 8 has created some mechanisms designed to address some of these, but they are in the very early stages.

The Order Details team at Ticketmaster wanted to experiment with design philosophies to encourage faster development and more flexibility; however, we had a large code base that we needed to add features to and we didn’t want to rewrite most of this code.  So instead, we decided to test something out – introducing a language that wouldn’t be as verbose as Java.  Enter Groovy.

Groovy is a language that compiles to the Java Virtual Machine.  Its syntax builds on top of Java, which makes it very good as a transitional language – you rename a file’s extension from .java to .groovy, and it just works.  Groovy has a lot of features that make it attractive: true closures, automatic field access, syntactic sugar to compact code, etc.

This isn’t possible without the JVM itself, which in itself allows you to use multiple languages in the same program.

Add in some Clojure

We needed to add a feature to our Order Details service that involved custom data translation.  The details for that will be in a future blog entry, but eventually we decided to use Clojure as our implementation language, as it’s naturally suited to this type of feature.  A similar feature that existed in one of the older versions of our service was 1-2 orders of magnitude larger in terms of lines of code, so we were motivated to make this change.

However, Clojure as a language is also a more difficult language to adapt.  To help ease the transition, the members of our team decided to read documentation and books on the subject; we also used pair programming heavily to spread information through the team quickly.

It took around 1-2 months to finish the feature; however, the schemas to create the translations were easy to understand, and by and large the engine needs very little modification once it was written, so it was effective.

Advantages to the Approach

The main advantage in this approach is that you can write code that is particularly efficient for a particular problem without having to throw out existing code.  The Clojure code, when written in Java, would have been several times larger and much harder to debug.  In addition, if we needed something that was better to do in Groovy, we wrote it in Groovy.  This included unit and integration tests, which is not well supported in Clojure.  The Groovy change allowed us to slowly transition into a language that allowed us to move faster without having to slow down and learn an entirely new language.

Disadvantages to the Approach

However, languages need to be supported and new team members (whether they’re new hires or other teams working with our code base) need more time to ramp up to the code, especially if they’re from a mainly Java background.  We decided the code for the feature was isolated enough that the transition can be done over time, but it’s still an issue.

In addition, there are certain architectural problems you won’t be able to solve with solely a language change.  In that case, it may be better to start from scratch.

Who should use this approach and why would they benefit?

I don’t think this approach should be used for all teams; but teams with the characteristics below could be well served by this approach:

  • A high degree of autonomy and engineers who understand multiple languages
  • A domain-specific problem set better suited with another language
  • An experimental approach to target a specific technology without leaving their current development environment



Using multiple languages in the same development environment can be useful in the case of a specific problem domain or the desire to transition to an approach where they believe they can do work more efficiently.  While this approach uses the JVM as a target compilation platform, Javascript can and has been used as a platform as well.  The approach isn’t for all teams, but it can yield great gains when used correctly.

2015 London tmTechEvent – Ticketmaster’s in-house tech conference


“Welcome to the tmTechEvent 2015,” John McIntyre, head of PMO at Ticketmaster International announced as he scanned my badge. The scanner beeped appreciatively as it recognised my credentials and I was granted entry – just like going to the O2! Apart from the fact that this was our annual Ticketmaster technology summit meeting in a conference room just up the road from our offices at the edge of London’s Silicon Roundabout. But this wasn’t any old conference. Being in the live entertainment space, we had quizzes (using Kahoot.it), a live Twitter feed (@TMTechEvent) and a party in our London HQ’s basement bar complete with gourmet burger van!

As I entered the room there was a palpable sense of tension and nervous excitement in the air as I greeted my colleagues from the Sports division: with the Rugby World Cup starting on Friday, their systems would be under the spotlight – or was it just a matter of doubt over England’s ability to deal with a tricky opening tie with Fiji? Ultimately both fears proved unfounded – all the events ran smoothly and England prevailed.

Along with leaders from all of Ticketmaster’s other technology teams, we had joined together to take stock of our progress in revamping our technology real estate. The 4 day event was packed full of seminars, workshops and group sessions, with the overall aim of evaluating our strategies and determining where to correct our course. We followed the guiding principle of “focus where you want to go, not what you fear.”


We pulled in leaders from all over the business to help shape our vision of where we’re going. We combined this with focused feedback sessions on the various aspects of our strategy to determine the best way to pivot to adapt to the changing landscape.  Using workshops to facilitate a rich exchange of ideas we covered subjects from staff satisfaction to talent management to deeper technical subject matter such as engineering KPIs and creating reference architectures.

Other highlights included live demos of in-house tools, including one that had been created to show the visibility of progress in our DevOps program. It was really cool to see each team’s progress in one place across our home grown four- part maturity model. Even better was sharing the whole event with colleagues across Engineering and Operations and feeling a real sense of unity, proving that good DevOps is about culture change and not just a bunch of new processes!

conference 2015

The sheer scope and depth of material covered was brain bending. We rolled out our new career mapping program, providing a structured career map and promotion process across all of our engineering teams. We had a thorough review of the initial results of our innovative and much talked about technical debt management program that we rolled out at the start of the year. We reviewed the progress being made on our employee feedback survey, to ensure that the concerns of our engineers are being taken seriously (it’s not just about having more opportunity to play ping pong!)

Overall this was an inspiring event. There was a tangible confidence and will to achieve our ambitions, based on a very real sense of achievement from how much we had already changed things for the better in Ticketmaster Engineering. The desire to increase collaboration and tackle the bigger challenges ahead together was strong.

Having reset our sights on the vision of the organisation, our technology vision and our engineering vision, the result was a noticeably energised and motivated group of rock-star tech leaders ready to take that vision back out to their teams and the company. The future of better live entertainment starts here!

In Search of a Reactive Framework (or: How we select new technologies)

About seven or eight months ago we started looking at new and improved ways of creating services instead of our tried and tested Spring framework based approaches. The Spring framework certainly has its merits and we have used it with much success, however, the service we were about to write, called Gateway, was to route millions of requests to other services further down in our architecture layers:


We knew that the service had to run efficiently (thereby saving on hardware) and scale effectively. The event driven, reactive approach was one that we were looking to embrace. After much research we had some concerns. Not only is that space full of different frameworks at different levels of maturity, but we also had to consider our current skill set, which is predominantly Java with a small amount of Scala, Ruby, Clojure (which our data science guys use) and a handful of other languages we’ve picked up through company acquisitions. How could we adopt this new paradigm in the easiest possible way?

What this blog post will detail is the approach we used to select the framework we chose. It’s a tried and tested approach we’ve used before and will continue to use and improve upon in the future.

How we did it

Before we describe the stages of what we did, suffice to say that there is no point in doing a technology selection without a business context and an idea of the service(s) a framework will be used to build.

The technology selection was broken up into the following steps:

  • Identify principles – these are the rules that the framework must adhere to. These are properties that a framework can meet in different ways and contain a degree of flexibility
  • Identify constraints – these are rules that cannot be broken or deviated from in any way. If any are broken, the framework is no longer a candidate.
  • Create a short list of 5 – 10 candidate frameworks
  • Determine high-level requirements and rank using MoSCoW prioritisation
    • Read the documentation, Google groups and other relevant articles and trend data to determine conformance to the requirements, including all options and workarounds if applicable.
    • If any of the Musts are broken by a candidate then it is dropped
    • Create a short list of, ideally, three candidates
  • Create a second set of more detailed requirements, rank using MoSCoW and weight in terms of importance
  • Determine the architecturally significant business stories and error scenarios that the service to be built from the framework needs to implement:
    • Write the end-to-end acceptance tests for these stories. The primary scenario with one or two error scenarios is sufficient.
    • Implement these end to end acceptance tests in all three frameworks – this will give an idea of how well the framework meets the service’s paradigm(s), how easy it is to work with in the develop/test cycle and also make sure to post on the message boards or mailing lists to see how quickly a response arrives from the framework maintainers.
    • Update the second set of more detailed requirements with the results of this experience

Our results are pretty detailed so have been added into a separate PDF that you can download here. We have left this in raw format and hope that they will be a good reference for others.

Outcome and experiences to date

As you can see from the PDF of results, we chose Vertx. It won out not only because of its raw power, but because of it’s fantastic architecture, implementation, ease of use, Google Groups support and the fact that Red Hat employs a small team to develop and maintain it. Indeed, a few weeks after we selected it, it was announced that Red Hat hired two more engineers to work on Vertx.

So overall we have been very happy with our selection of Vertx. We had version 2.1.5 running in production for several months and recently upgrade to Vertx 3. The maintainers’ swift response on the Vertx Google Group definitely helped during our initial development phase and during the upgrade to version 3. Performance wise, the framework is extremely fast and we know that any slow down is most likely due to what we have implemented. Adoption has been a success. From a team of two developers, we scaled to four and now eight. Choosing a Java based framework has been a boon as the only additional complexity that needed to be learned by the developers joining the team was the event driven nature of Vertx (i.e. the framework itself). Had we chosen Scala/Play it would have been much harder. Indeed, with the success of Vertx, our decision to standardise on the JVM as a platform and our embracing of the reactive approach, we have a couple of services being built using Scala and one using Scala/Play. It would be great to hear of your experiences using reactive frameworks. Which ones did you choose? How easy were they to adopt? Please leave a comment, below.