On the last night before employees were leaving for Christmas break and the 2015 New Year, desks on the 10th floor of the Ticketmaster Hollywood office were littered with red plastic cups foaming with champagne, and greasy slices of fresh Raffallo’s pizza, a Hollywood staple. The normally tepid office was buzzing with enthusiastic conversation and clapping from our Engineering, QA and Product teams, still high on adrenaline from last-minute bug crushing and testing.
The mobile team was celebrating an audacious feat of full-stack software engineering. In a span of only 2 months, the iOS team, along with their counterparts across many teams and offices at Ticketmaster, had successfully implemented Apple Pay as a new payment method within the iOS Ticketmaster app. This was a daunting and technically ambitious success story for a company with many discrete payment and ticketing systems, not to mention business and legal requirements as Ticketmaster. With the new version of the app being released before Christmas, Apple had selected the Ticketmaster iOS app to be featured in the App Store as one of the first adopters of the new payment system.
Meanwhile the Android team hadn’t released a new version in over 6 months. Buried under a mountain of technical debt, a backlog of style updates and new feature requirements, externally there was probably very much a feeling that the Android app was falling even further behind its iOS brother. Internally, as the Android developers left the offices for their vacation, there was reason for optimism within their own development and QA teams as well. Only one message was scribbled along the corner of the whiteboard in the Android development area:
“2015: Year of the Android”
I moved to California to take up a Sr Android Developer position at Ticketmaster in September 2014.
I was excited about the great Los Angeles weather but I was also pleased to see the metrics associated with the application team I was joining. The metrics showed great year-over-year growth for both the Android application and mobile in general. More and more people were purchasing tickets using their mobile devices, ensuring that the role would have an immediate impact on the company’s short and long term strategy.
Since 2014 there was a 20% increase in mobile visits and a 35% increase in mobile ticket sales
Unfortunately this wasn’t the full story. For years the Android app had accumulated technical debt and there were other issues that required immediate attention before we could focus our attention on some of the exciting opportunities we had identified.
One of the growing philosophies at Ticketmaster is Lean and to think of products in terms of continuous improvement. The idea can be concisely summarized with the sketch from Henrick Knilberg’s “Succeeding with Lean Software Development.”
Deliver usable products to allow learning to take place. Illustration by Henrik Kniberg.
To help visualize the Android Application in this manner, I liked to pretend the app was itself a concert venue. Just like its real world counterpart, the Android app was serving a growing line of consumers who were migrating from the desktop web application to the mobile app. The app was already serving thousands of people per day, but it had plenty of room to grow. It was this metaphor that made a lot of the problems jump into focus.
A metaphor for the Ticketmaster Android app and a small concert venue
Our user base was growing, but how were we handling this increased load?
Very little security or organization in the main entrance
The line of users looks a little bit unorganized. What exactly are they doing while they are waiting in line? It doesn’t seem like we are providing very much guidance. Was everyone coming to the mobile app from the same place? Were we funneling everyone to the right place? Also, the security sure seemed a little bit light at the entrance.
Users confused by outdated and ambiguous navigation elements
Once people actually got into our application (And our “venue” in this metaphor), they were often confused by ambiguous navigation and UI elements. The app was a lot more complex and confusing in terms of UX experience than most of our users were expecting. A lot of this was due to legacy code and design. A lot of the designs from 2011 were persistent in our 2014 app, despite plenty of design stories in the backlog.
Maybe this wasn’t such a great opportunity after all! How do we address all of these issues at once? It was going to be a long time before the Android team was ready to celebrate its own release.
The first key was to make building the app a source of joy for developers and the QA team. The 10-minute build times using Maven and an increasingly confusing set of dependencies had to go. Luckily, there was a new company whose motto was “build happiness.”
Gradle To the Rescue
The migration to Gradle provided challenges to both our development team and our QA team. But once the migration was made, everything was easier. Build times went from 10 minutes to under 2. With advice we got at the Gradle Summit (a conference hosted in Santa Clara this year) , we were able to decrease our development build times to under 40 seconds using the Gradle daemon and the incubating features like parallel builds and configure-on-demand.
An Android team member discusses the improvements to our build time over Slack
When I was flying home from the Gradle Summit, I used the offline and no-rebuild flags to allow me to use my most recently downloaded dependencies and allow up to date builds while offline. Gradle plugins, such as the Jenkins plugin allow synchronisation between your local Gradle build arguments and the ones you are using in Continuous Integration.
Gradle has been a star tool for Android development, with new training available at:
http://gradle.org/getting-started-android/ and https://www.udacity.com/course/gradle-for-android-and-java–ud867
Every Android developer needs to learn how to best take advantage of this awesome tool. Gradle is more than a build and dependency management tool, it is truly a swiss army knife, as we quickly learned this past year.
As an example, we had to figure out a way to allow our automated UI tests to bypass some of our security measures without risking leaking these bypass mechanisms into the production app. These complex requirements became easy once we approached them as gradle tasks.
Track and manage Technical Debt
One of the lucky parts of starting at Ticketmaster is that I never felt like I was alone on an island as a developer. The QA team at Ticketmaster is the strongest I have ever worked with. They are engineers more than testers and their knowledge and experience with the Android app allowed me to grow into my role. The QA engineers rapidly built up our automated UI testing suite using Cucumber and Appium and added it into our continuous integration pipeline. Even though the app had a long way to go, it was definitely an aha moment when we got our first 100% passing UI tests report that was completed overnight.
Automated testing and nightly reports
Our automated tests would immediately detect and problems with our ongoing development. But the UI tests that our QA team was building wasn’t enough. They were great at catching new issues within our User experience. But they didn’t take account of technical debt and insidious bugs within the code base.
Most software developers have heard of the testing pyramid, and with the Android QA team pulling its weight with Automated UI testing, the need for unit tests was even more glaring.
We found the first step to cutting down our technical debt was measuring it. And SonarQube provided a way for us to do this within our build pipeline. Sonar calculates a number “Technical Debt” using a combination of factors: Unit test and Integration test coverage, java logic issues and duplicate code.
Adding unit test coverage was not easy. We had to figure out the JaCoCo gradle plugin so that we could calculate the coverage numbers before these could be added to our Sonar console.
When we first got Sonar integrated into our build pipeline the numbers were grim. Over 700 days of technical debt for the Android app alone. At least we were measuring it now and had a starting point. From now on, conversations between developers about technical debt had a point of reference that we could come back to as we debated approaches toward implementing new features and bug fixes. Often times there were opportunities and “low hanging fruit.”
If a developer completed a new feature, we could immediately examine its impact on technical debt. Were we adding any new major java logic issues? Had we added any technical debt to our already large code base? Were there deprecated classes or features we no longer needed that could be tackled within this story?
In January we were able to reduce our technical debt estimate by 158 days, and increase our unit test coverage by 20%. Part of this was easy; we had some unit and integration tests that hadn’t been measured by sonar. Other gains were made by getting rid of deprecated code that was no longer needed.
Code Quality Summary (Jan 01- Jan 30)
Technical Debt – Reduced by 158 days
Unit Test Coverage – Increased by 19.4%
Major Issues – Reduced by 458
SonarQube measures delta improvement to unit test coverage
In February we ripped off another big chunk, 298 additional days of technical debt removed from the app. As we moved closer to releasing our new re-skinned app, we were also cutting down technical debt. This two in one combo of adding features and making the app easier to maintain allowed us to make quicker and quicker improvements for our users. Not to mention, this also improved our build times even further.
Security isn’t an Afterthought
For many months, our focus was on getting the “Android Design Update” out the door, and we were aiming for a February release. Unfortunately, there were security issues that forced us to change our priorities and release schedule.
Going back to our “Event Venue” metaphor, we were hearing more and more from our API team and our data science team that the Android App was getting abused by malicious users.
Bots were grabbing the some of the best seats in the Ticketmaster Android App
This was a major issue that we needed to address. In Android, the basics of security come from three places:
- Secure coding practices
- Code obfuscation and minification using Proguard
- TLS network security
The Android app was using these practices already but it wasn’t doing enough. With a high profile application like Ticketmaster, malicious users are likely to go to extra effort to defeat traditional measures. We needed something extra.
Enter Dexguard. If you haven’t heard of Dexguard, it is an extra layer of security provided by the makers of Proguard but designed specifically for Android applications.
Dexguard allowed us to add not just one, but many additional layers of security.
Dexguard added lots of new security features to the Ticketmaster Android App
Some additional security layers provided with Dexguard that were not possible with Proguard:
- name obfuscation
- string encryption
- method reflection
- tamper detection (at runtime)
- environment checks (at runtime)
- class encryption
When using tools to decompile the Android app, Dexguard makes things much more difficult for reverse engineering. Dexguard allowed us to push a lot of the bot traffic out of the Android app immediately.
The 6 month delay between releases was too long, If not for the security updates, we would have gone this long without delivering improvements to fans. With Gradle, Sonar, and automated UI testing, we had the tools we needed to deliver constant updates without suffering out code quality. With each commit, we trigger tests, our lint and sonar code quality metrics, and our automated UI tests. All of these build steps are triggered by gradle tasks.
With a revamped CI pipeline with automated testing, new features can be added more reliably without incurring additional technical debt
After we released our Android Design Update update in late February, we started with bi weekly updates. After going through only 2-3 updates (sometimes only for security reasons) in 2014, we have released more than 10 updates to the Android app since February. This is the reason the Android developers were secretly happy back in December, we knew were close to putting it all together.
Think Big Picture
As an Android developer, it’s very easy to get sucked into the issues that affect you on a daily basis. How do I make this list update faster and make this UI consume less memory? What is the best way to create a clickable span within an expandable list? But it’s equally important to see the big picture. How will this app become what it is capable of
Fixing the process, and tracking was the most important step. Taking some time each week to think about the pipeline and how it fits into the larger picture was key. For mobile architecture, I strongly recommend using C4 diagrams which allow developers to layer additional details for more sophisticated engineers.
When I am introducing a new members to the Android code base, I can start with a Context diagram, and once I feel they understand it, move on to the containers, components, and of course, our existing technical debt and long term goals. Using C4 diagrams it is easy to visually pinpoint which parts of the codebase are holding more technical debt. It’s a springboard to software fluency.
There will always be new unforeseen challenges. A few weeks after we released our Android Design Update, most bots moved to attack the iOS app. Other more advanced hackers found a new way to exploit our existing security, requiring us to display Google’s reCAPTCHA to all users attempting to purchase tickets on the Android app. Our users were not happy.
Working with our API team, we are going to implement new features to overcome this issue and we will always have new challenges. With all the improvement that have happened to the Android app over the past year, it really validates the team’s work when we see feedback like this in our user reviews:
The scribble on the whiteboard was just an idea. A simple idea that our team could make the kind of improvements that had started become a part of our process and that releases could just become routine.
And then it happened.
JaCoCo Gradle Plugin: https://docs.gradle.org/current/userguide/jacoco_plugin.html
Jeff Kelsey is a Sr. Android Developer at Ticketmaster