Access Control Testing with Calabash

The JAC Scanner team is developing Android-based access control software to scan tickets for entry or exit at venues. The JAC Scanner application is designed to run on any Android cell phone using the camera to scan tickets, and it will eventually support dedicated scanning devices such as those available from Janam and Motorola. The application is being developed in C# using Xamarin. It has a hybrid user interface, with all controls contained in a web view object. We chose Xamarin and the hybrid interface with an eye towards more easily porting the application to iOS in the future.

Automating functional tests in this environment has raised some unique challenges. We need the tests to run on the target handheld device, yet be under control of a PC. We need to be able to present tickets to the device to be scanned. In addition, we need to control the state of the device during the test. To solve these issues, we considered three test frameworks: Selendroid, Eggplant Functional and Calabash. Each had its pros and cons; none of them provided a perfect solution.

We first looked at Selendroid, or “Selenium for Android”. This was an obvious first choice because many of the teams here are already testing web applications with Selenium. We found that testing with Selendroid is easy to set up, but that Selendroid has issues with applications generated with Xamarin. This looks like it would be fairly easy to fix, but we decided to explore other possibilities before going that route.

We also evaluated EggPlant Functional. This appears to be an excellent testing system with an advantage over the other frameworks we tested: it is automating the Android device, not the application under test. This means the tests could do things more difficult to do in other frameworks, such as exit the application under test, change settings, and re-enter the application. EggPlant is image based; the tests look for specific images or text on the screen and take actions when they appear. Some of the things we wanted to do, such as “click on the 5th item in a list” were difficult to do under this system without knowing ahead of time what the item looked like.

Lastly, we looked at Calabash. We expected this to be the easiest to set up for a Xamarin-based application, because it is maintained by Xamarin. In reality, we had a serious issue with it at first. Our user interface CSS contained zoom statements, and Calabash does not support them. This caused incorrect screen calculations, thus causing the tests to miss controls. We soon realized the zoom statements were the problem and removed them. Once we did so, everything fell into place, and Calabash performed well.

As you have probably guessed from the title of this article, we chose Calabash for our tests. Removing the zoom statements from our CSS got it working well. It is reasonably well documented, and because it is maintained by Xamarin, we have a reasonable expectation that it will remain compatible with Xamarin-based applications. In addition, it is open source, so we can modify it if we need to (and we did need to! More on that later…). Finally, Calabash tests are written using Cucumber, which our development teams prefer because it expresses application behavior in plan English.

Calabash works by creating a test server to manage the application under test. The application and the test server are both uploaded to the Android device. Calabash provides an API that tests can use to communicate with the test server; the test server, in turn, controls the application under test. A big benefit to this arrangement is that no modifications are required to the application under test.

Since Calabash tests are written for Cucumber, they are expressed as features and scenarios. A feature is a group of related scenarios, and each scenario is a test with one or more steps. To ensure the tests remain isolated from each other as much as possible, Calabash reloads the application under test onto the device at the beginning of every feature. It also stops and restarts the application for every scenario.

Once we had Calabash up and running, our next challenge was to automate presenting tickets to the device for scanning. We solved this by pointing the device at the screen of the PC running the tests, and writing steps to display the barcodes we needed on the screen. Steps are implemented in Ruby. In our case, we are using JRuby, so we use Swing to display the barcodes.


As our application grew, we ran into additional challenges. For example, there is a whole set of behaviors we want to test which happen when the Wi-Fi signal is lost. Since Calabash does not provide a way to enable and disable Wi-Fi during a test we modified Calabash to provide this capability. We created our own public fork of Calabash and added the Wi-Fi feature tailored to our needs.

As an example of the kinds of tests we are running, here is part of a set of tests that ensure the databases on the handheld device and on the server are updated (or not) as expected. These tests make use of our modification to Calabash that allows Wi-Fi control;

Scenario: When a ticket which has already been entered in online mode
          is scanned in local mode, it is rejected as a duplicate

    Given the scanner has been initialized for a test venue
    And a valid ticket that has not been scanned in
    When the valid ticket is presented
    And the scan button is pressed
    Then the scanner permits entry
    When the Wi-Fi connection is disabled
    Then the scanner is in local mode
    When the valid ticket is presented
    And the scan button is pressed
    Then the scanner rejects the ticket as a duplicate

Scenario: When a ticket is scanned for entry in local mode,
          the ticket cannot be scanned for exit in online mode
          if the local data has not been pushed to the server

    Given the scanner has been initialized for a test venue
    And a valid ticket that has not been scanned in
    And the scanner is taken offline
    When the valid ticket is presented
    And the scan button is pressed
    Then the scanner permits entry
    When the Wi-Fi connection is enabled
    Then the scanner is in online mode
    When the entry/exit button is clicked
    Then the scanner is in exit mode
    When the valid ticket is presented
    And the scan button is pressed
    Then the scanner denies exit because ticket has not entered

Using Calabash with our local changes, we have been able to fully automate our functional tests and keep the JAC Scanner application on track. So where are we going from here? There are always more challenges!

  • As our application becomes more complex, we will likely run into situations difficult to handle in Calabash. If so, we expect to extend Calabash as necessary to meet our needs. We will add these features to our local fork, and if we come up with anything generally useful, we will submit those changes back to the Calabash project.
  • If we do decide to port this application to iOS eventually, we may need to make similar changes to Calabash-iOS. Calabash-iOS has a slightly different architecture than Calabash Android, so this will be an opportunity to learn new things.
  • We hope to use this same system to automate user-acceptance tests and end-of-sprint demos. The challenge here is to make the tool simpler to use so that our Product and QA people will be able to use it more easily.
  • We are working towards a goal of attaching one of each of our primary target devices to a test machine and running the functional tests automatically after a build. Therefore we will be looking for ways to display the barcodes under control of Jenkins.

There is little danger that we will get bored any time soon!