Hardware Testing with Jenkins & TAP

At Adafruit, when we push or merge changes to one of our hundreds of Arduino libraries on GitHub, we have to ask ourselves “Does this change work?”. In a lot of cases, the best answer we can give, even after extensive manual testing, is “Maybe?”. For Adafruit IO, it is important that we are able to easily answer “Yes.” to that question. A large portion of Adafruit IO users could be impacted by a bug in a library such as the Adafruit IO Arduino Library, and we want to make sure our client code is reliable.

The hard truth is that there aren’t enough hours in the day to test each change to every library on all of the supported platforms, so most of the time we have to settle with testing one or two platforms manually, and hoping for the best on the rest of the supported platforms.

What do we mean by platform? In this case, we are talking about the multiple microcontrollers that are supported in the Arduino IDE via the Arduino 1.6.x Third Party Hardware Specification. Arduino Uno, ESP8266, Feather M0 (ATSAMD21), and the Feather WICED (STM32F205) are all good examples different platforms that require separate testing.

First Attempt: Build Verification using Travis CI

We had previously attempted automated build checking using Travis CI, and that method works great for checking if sketches build on multiple platforms.

As you may have guessed, this leaves us with a huge blind spot. A passing build does not mean the code will do what you expect it to do.

Second Attempt: Manual Compatibility Testing

Our next attempt involved building a ‘compatibility matrix’ web app that allowed for easy logging of the results of manual tests on supported platforms. The app pushes test results to a GitHub repo as JSON (in case we need to access it programatically). It also automatically adds/updates a compatibility table in the README of the target library.

Here’s an example of the README output:

This method is very effective for answering “Does this change work?”, but it is very impractical to manually test the daily changes to libraries on every supported platform.

Latest Attempt: Running Tests on Hardware

Our latest approach to this problem uses Jenkins CI running on an AWS EC2 instance, and small fleet of local Raspberry Pi test nodes. Each node uploads and runs unit & integration tests on hardware attached to USB ports.

Each Raspberry Pi can support multiple platforms, and each platform can be targeted by the test. The tests themselves output results using the TAP protocol over a serial connection, and the Raspberry Pi sends the results back to Jenkins so we are aware when builds fail.

Here’s a visual overview of how it works:

Test status can be monitored in real time from anywhere, and we are able to view the raw console output from the test nodes when needed.

TAP results are also parsed and attached to each build:

We will be sharing the details of this in a tutorial soon, but if you would like to take a look at our early experiments with Jenkins, head to jenkins.adafruit.com or check out our Jenkins Pipeline Library on GitHub.