Case Study: Saving Remote Test Results in SDK

Goal -

We needed a way to save test results for future analysis + we wanted to explore our own API library while away on a recent team PBH (product building holiday, of course...)

Background -

In the past, we've been able to see 'real time' test outcomes via terminal, but there is no way to go back in time to see test results and determine: which ones passed? which ones failed? when was the last time a test passed? how have test results changed over time? etc.

PBH Solution -

Using a generic dataset, we were able to successfully run a feature test and confirm the data saves to Secure Data Kit (SDK), therefore creating a 'test outcome log'.

Here's how it works:

And here's why it's awesome:

1. With this tool, we took advantage of the SDK API by adapting a separate test suite for a completely different project and saving its results to an SDK dataset. By doing so, it shows how any app using any language can do the same thing - essentially, we can remotely execute tests in whatever framework we want and remotely add the results to an SDK dataset!
2. Another advantage to this is we can report on 'pass/fail rates' or 'date last passed' for all of our projects, plugins, clients, etc. in a single dataset, which allows us to gather insights over a period of time (no matter the framework). Because code bases change over time and tests can start failing, if unmaintained, a log of test results is incredibly valuable for analysis + understanding the clear picture.

Sep. 14 2018