vendredi 26 juin 2015

Living test documentation

I’ve read these slides by Cyril Martraire. So I got pedantic (a custom on Twitter) and wrote:

So what am I talking about? For now, I’ll take the example of testing because it was quick and pleasant to do.

This particular project was to build load and transform processes to store call data reports in a data warehouse. The transform processes where written in PLSQL as they were embedded in an Oracle database.

They were pure batch processing so the testing could be easily automated. That’s what we did, using the following approach:

  • Tests are implemented in Java, because it’s a widely known language, accessing Oracle DB is easy with OJDBC, and testing environment is easily setup thanks to Maven (if you don’t believe me, try to connect Oracle with PHP on Linux…).
  • The whole context is inserted in the database on test setup. The conext and the results are deleted on test teardown.
  • The stored procedures are launched in the test cases and assertions are run on the results retrieved with SQL queries.

Unfortunately, we did not propose these test suites as a deliverable to our client. Rather than that, we committed to provide a (manual) test result report, extracted from Quality Center. Boring. So the idea was: “why not generate test steps descriptions so that they can be run manually”.

This was easy because this kind of test were made of a repeatable pattern:

Given a GPRS call report
When the field XXX has value YYY and ZZZ field has value ABC
Then the corresponding field in table BBB has value CCC

So I associate the patterns with methods to which I pass the test input and outcome has parameters:

public void checkBBBResultsDependingOnXXXandZZZ(YYY, ABC, CCC) { … }

The test file was clean. The test where automated and it was easy to parse the source code to translate the tests. I did that in two parts.

First, I extract the delete and insert queries to put them in an initialisation part of my documented test case.

Then I parse my test class to extract calls to method checkBBBResultsDependingOnXXXandZZZ and their parameters to translate them in non developer language (French in that situation).

Translation scripts where made in Python. They were stored in VCS along with the Java test cases. They produce a result in plain text that I used to copy and paste in quality center. Then I manually declared that the whole campaigns were passing. The software is smart enough to generate a test report as a MS Word file from the campaigns results.


  • I think I could have automated the insertion in quality center. The API looked complex though and wanted to be able to check what I was doing step by step in the copy and paste sessiosn.
  • I’m pretty sure the client did not look at the tests afterward (who could read these boring reports anyway?). They were just in case to prove that we did test the application, and to be able to reproduce the test suits if needed. Therfore I’m glad not to have spent to much time on it.


People talk often about BDD, using Gherkin syntax and so on. Sometimes its improductive, because you have no access to the business people, you add a translation layer that is a potential source of error and that is long to setup.

Generate your QA test cases from the souce code is the opposite and can be an alternative if you have no real access to the business, if the business cases are simple to explain and to automate. Finally, it allows the busniess prople to check that our implementation is correct with a language they can understand.