OpenJDK Test Fest (23rd March 2013)

Better late than never, I'm writing about my very first hack-day working on OpenJDK testing :)

Introduction

After LJC Conf 2012 I got interested in the Adopt OpenJDK project. Sadly it was only in early 2013 when I finally managed to get fully set up for the project. (Building Java has gotten a lot easier!)

A big issue for the OpenJDK project is that currently there are little to none open tests. Whilst Oracle and others have many prioritory tests for the JDK, they are unlikely to be released anytime soon. Hence there is a big push to improve the quality of exisiting tests and increase test coverage.

What I hope to cover in this post is:

1. Getting started testing OpenJDK (using JTReg)
2. Guidance on testing OpenJDK
3. Next steps (how you can get involved!)

Getting Started - Seed project

OpenJDK tests are primarily written using a tool called JTReg. The reason behind this that the very first tests pre-date JUnit and TestNG. Sadly this does make writing OpenJDK tests a bit of a pain...

To get started easily, I recommend you utilise the seed project I set up here:
https://github.com/arkangelofkaos/OpenJdkTesting

Within it you will find a copy of JTReg and a few very simple tests to show you the basics. I highly recommend you go through the JTReg tutorial (available here).

Guidance

During the hack-day we were given a list of "Ten Golden Rules" for testing on OpenJDK, here is my take on them:

1. Think before you code! 
• Have a specific goal, target specific methods.
2. Make tests understandable
• Fill in the summary tag
• Be clear and make your test goal explicit
3. Make tests "small and simple"
• Use setup and teardown as necessary.
• Keep tests idempotent. 
• Not cleaning up is not only impolite but can cause errors for other test cases!
4. Test only one thing!
• 1 assert per test = easy to understand and simple :)
5. Fast tests only! 
• Quick tests required as they will be run over and over.
6. Absolute repeatability
• Tests ONLY fail if there is a bug in the code under test.
• One pitfall is using Thread.sleep to handle concurrency issues
• Understand what your goal is (draw it out!)
• Use proper concurrency tools to enforce an execution order (e.g. countdown latches)
7. Independent tests
• Tests should not rely on other tests or be affected by execution order.
8. Provide diagnostic data on failure
• Use messages on assertions and throw sensible exceptions.
9. Ensure portability
• Do not hardcode environment details into the test.
10. Silence is golden
• Only speak up if the test is broken. Don't pollute the output. 
• However, sometimes a little additional output is very helpful for debugging.

As always with any software rules, the rules depend on the circumstance. Sometimes you will need to break these rules but it is important we understand why!

Conclusion

Java needs our help to stay relevant and increase the pace of upgrades. This means testing needs to improve and we can contribute to this through OpenJDK. If you are interested, I highly recommend you grab the seed project, join up and give it a go!