A huge part of our community is now using test automation as part of their testing activities and we have a growing demand for advices and best practices. First, let me say there is no best practices, there are good practices that can help you when building a test automation process that scales. Here is a round up of the discussions and outcomes.
#1 Test automation is not about ROI but about velocity
If you think of automation as no more manual work to do, you’ll be disappointed. Using test automation is expensive and requires a lot of work and maintenance. And the reason why we may use it and have a pay back is velocity. Automation is about speed of feedback. It is not about cost and it is an illusion that automation means $0.
Parkeon, one of our customer and leader in the smart cities industry, has developed a huge system (more than 300 man/years) to manage ticketing and transportation for a big city. They have adopted the agile practices and built thousands of automated tests using BDD. Part of the tests were executed at the API level, part at the GUI level and a few of them on real device with a robot to give the input.
As you can imagine, building such an infrastructure was expensive. But it was worth the investment because the value they were looking for was speed of feedback. They were able to detect unanticipated impacts due to changes in the code very quickly. That enables the team to iterate faster, add new or modify features with confidence.
#2 Be careful with record and playback tools
You use test automation and try to scale your practice with record & playback tools? Then this is how your practice probably looks like:
You push the snow in front of you and the more you move forward the bigger the pile of snow is. At some point you will not be able to move anymore. That is the maintenance problem all teams using record & playback face.
I am sorry to say the ugly truth. There is no magic and the promise “you can automate without any technical skills” is a dead end. Record and playback can be useful as a quick start strategy but then you should definitely modify and maintain the scripts directly at the code level.
#3 Make sure you have the skills
As a consequence if you want to start using test automation, first make sure you have the right skills in your team. Basically you should only work on test automation if you have the ability to work on the production code, either you call these persons test automation engineers, developers or SDETs (Software developer engineer in test). And the code should live in the same repository as your application.
When you automate a test you have to decide if the behavior should be tested at the GUI level, API level or if there are redundancies with the unit tests. So there is a real discussion that has to happen between developers and testers. I remember a discussion I have had a couple of months ago with a test leader that wanted to get started with automation. I was asking a couple of questions about the stack used by their development team. His answer: “I don’t know they are in India and I have no idea about the language and frameworks they use…”. No collaboration with the developers means a really really bad starting point.
#4 Every Test should tell a story about your application
One of the most popular question we get is: which tests should I automate? And my first answer is always the same: why have you created this script? It doesn’t matter if the execution is manual or automated. Tell me what is the intent of this test! Guess what? Most of time the answer is not clear. So people are talking about automation, thus making a big investment and the “why” is not clear.
So make sure your tests are readable and with a clear intent. Every single test should tell a story about the application. This is where I have found BDD a really great approach. Your tests/examples are both the living specification of your software and the acceptance tests for the developers: 2 sides of the same coin.
Once you’ve made clear why you have a test and the value of it then you can start thinking about the automation.
Of course if you do continuous delivery & deployment like we do, you should try to automate all your scripts (when possible) because you need speed of feedback.
#5 Build your test framework into layers
Last, if you want your test framework to scale then use a multiple layered approach.
At the business layer level you define your test using a consistant business terminology (steps). This is where auto-completion features and refactoring are really key when designing a test. Make sure you don’t end up with “Sign in” step and “Login” step that have exactly the same meaning but written in 2 different ways. That will make the automation much more difficult.
Once you have your tests with a good design, readable and with a clear intent, it is time to implement the steps (test fixtures). There are many patterns you can follow for the implementation like Page Object or Journey pattern.
Having a framework with different layers will enable you to limit the impact when a change occurs and will make your maintenance much easier.
Conclusion: if you really want your test automation framework to scale there is no shortcut, you’ll need: hard work, technical skill and collaboration between developers and testers.