5 tips to automate your tests at scale

A huge part of our community is now using test automation as part of their testing activities and we have a growing demand for advices and best practices. First, let me say there is no best practices, there are good practices that can help you when building a test automation process that scales. Here is a round up of the discussions and outcomes.

 #1 Test automation is not about ROI but about velocity

If you think of automation as no more manual work to do, you’ll be disappointed. Using test automation is expensive and requires a lot of work and maintenance. And the reason why we may use it and have a pay back is velocity. Automation is about speed of feedback. It is not about cost and it is an illusion that automation means $0.

Parkeon, one of our customer and leader in the smart cities industry, has developed a huge system (more than 300 man/years) to manage ticketing and transportation for a big city. They have adopted the agile practices and built thousands of automated tests using BDD. Part of the tests were executed at the API level, part at the GUI level and a few of them on real device with a robot to give the input.

Robot typing on keyboard

As you can imagine, building such an infrastructure was expensive. But it was worth the investment because the value they were looking for was speed of feedback. They were able to detect unanticipated impacts due to changes in the code very quickly. That enables the team to iterate faster, add new or modify features with confidence.

#2 Be careful with record and playback tools

You use test automation and try to scale your practice with record & playback tools? Then this is how your practice probably looks like:

Automation at scale

You push the snow in front of you and the more you move forward the bigger the pile of snow is. At some point you will not be able to move anymore. That is the maintenance problem all teams using record & playback face.

I am sorry to say the ugly truth. There is no magic and the promise “you can automate without any technical skills” is a dead end. Record and playback can be useful as a quick start strategy but then you should definitely modify and maintain the scripts directly at the code level.

#3 Make sure you have the skills

As a consequence if you want to start using test automation, first make sure you have the right skills in your team. Basically you should only work on test automation if you have the ability to work on the production code, either you call these persons test automation engineers, developers or SDETs (Software developer engineer in test). And the code should live in the same repository as your application.

When you automate a test you have to decide if the behavior should be tested at the GUI level, API level or if there are redundancies with the unit tests. So there is a real discussion that has to happen between developers and testers. I remember a discussion I have had a couple of months ago with a test leader that wanted to get started with automation. I was asking a couple of questions about the stack used by their development team. His answer: “I don’t know they are in India and I have no idea about the language and frameworks they use…”. No collaboration with the developers means a really really bad starting point.

#4 Every Test should tell a story about your application

One of the most popular question we get is: which tests should I automate? And my first answer is always the same: why have you created this script? It doesn’t matter if the execution is manual or automated. Tell me what is the intent of this test! Guess what? Most of time the answer is not clear. So people are talking about automation, thus making a big investment and the “why” is not clear.

So make sure your tests are readable and with a clear intent. Every single test should tell a story about the application. This is where I have found BDD a really great approach. Your tests/examples are both the living specification of your software and the acceptance tests for the developers: 2 sides of the same coin.

Once you’ve made clear why you have a test and the value of it then you can start thinking about the automation.

Of course if you do continuous delivery & deployment like we do, you should try to automate all your scripts (when possible) because you need speed of feedback.

#5 Build your test framework into layers

Last, if you want your test framework to scale then use a multiple layered approach.

Multiple layered approach

At the business layer level you define your test using a consistant business terminology (steps). This is where auto-completion features and refactoring are really key when designing a test. Make sure you don’t end up with “Sign in” step and “Login” step that have exactly the same meaning but written in 2 different ways. That will make the automation much more difficult.

Once you have your tests with a good design, readable and with a clear intent, it is time to implement the steps (test fixtures). There are many patterns you can follow for the implementation like Page Object or Journey pattern.

Having a framework with different layers will enable you to limit the impact when a change occurs and will make your maintenance much easier.

Conclusion: if you really want your test automation framework to scale there is no shortcut, you’ll need: hard work, technical skill and collaboration between developers and testers.

10 thoughts on “5 tips to automate your tests at scale

    1. Hi George. Not this one 🙂 The Robot used in that project was developed in house and driven by an arduino. There is a video about it but I’am not sure whether it is public or not. I’ll check if available on youtube and add the link


  1. I don’t agree with #2. Ever heard of TestComplete? It’s keyword scripting capabilities are good and for advanced actions which cannot be done by keyword driven testing, you can switch (or fallback, if you like) to scripting (several scripting languages supported). So you can keep a basis of KDT and switch over to scripting in necessary parts, and switch back to KDT to keep the right level op abstraction as much as possible.


    1. The main statement here is: test automation requires dev skills. I know TestComplete, Sahi, SeleniumIDE…
      And I agree with the use case you describe here. It doesn’t rely only on capture and replay.


  2. “…code should live in the same repository as your application.” – Let’s say we have multiple teams using different languages for development. Do you suggest having automation framework in different languages? What about maintainability and reusability. IMO, UI/API tests can be written in any language irrespective of the development stack. Of course, I would think, pick Java if it’s predominantly Java shop. Any comments?


  3. Hi Jo,

    That’s a good question and it mainly depends on how the team is organized. Let’s say for example your architecture relies on micro-services (using each its own language).

    If your tests concerns a specific micro-service, then you should have the dedicated tests exported in the repository handling the code for this specific micro-service in the same language than the micro-service (and the other tests for the other micro-service shoudl also be exported to the correct repo, obviously).
    If your tests concern the main application (for example E2E tests), then they would be exported to the repository of the main app, once again using the same language.

    Now another case, you have the same app in multiple languages and tests that should be shared (one example would be for example several libraries for an API to support multiple languages). In that case, I would have a different export in each language (that’s basically what I do with the coffee machine project, where we have a repo for each language/framework supported by hiptest-publisher).

    I hope this answers your question 🙂


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s