Tag Archives: Test Huddle

Excellent Resources on Test Automation and My Learning Points

Experiences of Test Automation Case Studies of Software Test Automation by Dorothy Graham and Mark Fewster
Image courtesy of Amazon

I like to think about Test Automation as providing tools that can execute important checks that might be tedious, time consuming, or there might not be enough time to manually perform those tests. Instead of performing these repetitive, time-consuming tests, the Testers can spend that time exploring risk areas where using their intelligence to make decisions on tests to perform is a better use of their time. (And a lot more fun than performing repetitive tests!)

Test Automation is not a trivial undertaking – research should be conducted to develop a strategy, identify tools, and determine the best approach to automate tests that are being performed manually. Test automation is not a silver bullet and it takes time to develop your approach over time by what you learned.

Do you want to learn more about Test Automation and what has worked and not worked so great at other companies? An excellent book to read that I highly recommend: Experiences of Test Automation Case Studies of Software Test Automation by Dorothy Graham and Mark Fewster. This book contains 29 case studies written by those involved with a test automation project. Each case study begins with a background including if the project was successful and if test automation is still being used at that organization. Consider starting a book club where everyone reads a different chapter(s), shares what they learned, and develop a strategic plan for your company on how you might approach test automation or improve your existing approach. (And consider adding food as part of your book club, food always make any event more fun!)

Test Huddle has a series of Test Automation webinars: Click here. As part of Test Huddle, Dorothy Graham has a wonderful webinar “Blunders in Test Automation” you can view by clicking here.

Key Take Aways:

  1. Test automation is not necessarily performed by Testers. It is performed by the person best qualified to use the tool selected that can include the ability to write code!
  2. Test automation does NOT replace Testers but should supplement your testing strategy.
  3. Test automation is basically maintaining a code base to test your Product’s code base.

Below are learning points; however the above resources provide a wealth of information that is not summarized within this posting.

Strategy:

  1. Make it a real project with dedicated people, time, and funding. Start with a small pilot with key objectives and gain buy in for your pilot by key stakeholders.
  2. Start with simple UI tests or smoke tests as a great way to get some initial protection around the code.
  3. Consider test automation strategy by: technology facing tests (unit, components, integration) and business facing tests (acceptance, business, GUI).
  4. Concentrate on testing 1 system first as a model – the system producing the most business (start small, focus on a single but important part of the system).
  5. It is expensive to automate all types of tests including edge cases. Identify what tests need to be in the automation suite. Think about the value of the test and the question it answers or information provided. What is the ROI?
  6. Tool training or employees who have experience with the tool is important to reduce expensive mistakes.
  7. Can Testers write the GUI tests and the Developers write the unit tests? Who will write the business layer tests? That depends on the tool selected and skills of the team.
  8. The developer may write the tests with the Testers running them.
  9. Be sure to include business people when selecting the business layer tests.

Tools & Approach:

  1. New code is often easier to automate than legacy code.
  2. Different tools / programming languages can be used for the test automation pyramid: Unit, acceptance tests, business layer, GUI.
  3. Test automation is good for mature code that is not constantly changing.
  4. Lean regression suites help keep maintenance costs low while still providing value.
  5. Be careful how you write your tests – reduce failures that really are not a problem.
  6. All tests running green do not mean there aren’t any problems.
  7. Once tests are automated, train Developers, Testers to understand how to maintain them.
  8. Determine if the tool needs to clean up after the test was executed.
  9. Ability to run the code across platforms, browsers, releases.

Managing Test Suites:

  1. Different type of test suites: nightly test suite, weekly test suite, candidate test suite.
  2. Keep the tests small and maintain the flexibility to run tests in logical test suites to reduce the time it takes to execute them. For example:
    1. you may need to execute some key tests during the day when time is running low.
    2. you may execute core tests with every new build for critical functionality.

Support:

  1. Strong management support is necessary to understand the cost of the pilot program and long-term support that includes ongoing maintenance of modifying and adding automated tests.
  2. Strong collaboration with Development is important – code needs to be written in a manner that it can have automated tests written.
  3. Make the benefits of test automation visible to stakeholders for continued support.
  4. Ideally Developers / Testers are dedicated to test automation to ensure that new and changed code is being reviewed to determine tests to add or tests to change.
  5. Identify different champions: tool champion, change agent, stakeholders, etc.

Metrics:

  1. Count number of tests for each test tool – trend over time to see how the tests increase in number.
  2. Send business users a calendar of green vs red if any of the tests are red.
  3. Build time is a critical metric since it includes the time to run the automated tests.
  4. On what O/S were more bugs found and frequency?
  5. Detection rate of bugs for tests. What tests found the most bugs? No bugs? Is the percentage of green tests trending upwards over time?