Tag Archives: test automation

Excellent Resources on Test Automation and My Learning Points

Experiences of Test Automation Case Studies of Software Test Automation by Dorothy Graham and Mark Fewster
Image courtesy of Amazon

I like to think about Test Automation as providing tools that can execute important checks that might be tedious, time consuming, or there might not be enough time to manually perform those tests. Instead of performing these repetitive, time-consuming tests, the Testers can spend that time exploring risk areas where using their intelligence to make decisions on tests to perform is a better use of their time. (And a lot more fun than performing repetitive tests!)

Test Automation is not a trivial undertaking – research should be conducted to develop a strategy, identify tools, and determine the best approach to automate tests that are being performed manually. Test automation is not a silver bullet and it takes time to develop your approach over time by what you learned.

Do you want to learn more about Test Automation and what has worked and not worked so great at other companies? An excellent book to read that I highly recommend: Experiences of Test Automation Case Studies of Software Test Automation by Dorothy Graham and Mark Fewster. This book contains 29 case studies written by those involved with a test automation project. Each case study begins with a background including if the project was successful and if test automation is still being used at that organization. Consider starting a book club where everyone reads a different chapter(s), shares what they learned, and develop a strategic plan for your company on how you might approach test automation or improve your existing approach. (And consider adding food as part of your book club, food always make any event more fun!)

Test Huddle has a series of Test Automation webinars: Click here. As part of Test Huddle, Dorothy Graham has a wonderful webinar “Blunders in Test Automation” you can view by clicking here.

Key Take Aways:

  1. Test automation is not necessarily performed by Testers. It is performed by the person best qualified to use the tool selected that can include the ability to write code!
  2. Test automation does NOT replace Testers but should supplement your testing strategy.
  3. Test automation is basically maintaining a code base to test your Product’s code base.

Below are learning points; however the above resources provide a wealth of information that is not summarized within this posting.

Strategy:

  1. Make it a real project with dedicated people, time, and funding. Start with a small pilot with key objectives and gain buy in for your pilot by key stakeholders.
  2. Start with simple UI tests or smoke tests as a great way to get some initial protection around the code.
  3. Consider test automation strategy by: technology facing tests (unit, components, integration) and business facing tests (acceptance, business, GUI).
  4. Concentrate on testing 1 system first as a model – the system producing the most business (start small, focus on a single but important part of the system).
  5. It is expensive to automate all types of tests including edge cases. Identify what tests need to be in the automation suite. Think about the value of the test and the question it answers or information provided. What is the ROI?
  6. Tool training or employees who have experience with the tool is important to reduce expensive mistakes.
  7. Can Testers write the GUI tests and the Developers write the unit tests? Who will write the business layer tests? That depends on the tool selected and skills of the team.
  8. The developer may write the tests with the Testers running them.
  9. Be sure to include business people when selecting the business layer tests.

Tools & Approach:

  1. New code is often easier to automate than legacy code.
  2. Different tools / programming languages can be used for the test automation pyramid: Unit, acceptance tests, business layer, GUI.
  3. Test automation is good for mature code that is not constantly changing.
  4. Lean regression suites help keep maintenance costs low while still providing value.
  5. Be careful how you write your tests – reduce failures that really are not a problem.
  6. All tests running green do not mean there aren’t any problems.
  7. Once tests are automated, train Developers, Testers to understand how to maintain them.
  8. Determine if the tool needs to clean up after the test was executed.
  9. Ability to run the code across platforms, browsers, releases.

Managing Test Suites:

  1. Different type of test suites: nightly test suite, weekly test suite, candidate test suite.
  2. Keep the tests small and maintain the flexibility to run tests in logical test suites to reduce the time it takes to execute them. For example:
    1. you may need to execute some key tests during the day when time is running low.
    2. you may execute core tests with every new build for critical functionality.

Support:

  1. Strong management support is necessary to understand the cost of the pilot program and long-term support that includes ongoing maintenance of modifying and adding automated tests.
  2. Strong collaboration with Development is important – code needs to be written in a manner that it can have automated tests written.
  3. Make the benefits of test automation visible to stakeholders for continued support.
  4. Ideally Developers / Testers are dedicated to test automation to ensure that new and changed code is being reviewed to determine tests to add or tests to change.
  5. Identify different champions: tool champion, change agent, stakeholders, etc.

Metrics:

  1. Count number of tests for each test tool – trend over time to see how the tests increase in number.
  2. Send business users a calendar of green vs red if any of the tests are red.
  3. Build time is a critical metric since it includes the time to run the automated tests.
  4. On what O/S were more bugs found and frequency?
  5. Detection rate of bugs for tests. What tests found the most bugs? No bugs? Is the percentage of green tests trending upwards over time?

My Rewarding Experience as a Reviewer and Contributor to “More Agile Testing: Learning Journeys for the Whole Team” by Janet Gregory and Lisa Crispin

More Agile Testing
Picture courtesy of Amazon and where you can order this excellent book!

For many years I have written articles for the Testing Community and shared them through Testing Circus and Tea-Time with Testers. One warm summer day in July 2013, I received a pleasant email from Lisa Crispin asking if I would like to be one of the reviewers and contributors to their upcoming book that would compliment their existing book “Agile Testing: A Practical Guide for Testers and Agile Teams”. It was an honor to be asked and of course my response confirmed my interest. An international review team was brought together comprising of people with different experiences and expertise in their fields.

This was a rewarding experience both professionally and personally as there were many learning opportunities reviewing the chapters and discussions with the team. Checking my email was fun since there was a potential for an email from Lisa and Janet announcing what was ready for review or from a fellow reviewer with a question to foster a lively discussion. New chapters were quickly printed for when there would be time to review, digest, and comment on them. Lisa and Janet worked with the reviewers to help us identify where we could contribute sidebars to their new book. It was fun reading what the other reviewers were writing based upon their experience. For me, this is what really makes this book come alive – Lisa and Janet are sharing a lot of valuable information that is supplemented by real-life experiences. As an example I enjoyed reading how other Testers are using Session-Based Testing and Thread-Based Testing to meet the challenges of testing in a fast-paced environment!

It was a great opportunity to witness how a book comes together, the chapters under review might not be named nor are they in the order of the final publication. As the book evolved, chapters may be combined or information moved to a different chapter. After reviewing each chapter it was interesting to read the book cover to cover. It reminds me of a movie production where scenes are filmed out of order and then brought together to tell the story.

Once the draft book was completed, there was a great sense of accomplishment and a feeling of disappointment. Being part of a community working towards a common goal is fulfilling and exciting. It reminds me of working with a great project team. When the project is done there is pride of the work completed but yet a sadness that the project is completed.

The information and experiences within this book series is part of the overall testing picture. As an artist blends together different colors and uses different tools to create a beautiful piece of artwork, we do the same in Software Testing. We blend together different approaches and techniques to design and execute testing strategies to tackle complex testing problems.

As with the first Agile book, there are great learning opportunities for you to determine how to apply the information to further your testing career and tackle testing problems. As an example I love the “Pillars of Testing” model by David Evans designed with different factors that integrate to improve our confidence in the information we provide about the Quality of the Product we test. This model helped me think through the different factors to understand our strengths and weaknesses in each area of this model. Adam Knight’s discussion on “T-Shaped Skills” inspired me to think differently about training by identifying the “deeper” skills required for different flavors of technical testing and Product testing. At this time, my background includes limited automated testing experience; however, this book provides a foundation for me to better understand both the benefits and challenges to have a reasonable conversation. The chapter on “Thinking Skills for Testing” is excellent and in particular I like how Sharon Robson discussed that you need to use different type of thinking based upon the problem such as when to use critical thinking, analytical thinking, and creative thinking. Plus sample tools are suggested that can be researched further.

Lastly and most importantly, I would like to thank Janet and Lisa for this wonderful opportunity and to their continual dedication to the Testing Community. My recommendation for my readers is that you will add this book to your testing toolbox and discover how it will influence your testing world!