Excellent Resources on Test Automation and My Learning Points

Experiences of Test Automation Case Studies of Software Test Automation by Dorothy Graham and Mark Fewster
Image courtesy of Amazon

I like to think about Test Automation as providing tools that can execute important checks that might be tedious, time consuming, or there might not be enough time to manually perform those tests. Instead of performing these repetitive, time-consuming tests, the Testers can spend that time exploring risk areas where using their intelligence to make decisions on tests to perform is a better use of their time. (And a lot more fun than performing repetitive tests!)

Test Automation is not a trivial undertaking – research should be conducted to develop a strategy, identify tools, and determine the best approach to automate tests that are being performed manually. Test automation is not a silver bullet and it takes time to develop your approach over time by what you learned.

Do you want to learn more about Test Automation and what has worked and not worked so great at other companies? An excellent book to read that I highly recommend: Experiences of Test Automation Case Studies of Software Test Automation by Dorothy Graham and Mark Fewster. This book contains 29 case studies written by those involved with a test automation project. Each case study begins with a background including if the project was successful and if test automation is still being used at that organization. Consider starting a book club where everyone reads a different chapter(s), shares what they learned, and develop a strategic plan for your company on how you might approach test automation or improve your existing approach. (And consider adding food as part of your book club, food always make any event more fun!)

Test Huddle has a series of Test Automation webinars: Click here. As part of Test Huddle, Dorothy Graham has a wonderful webinar “Blunders in Test Automation” you can view by clicking here.

Key Take Aways:

  1. Test automation is not necessarily performed by Testers. It is performed by the person best qualified to use the tool selected that can include the ability to write code!
  2. Test automation does NOT replace Testers but should supplement your testing strategy.
  3. Test automation is basically maintaining a code base to test your Product’s code base.

Below are learning points; however the above resources provide a wealth of information that is not summarized within this posting.

Strategy:

  1. Make it a real project with dedicated people, time, and funding. Start with a small pilot with key objectives and gain buy in for your pilot by key stakeholders.
  2. Start with simple UI tests or smoke tests as a great way to get some initial protection around the code.
  3. Consider test automation strategy by: technology facing tests (unit, components, integration) and business facing tests (acceptance, business, GUI).
  4. Concentrate on testing 1 system first as a model – the system producing the most business (start small, focus on a single but important part of the system).
  5. It is expensive to automate all types of tests including edge cases. Identify what tests need to be in the automation suite. Think about the value of the test and the question it answers or information provided. What is the ROI?
  6. Tool training or employees who have experience with the tool is important to reduce expensive mistakes.
  7. Can Testers write the GUI tests and the Developers write the unit tests? Who will write the business layer tests? That depends on the tool selected and skills of the team.
  8. The developer may write the tests with the Testers running them.
  9. Be sure to include business people when selecting the business layer tests.

Tools & Approach:

  1. New code is often easier to automate than legacy code.
  2. Different tools / programming languages can be used for the test automation pyramid: Unit, acceptance tests, business layer, GUI.
  3. Test automation is good for mature code that is not constantly changing.
  4. Lean regression suites help keep maintenance costs low while still providing value.
  5. Be careful how you write your tests – reduce failures that really are not a problem.
  6. All tests running green do not mean there aren’t any problems.
  7. Once tests are automated, train Developers, Testers to understand how to maintain them.
  8. Determine if the tool needs to clean up after the test was executed.
  9. Ability to run the code across platforms, browsers, releases.

Managing Test Suites:

  1. Different type of test suites: nightly test suite, weekly test suite, candidate test suite.
  2. Keep the tests small and maintain the flexibility to run tests in logical test suites to reduce the time it takes to execute them. For example:
    1. you may need to execute some key tests during the day when time is running low.
    2. you may execute core tests with every new build for critical functionality.

Support:

  1. Strong management support is necessary to understand the cost of the pilot program and long-term support that includes ongoing maintenance of modifying and adding automated tests.
  2. Strong collaboration with Development is important – code needs to be written in a manner that it can have automated tests written.
  3. Make the benefits of test automation visible to stakeholders for continued support.
  4. Ideally Developers / Testers are dedicated to test automation to ensure that new and changed code is being reviewed to determine tests to add or tests to change.
  5. Identify different champions: tool champion, change agent, stakeholders, etc.

Metrics:

  1. Count number of tests for each test tool – trend over time to see how the tests increase in number.
  2. Send business users a calendar of green vs red if any of the tests are red.
  3. Build time is a critical metric since it includes the time to run the automated tests.
  4. On what O/S were more bugs found and frequency?
  5. Detection rate of bugs for tests. What tests found the most bugs? No bugs? Is the percentage of green tests trending upwards over time?

Please Participate in the Third Annual State of Testing Survey 2016

state of testing
It is that time again for the State of Testing Survey! The first year about 600 people participated and last year it grew to almost 900 Testers! I hope you were one of them. Can you take a few minutes to participate in this important survey? Click here to participate. The survey is sponsored jointly by QA Intelligence – Testing & QA Management blog by PractiTest and Tea-Time with Testers.

The survey seeks to identify the existing characteristics, practices and challenges facing the testing community in hopes to shed light and provoke a fruitful discussion towards improvement. If you are not sure if you want to participate, take a look at last year’s results by clicking here. A lot of valuable information on how other Testers are spending their time, how they plan, methodologies used, challenges faced and much more! I hope you will take some time to complete this valuable survey as I believe you will find the information helpful!

And can you do us a favor? Reblog this posting to let other people know about this survey. Or perhaps you can ask co-workers or friends to complete the survey. Thank you for your consideration.

Women in Testing Website is Live!

I wanted to let everyone know about a new website called: Women in Testing started by Jay Philips CEO & President of Project Realms, Inc. & TeamQualityPro.

If you are a woman in testing or if you know of any women in testing please let them know about this new website. It contains bios of women in testing; information on different events such as conferences and webinars; and a newsletter. Plus there is a Facebook page and a Twitter account – plus more is coming!

If you would like to be listed in the directory click here. I hope to see you listed on Women in Testing!

My Rewarding Experience as a Reviewer and Contributor to “More Agile Testing: Learning Journeys for the Whole Team” by Janet Gregory and Lisa Crispin

More Agile Testing
Picture courtesy of Amazon and where you can order this excellent book!

For many years I have written articles for the Testing Community and shared them through Testing Circus and Tea-Time with Testers. One warm summer day in July 2013, I received a pleasant email from Lisa Crispin asking if I would like to be one of the reviewers and contributors to their upcoming book that would compliment their existing book “Agile Testing: A Practical Guide for Testers and Agile Teams”. It was an honor to be asked and of course my response confirmed my interest. An international review team was brought together comprising of people with different experiences and expertise in their fields.

This was a rewarding experience both professionally and personally as there were many learning opportunities reviewing the chapters and discussions with the team. Checking my email was fun since there was a potential for an email from Lisa and Janet announcing what was ready for review or from a fellow reviewer with a question to foster a lively discussion. New chapters were quickly printed for when there would be time to review, digest, and comment on them. Lisa and Janet worked with the reviewers to help us identify where we could contribute sidebars to their new book. It was fun reading what the other reviewers were writing based upon their experience. For me, this is what really makes this book come alive – Lisa and Janet are sharing a lot of valuable information that is supplemented by real-life experiences. As an example I enjoyed reading how other Testers are using Session-Based Testing and Thread-Based Testing to meet the challenges of testing in a fast-paced environment!

It was a great opportunity to witness how a book comes together, the chapters under review might not be named nor are they in the order of the final publication. As the book evolved, chapters may be combined or information moved to a different chapter. After reviewing each chapter it was interesting to read the book cover to cover. It reminds me of a movie production where scenes are filmed out of order and then brought together to tell the story.

Once the draft book was completed, there was a great sense of accomplishment and a feeling of disappointment. Being part of a community working towards a common goal is fulfilling and exciting. It reminds me of working with a great project team. When the project is done there is pride of the work completed but yet a sadness that the project is completed.

The information and experiences within this book series is part of the overall testing picture. As an artist blends together different colors and uses different tools to create a beautiful piece of artwork, we do the same in Software Testing. We blend together different approaches and techniques to design and execute testing strategies to tackle complex testing problems.

As with the first Agile book, there are great learning opportunities for you to determine how to apply the information to further your testing career and tackle testing problems. As an example I love the “Pillars of Testing” model by David Evans designed with different factors that integrate to improve our confidence in the information we provide about the Quality of the Product we test. This model helped me think through the different factors to understand our strengths and weaknesses in each area of this model. Adam Knight’s discussion on “T-Shaped Skills” inspired me to think differently about training by identifying the “deeper” skills required for different flavors of technical testing and Product testing. At this time, my background includes limited automated testing experience; however, this book provides a foundation for me to better understand both the benefits and challenges to have a reasonable conversation. The chapter on “Thinking Skills for Testing” is excellent and in particular I like how Sharon Robson discussed that you need to use different type of thinking based upon the problem such as when to use critical thinking, analytical thinking, and creative thinking. Plus sample tools are suggested that can be researched further.

Lastly and most importantly, I would like to thank Janet and Lisa for this wonderful opportunity and to their continual dedication to the Testing Community. My recommendation for my readers is that you will add this book to your testing toolbox and discover how it will influence your testing world!

Mr. Spock and Star Trek’s Influence on my Life and as a Software Testing Manager

Mr Spock
Picture courtesy of the BBC News Entertainment & Arts.

I was saddened to hear of Leonard Nimoy passing at the age of 83. Trekkies knew he was sick and recently was in the hospital. Of course selfishly I wanted Leonard Nimoy to be part of the 50th anniversary Star Trek movie. It was wonderful to see him in the 2009 Star Trek movie. I loved the scene where he encounters a young Kirk – and greets him warmly with: I have been and always shall be your friend. Such profound and deep words if you really understand the friendship between Kirk and Spock. And Spock’s father who tells him that he married his mother (a human) because he loved her. Those are my favorite scenes from the newer movies – we are to witness some of the Vulcan’s emotions.

Mr. Spock was one of my favorites from Star Trek the original series. As a child I faithfully watched Star Trek in reruns and never got tired of watching it over and over again! The space travel and encountering new worlds and aliens was fascinating. Even today just hearing the Star Trek theme means an adventure awaits! The Enterprise is the most beautiful ship. My love of technology comes from those early days of Star Trek. Over the years we have witnessed how Star Trek has become reality through current technology. Look at their communicators and today we have mobile devices. In The Next Generation Picard uses a small flat screen computer – now we have iPads and Skype! Technology has dramatically changed over the decades. Earlier in my career computers were DOS based and printers were dot-matrix – they were loud and slow! My initial programming classes included writing and executing JCL through punched cards. Luckily that was only one semester! Click here for more information and the below picture is courtesy of wikipedia. But my education served me well as a programmer for 12 years and then many years in the testing field!

Punched card

During the 90’s computer magazines were filled with articles on how technology was changing. Of interest was the Chicago project that ended up being Windows 95. This earlier computer was expensive with a hard drive that was 1 gig. At that time it was considered to be a lot of storage! Those early days of computers with the Apple and Windows wars was exciting

Many nights started with the intention of spending an hour on the computer but ended up with late nights learning the new technology. At work my nickname was “geekette” as I helped my co-workers with the technology that was confusing for them. But for me it was a love affair! Technology was not as common as it is today with everyone having an iPad or a smart phone. Unlike the current generation, in kindergarten we did not have computers. Those of us who understood technology was a smaller group of people and many did not relate to us. But that was okay because we had Star Trek! Never underestimate the Trekkie bond!

Mr. Spock was interesting as he was half human and half Vulcan. He had the logic and self-control of the Vulcan’s but there were times when we witnessed his human emotions. When the Enterprise is under attack or there is a crisis, Spock is logical and in control. He can be heard to tell the crew that there is no reason to yell. Through the decades from TOS and the many movies we learn more about Spock and how he had to learn to keep his emotions under control and not let his human side take over.

I love this picture of the Enterprise and it is courtesy of Once Upon a Geek.
NCC Enterprise

There is an interesting parallel between Mr. Spock and my life as a Software Testing Manager. My love of logic and thinking through a situation by remaining neutral comes from Mr. Spock. Often I have been told I can appreciate both sides of a story and make decisions while remaining neutral and unemotional. I believe this is an important ability for a Testing Manager. We have to keep our emotions in control when the pressure of a pending release is high or we are juggling a lot of risks trying to determine the best use of time. It would be easy to cave into the human emotions. Instead we need to focus on the facts and test assumptions to make the best decisions. High emotions can cause us to have tunnel vision by looking at a narrow piece of the problem instead of trying to understand the overall situation.

My love for technology and logic comes from Star Trek and Mr. Spock. As I write this tribute I have a marathon of TOS playing on our Smart TV and one of my favorite two-part series in currently playing: The Menagerie. One day my husband and I went to Best Buy and I saw the Smart TV and well… had to have it! Some people are dedicated to one operating system whereas I use Windows, IOS, and Android. Charging multiple devices with different operating systems every day seems to be the norm.

Mr spock vulcan
Image courtesy of Fanpop.

Lastly, here is a video where Leonard Nimoy discusses the Jewish story behind Spock’s hand gesture that he brought into Star Trek. Leonard Nimoy will be missed but he left a wonderful legacy and influenced countless lives. But we still hate to part so soon!

What are your Mr. Spock memories?

Live long and Prosper!

Articles I Wrote that were Published through Tea-Time With Testers

books

Over the years, I had the pleasure of writing articles and several series for Tea-Time with Testers. My series included: How to Develop a Strategy for Cross-Browser Testing, Career Development and Learning Strategies for Testers and Introducing Session-Based Testing into your Testing department. Below are my articles in their original format. The attachments are in PDF format and you can read a lot of wonderful articles on Tea-Time with Testers.

2013 Articles

Introducing Session-Based Testing to Your Team Part 2 – December 2013 / January 2014
Click here to read
This article is focused on introducing session-based testing in your testing group after attending training.

Introducing Session-Based Testing to Your Team Part 1 – October 2013
Click here to read
This series will discuss how you can use session-based testing and tools like mind maps while providing suggestions on how to implement change within your testing team or department.

Prepare for Promotion Now! – Women in Testing Special September 2013
Click here to read
We often hear how the number of women moving up the corporate ladder becomes smaller at each rung. There are a lot of good conversations discussing the various reasons that factor into this problem. This article is based upon my experience on a few things women can do to let their voices be heard, have more influence, and prepare for senior level positions.

Using Mind Maps to Organize your Cross-Browser Testing Part 5 – February / March 2013
Click here to read
In this part of the series, I shared sample mind maps that I find useful in organizing my testing.

2012 Articles

Balancing Time with Cross-Browser Testing Part 4 – December 2012
Click here to read
In this part of the series, I shared sample mind maps that I find useful in organizing my testing.

Importance of Social Networking Part 3 – August 2012
Click here to read
In this article I shared information I gathered through the Testing Community on how to perform cross-browser testing.

Understanding Browser Compatibility Strategies Part 2 – July 2012
Click here to read
In this article I shared information I gathered through the Testing Community on how to perform cross-browser testing.

Understanding Browser Compatibility Strategies Part 1 – June 2012
Click here to read
In this article I shared information I gathered through the Testing Community on how to perform cross-browser testing.

Contributing to the Testing Community – May 2012
Click here to read
In this article I discussed the importance of building relationships within the Testing Community and contributing.

Training Someone New to Testing Part 2 – April 2012
Click here to read
This series focus on training someone who is new to the Testing field.

Training Someone New to Testing Part 1 – March 2012
Click here to read
This series focus on training someone who is new to the Testing field.

Facilitating a Journal Club – February 2012
Click here to read
In this article I provided suggestions on how to start to start a journal club to provide opportunities for Testers to discuss current trends.

Developing a Career Plan – January 2012
Click here to read
The focus of this article is developing a career plan with approaches that a manager can implement for his department and testers can adopt for their own professional development.

2011 Articles

Onboarding New Testers – November 2011
Click here to read
The focus of this article is onboarding newly hired testers into your department.

Articles I Wrote That Were Published in Testing Circus #Testing

publicationsOver the years, I had the pleasure of writing articles on testing, training, career development and other topics for Testing Circus. Below are my articles in their original format. Some of the articles I have revised and posted on The Testers Edge based upon what I have learned since that time or to take a different approach on the material. The attachments are PDF and you can find the magazines with a lot of great articles on Testing Circus.

2013 Articles

Are You Listening to Your Intuition? – 2013 September
Click here to read.
A lot has been written on how to progress your testing skills but an area often overlooked is listening to our intuition.

How Do I Manage My Time When There Is So Much To Do, 2nd part – 2013 April
Click here to read.
This article is the second part of “How Do I Manage My Time When there is so Much to Do?”

How Do I Manage My Time When There Is So Much To Do, 1st part – 2013 March
Click here to read.
With this article I addressed a few questions and challenges I face as a Software Testing Manager.

Who do I ask questions? – 2013 February
Click here to read.
In this article I discussed the importance of building relationships to help you identify the right person to ask questions and gather information.

How do I know when to stop testing – 2013 January
Click here to read.
In this article I addressed the often-asked question “how do I know when to stop testing?”

2012 Articles

How can I learn to Test – 2012 December
Click here to read.
In this article I addressed the question “how can I learn to test?”

Do you think I could be a tester, Part 2 – 2012 November
Click here to read.
In this article I addressed the question “do you think I could be a tester?”

Do you think I could be a tester, Part 1 – 2012 October
Click here to read.
In this article I addressed the question “do you think I could be a tester?”

Performing Risk Assessment Part 3 – 2012 September
Click here to read.
The last part of this series discusses a post-testing risk assessment or what is sometimes called lesson learned.

Performing Risk Assessment Part 2 – 2012 August
Click here to read.
The second part of this series is focused on monitoring initial risks and identifying new risks based upon what is learned during testing.

Performing Risk Assessment Part 1 – 2012 June
Click here to read.
The first part of this series is focused on identifying risks prior to testing.

Can We Approach Metrics Differently? Part 3 – 2012 May
Click here to read.
This part addressed defining a goal, metric setting, and guidelines for presenting information to senior management.

Can We Approach Metrics Differently? Part 2 – 2012 April
Click here to read.
This part addressed “The Five Whys” which is a fact-based approach to identifying the root cause of a problem.

Can We Approach Metrics Differently? Part 1 – 2012 March
Click here to read.
This part addressed providing a foundation in analyzing information to define your problems.

Four Questions to Consider When Writing a Test Report – 2012 February
Click here to read.
This article discussed guidelines for writing a test report.

What is a Tester’s Role in Quality Part 2 – 2012 January
Click here to read.
The second part of this article discussed how the processes and testing approaches adopted by a Testing Department and how they impact the product’s quality

2011 Articles

Guidelines on Writing a Meaningful Problem Report – 2011 November
Click here to read.
General guidelines that a tester can follow when researching and developing a problem report

You Want it When? Part 2 – 2011 October
Click here to read.
This part discussed approaches to prepare for testing when timelines have been reduced which may require the testing team to work extended hours

You Want it When? Part 1 – 2011 September
Click here to read.
This part discussed how to prepare for testing to hit the ground running.

Regression Risk-Based Testing – 2011 August
Click here to read.
This article discussed different approaches to assessing risk that can be used to develop both a modified- and core-regression strategy.

Breaking the Testing Mindsets – 2011 July
Click here to read.
This article discussed different ways to approach a testing problem.

A Test Matrix Approach for Organizing Testing – 2011 June
Click here to read.
This article discussed how to use a test matrix instead of test cases.

Planning Your Next Testing Assignments – 2011 May
Click here to read.
This article provided suggestions on planning a testing assignment. Translate the word “PLAN” into four actionable steps: Prepare, Layout, Analyze, and Navigate.

Please Participate in the State of Testing 2015 Survey!

It is that time for the second survey on the State of Testing! If you did not participate in the first survey, you will definitely want to participate this year! The first survey had 600 participants and this year we are hoping for at least 1,000! The survey seeks to identify the existing characteristics, practices and challenges facing the testing community in hopes to shed light and provoke a fruitful discussion towards improvement. Click here to participate.

If you are not sure if you want to participate, please take a few minutes to review last year’s survey results. A lot of valuable information on how other Testers are spending their time, how they plan, methodologies used, challenges faced and much more! I hope you will take some time to complete this valuable survey as I believe you will find the information helpful!

Women Testers – October 2014 Edition is Out! How to Keep Yourself Organized When There are Too Many Demands on Your Time!

get organized

I hope you take an opportunity to read the latest edition of Women Testers! For this edition I have an article called: How to Keep Yourself Organized When There are Too Many Demands on Your Time! I plan on writing a second part to this article and would appreciate any questions or feedback that could be addressed in the next edition of Women Testers.

If you would like to submit an article for a future edition or help out in any way, please contact Jyothi Rangaiah at editor@womentesters.com and you can find more information by clicking here. Enjoy the magazine and please let people know about it including women who might be interested in a tech career!

Developing An Interactive Shadowing Program With Videos and Exercises

job shadowing

Recently, I had a high school student spend two-days in the Testing department to gain a basic understanding of what testing is all about. Most of us know the challenges of explaining a Testers’ responsibilities without making it look too easy or so difficult that you completely lose them. Some of these challenges are discussed in a prior posting that you can find here. If you are hosting a shadowing day or any similar opportunity, below are a few ideas to get you started.

I would like to thank the following people who contributed to providing suggestions and links to potential training material. Griffin Jones, Teri Charles, Jennifer Hannon, and Stephen Becht.

  • Create an initial agenda and identify a few alternatives. During the shadowing program there might be a need to change direction based upon how the person is reacting to the information. Some material might progress more quickly than expected. Other material you might decide is not appropriate once you get to know him better. Plus remember to have fun. Do not make it so serious or intense that you lose him! And make it interactive to keep him engaged.
  • Create a Testers’ Profile by asking the Software Testers to answer a few questions on how they selected testing as a career, what they enjoy about their job, and what advice they would give a new Tester. Compile the information in a newsletter or some other fun format to provide as a take away. Host a conversation where the Testers can discuss their careers so he has an opportunity to ask questions. Try to maintain a casual environment. Look for a location that has comfortable chairs like a coffee break room or a conference room that is not the board room.
  • Guru 99 has free software testing videos that last from about 1-minute to 5-minutes. They are basically snippets of information with some of them having small exercises to perform. Once the video is completed, discuss any questions and relate the material to something that he is familiar such as Facebook, Twitter etc. Reinforcing the material is important.
  • Whodunnit? This video is a lot of fun providing some great opportunities for a conversation afterwards. Before playing the video instruct him to write down his observations. Do not watch the end of the video until he writes down observations on what has changed during the murder investigation. Consider replaying the beginning part of the video a few times so he can have a chance to find what has changed. Most people are surprised when they find out how much has changed during that short video.
  • Basketball Awareness Test: Moonwalking Bear. From an awareness perspective a Tester may notice something that does not make sense or is out of scope. At that point, the Tester has to make a decision. Go down the new path to explore or quickly note the observation returning back to the original test. An interesting conversation – if you do not notice the moonwalking bear what does that mean? Is it good or is it bad?
  • Easy Button Exploratory Testing by James and Jonathan Bach. This is a great video since it uses a lot of the testing terms in a fun manner. A great option for after lunch when he might be feeling that carb sleepiness. If possible purchase an Easy Button to reproduce some of the tests and relate them to your Testing department.
  • Cast 2011 Promo 2 Video. Another video by James and Jonathan Bach. The Hidden Puzzle is very interesting since each Tester took different test paths to identify the picture. If possible, recreate the Hidden Puzzle exercise with him and include a few Testers. How did each person approach the testing? Who found the hidden picture? How many tests did it take? This could be a fun exercise while providing some real insight on testing.
  • Steve McQueen, Consulting Software Tester. A short scene from the Towering Inferno with subtitles relating to Testing. It is best to play the video and then play it again but pausing at different times for a discussion.
  • TestBash 2.0 – A-Galumphing We Go James Bach. A longer video. After watching it determine how you will evaluate if he is ready for the material.
  • Open Lecture by James Bach on Software Testing. This video is longer and could work well if you stop it at different points for a discussion.
  • Brain Focus and other similar exercises are great since they can parallel testing in terms of understanding the requirements, working through different testing paths, and the potential danger of making assumptions. When appropriate create a decision table / tree or other diagram similar to how your team approaches a testing problem.
  • Create an exercise form with discussion questions for the videos or other material to be reviewed. This helps foster a conversation between each exercise and provides a place for him to take notes.

And most importantly… do not forget the food! Cookies, donuts, candy, going out for lunch… food is always a nice touch!

Let me know if you have any other ideas on how we might sponsor a shadowing or internship for a student whether high school or college to introduce them to Software Testing!