Image courtesy of Amazon
I like to think about Test Automation as providing tools that can execute important checks that might be tedious, time consuming, or there might not be enough time to manually perform those tests. Instead of performing these repetitive, time-consuming tests, the Testers can spend that time exploring risk areas where using their intelligence to make decisions on tests to perform is a better use of their time. (And a lot more fun than performing repetitive tests!)
Test Automation is not a trivial undertaking – research should be conducted to develop a strategy, identify tools, and determine the best approach to automate tests that are being performed manually. Test automation is not a silver bullet and it takes time to develop your approach over time by what you learned.
Do you want to learn more about Test Automation and what has worked and not worked so great at other companies? An excellent book to read that I highly recommend: Experiences of Test Automation Case Studies of Software Test Automation by Dorothy Graham and Mark Fewster. This book contains 29 case studies written by those involved with a test automation project. Each case study begins with a background including if the project was successful and if test automation is still being used at that organization. Consider starting a book club where everyone reads a different chapter(s), shares what they learned, and develop a strategic plan for your company on how you might approach test automation or improve your existing approach. (And consider adding food as part of your book club, food always make any event more fun!)
Key Take Aways:
- Test automation is not necessarily performed by Testers. It is performed by the person best qualified to use the tool selected that can include the ability to write code!
- Test automation does NOT replace Testers but should supplement your testing strategy.
- Test automation is basically maintaining a code base to test your Product’s code base.
Below are learning points; however the above resources provide a wealth of information that is not summarized within this posting.
- Make it a real project with dedicated people, time, and funding. Start with a small pilot with key objectives and gain buy in for your pilot by key stakeholders.
- Start with simple UI tests or smoke tests as a great way to get some initial protection around the code.
- Consider test automation strategy by: technology facing tests (unit, components, integration) and business facing tests (acceptance, business, GUI).
- Concentrate on testing 1 system first as a model – the system producing the most business (start small, focus on a single but important part of the system).
- It is expensive to automate all types of tests including edge cases. Identify what tests need to be in the automation suite. Think about the value of the test and the question it answers or information provided. What is the ROI?
- Tool training or employees who have experience with the tool is important to reduce expensive mistakes.
- Can Testers write the GUI tests and the Developers write the unit tests? Who will write the business layer tests? That depends on the tool selected and skills of the team.
- The developer may write the tests with the Testers running them.
- Be sure to include business people when selecting the business layer tests.
Tools & Approach:
- New code is often easier to automate than legacy code.
- Different tools / programming languages can be used for the test automation pyramid: Unit, acceptance tests, business layer, GUI.
- Test automation is good for mature code that is not constantly changing.
- Lean regression suites help keep maintenance costs low while still providing value.
- Be careful how you write your tests – reduce failures that really are not a problem.
- All tests running green do not mean there aren’t any problems.
- Once tests are automated, train Developers, Testers to understand how to maintain them.
- Determine if the tool needs to clean up after the test was executed.
- Ability to run the code across platforms, browsers, releases.
Managing Test Suites:
- Different type of test suites: nightly test suite, weekly test suite, candidate test suite.
- Keep the tests small and maintain the flexibility to run tests in logical test suites to reduce the time it takes to execute them. For example:
- you may need to execute some key tests during the day when time is running low.
- you may execute core tests with every new build for critical functionality.
- Strong management support is necessary to understand the cost of the pilot program and long-term support that includes ongoing maintenance of modifying and adding automated tests.
- Strong collaboration with Development is important – code needs to be written in a manner that it can have automated tests written.
- Make the benefits of test automation visible to stakeholders for continued support.
- Ideally Developers / Testers are dedicated to test automation to ensure that new and changed code is being reviewed to determine tests to add or tests to change.
- Identify different champions: tool champion, change agent, stakeholders, etc.
- Count number of tests for each test tool – trend over time to see how the tests increase in number.
- Send business users a calendar of green vs red if any of the tests are red.
- Build time is a critical metric since it includes the time to run the automated tests.
- On what O/S were more bugs found and frequency?
- Detection rate of bugs for tests. What tests found the most bugs? No bugs? Is the percentage of green tests trending upwards over time?
It is that time again for the State of Testing Survey! The first year about 600 people participated and last year it grew to almost 900 Testers! I hope you were one of them. Can you take a few minutes to participate in this important survey? Click here to participate. The survey is sponsored jointly by QA Intelligence – Testing & QA Management blog by PractiTest and Tea-Time with Testers.
The survey seeks to identify the existing characteristics, practices and challenges facing the testing community in hopes to shed light and provoke a fruitful discussion towards improvement. If you are not sure if you want to participate, take a look at last year’s results by clicking here. A lot of valuable information on how other Testers are spending their time, how they plan, methodologies used, challenges faced and much more! I hope you will take some time to complete this valuable survey as I believe you will find the information helpful!
And can you do us a favor? Reblog this posting to let other people know about this survey. Or perhaps you can ask co-workers or friends to complete the survey. Thank you for your consideration.
My Rewarding Experience as a Reviewer and Contributor to “More Agile Testing: Learning Journeys for the Whole Team” by Janet Gregory and Lisa Crispin
Picture courtesy of Amazon and where you can order this excellent book!
For many years I have written articles for the Testing Community and shared them through Testing Circus and Tea-Time with Testers. One warm summer day in July 2013, I received a pleasant email from Lisa Crispin asking if I would like to be one of the reviewers and contributors to their upcoming book that would compliment their existing book “Agile Testing: A Practical Guide for Testers and Agile Teams”. It was an honor to be asked and of course my response confirmed my interest. An international review team was brought together comprising of people with different experiences and expertise in their fields.
This was a rewarding experience both professionally and personally as there were many learning opportunities reviewing the chapters and discussions with the team. Checking my email was fun since there was a potential for an email from Lisa and Janet announcing what was ready for review or from a fellow reviewer with a question to foster a lively discussion. New chapters were quickly printed for when there would be time to review, digest, and comment on them. Lisa and Janet worked with the reviewers to help us identify where we could contribute sidebars to their new book. It was fun reading what the other reviewers were writing based upon their experience. For me, this is what really makes this book come alive – Lisa and Janet are sharing a lot of valuable information that is supplemented by real-life experiences. As an example I enjoyed reading how other Testers are using Session-Based Testing and Thread-Based Testing to meet the challenges of testing in a fast-paced environment!
It was a great opportunity to witness how a book comes together, the chapters under review might not be named nor are they in the order of the final publication. As the book evolved, chapters may be combined or information moved to a different chapter. After reviewing each chapter it was interesting to read the book cover to cover. It reminds me of a movie production where scenes are filmed out of order and then brought together to tell the story.
Once the draft book was completed, there was a great sense of accomplishment and a feeling of disappointment. Being part of a community working towards a common goal is fulfilling and exciting. It reminds me of working with a great project team. When the project is done there is pride of the work completed but yet a sadness that the project is completed.
The information and experiences within this book series is part of the overall testing picture. As an artist blends together different colors and uses different tools to create a beautiful piece of artwork, we do the same in Software Testing. We blend together different approaches and techniques to design and execute testing strategies to tackle complex testing problems.
As with the first Agile book, there are great learning opportunities for you to determine how to apply the information to further your testing career and tackle testing problems. As an example I love the “Pillars of Testing” model by David Evans designed with different factors that integrate to improve our confidence in the information we provide about the Quality of the Product we test. This model helped me think through the different factors to understand our strengths and weaknesses in each area of this model. Adam Knight’s discussion on “T-Shaped Skills” inspired me to think differently about training by identifying the “deeper” skills required for different flavors of technical testing and Product testing. At this time, my background includes limited automated testing experience; however, this book provides a foundation for me to better understand both the benefits and challenges to have a reasonable conversation. The chapter on “Thinking Skills for Testing” is excellent and in particular I like how Sharon Robson discussed that you need to use different type of thinking based upon the problem such as when to use critical thinking, analytical thinking, and creative thinking. Plus sample tools are suggested that can be researched further.
Lastly and most importantly, I would like to thank Janet and Lisa for this wonderful opportunity and to their continual dedication to the Testing Community. My recommendation for my readers is that you will add this book to your testing toolbox and discover how it will influence your testing world!
Picture courtesy of the BBC News Entertainment & Arts.
I was saddened to hear of Leonard Nimoy passing at the age of 83. Trekkies knew he was sick and recently was in the hospital. Of course selfishly I wanted Leonard Nimoy to be part of the 50th anniversary Star Trek movie. It was wonderful to see him in the 2009 Star Trek movie. I loved the scene where he encounters a young Kirk – and greets him warmly with: I have been and always shall be your friend. Such profound and deep words if you really understand the friendship between Kirk and Spock. And Spock’s father who tells him that he married his mother (a human) because he loved her. Those are my favorite scenes from the newer movies – we are to witness some of the Vulcan’s emotions.
Mr. Spock was one of my favorites from Star Trek the original series. As a child I faithfully watched Star Trek in reruns and never got tired of watching it over and over again! The space travel and encountering new worlds and aliens was fascinating. Even today just hearing the Star Trek theme means an adventure awaits! The Enterprise is the most beautiful ship. My love of technology comes from those early days of Star Trek. Over the years we have witnessed how Star Trek has become reality through current technology. Look at their communicators and today we have mobile devices. In The Next Generation Picard uses a small flat screen computer – now we have iPads and Skype! Technology has dramatically changed over the decades. Earlier in my career computers were DOS based and printers were dot-matrix – they were loud and slow! My initial programming classes included writing and executing JCL through punched cards. Luckily that was only one semester! Click here for more information and the below picture is courtesy of wikipedia. But my education served me well as a programmer for 12 years and then many years in the testing field!
During the 90’s computer magazines were filled with articles on how technology was changing. Of interest was the Chicago project that ended up being Windows 95. This earlier computer was expensive with a hard drive that was 1 gig. At that time it was considered to be a lot of storage! Those early days of computers with the Apple and Windows wars was exciting
Many nights started with the intention of spending an hour on the computer but ended up with late nights learning the new technology. At work my nickname was “geekette” as I helped my co-workers with the technology that was confusing for them. But for me it was a love affair! Technology was not as common as it is today with everyone having an iPad or a smart phone. Unlike the current generation, in kindergarten we did not have computers. Those of us who understood technology was a smaller group of people and many did not relate to us. But that was okay because we had Star Trek! Never underestimate the Trekkie bond!
Mr. Spock was interesting as he was half human and half Vulcan. He had the logic and self-control of the Vulcan’s but there were times when we witnessed his human emotions. When the Enterprise is under attack or there is a crisis, Spock is logical and in control. He can be heard to tell the crew that there is no reason to yell. Through the decades from TOS and the many movies we learn more about Spock and how he had to learn to keep his emotions under control and not let his human side take over.
I love this picture of the Enterprise and it is courtesy of Once Upon a Geek.
There is an interesting parallel between Mr. Spock and my life as a Software Testing Manager. My love of logic and thinking through a situation by remaining neutral comes from Mr. Spock. Often I have been told I can appreciate both sides of a story and make decisions while remaining neutral and unemotional. I believe this is an important ability for a Testing Manager. We have to keep our emotions in control when the pressure of a pending release is high or we are juggling a lot of risks trying to determine the best use of time. It would be easy to cave into the human emotions. Instead we need to focus on the facts and test assumptions to make the best decisions. High emotions can cause us to have tunnel vision by looking at a narrow piece of the problem instead of trying to understand the overall situation.
My love for technology and logic comes from Star Trek and Mr. Spock. As I write this tribute I have a marathon of TOS playing on our Smart TV and one of my favorite two-part series in currently playing: The Menagerie. One day my husband and I went to Best Buy and I saw the Smart TV and well… had to have it! Some people are dedicated to one operating system whereas I use Windows, IOS, and Android. Charging multiple devices with different operating systems every day seems to be the norm.
Image courtesy of Fanpop.
Lastly, here is a video where Leonard Nimoy discusses the Jewish story behind Spock’s hand gesture that he brought into Star Trek. Leonard Nimoy will be missed but he left a wonderful legacy and influenced countless lives. But we still hate to part so soon!
What are your Mr. Spock memories?
Live long and Prosper!
It is that time for the second survey on the State of Testing! If you did not participate in the first survey, you will definitely want to participate this year! The first survey had 600 participants and this year we are hoping for at least 1,000! The survey seeks to identify the existing characteristics, practices and challenges facing the testing community in hopes to shed light and provoke a fruitful discussion towards improvement. Click here to participate.
If you are not sure if you want to participate, please take a few minutes to review last year’s survey results. A lot of valuable information on how other Testers are spending their time, how they plan, methodologies used, challenges faced and much more! I hope you will take some time to complete this valuable survey as I believe you will find the information helpful!
There is a new testing e-magazine called “Women Testers” that has recently been published. I hope you take some time to check out their first edition by clicking here. I would like to thank Jyothi Rangaiah and her team for pulling together this e-magazine, which is a lot of hard work and dedication to make this happen! I wish them much success with bringing together different viewpoints and approaches on testing through this e-magazine. We can all have an impact on testing through the international Testing Community. Take some time to not only read the articles but determine where you can contribute. Can you write an article or perhaps spend some time reviewing articles. Many hands make the work lighter! If you wish to be involved, contact Jyothi at firstname.lastname@example.org.
Recently, I want to learn more about how Testers are gathering their testing artifacts to help them make a decision on whether the code they are testing is ready to go to production. As Testers we know we will never be able to cover all of the potential tests. People outside of testing often have a hard time understanding why full testing coverage can be impossible especially for larger modules / functionality we are testing. This includes the sheer number of tests to perform, understanding the critical risks, what is important to the customer, among other considerations and then balancing all of that with time available for testing. Recording your testing may seem scary or intimidating at first, but it can have a lot of benefits. A recording can be used when working with a Developer on how to reproduce a problem; demonstrating to another Tester what you saw in testing to determine next steps; and providing physical evidence during an audit.
Recording your testing may or may not help defend your sampling strategy but if you add journal notes with your recordings you may be able to show the progression of your testing and why you made certain decisions. This can be important when you need to go back to review why you made certain decisions or review how you tested something. Most of us cannot rely on our memory to remember why we made certain decisions. Therefore, a certain level of documentation is important.
Understanding why and how we document information is important. A great starting point to understand more about how and why Testers collect journal notes is discussed in Michael Bolton’s article: An Exploratory Tester’s Notebook.
Another great article Have You Considered Evidence of Testing?
I have had a lot of informative conversations with Griffin Jones on the importance of gathering testing evidence. Please refer to his session talk at CAST 2013 What is good evidence? You can learn more about Griffin’s consulting business Congruent Compliance by clicking here.
You can find even more information from Software Testing Club where Testers shared their thoughts on recording tools by clicking here.
I gathered, through social medial, what video recording and documentation tools Testers are using to gather their testing artifacts. I would like to thank the following people for responding: Teri Charles, Raji, Gagan Talwar, Mohinder Khosla, Ajay Balamurugadas, Richard Bradshaw, Lisa Crispin, Srinivas Kadiyala, Adam Yuret, and Griffin Jones. As always, we have an awesome Testing Community that is always willing to share information.
Below is a summary of the findings. Some of these tools are open source and others you must purchase but they may have a short trial period. I would suggest identifying what is important to you in a recording tool before you start your evaluation to make a better decision.
- Debut video capture
- Rapid Reporter
- BB TestAssistant
- BB FlashBack Express
- CamStudio OpenSource
Another suggestion by Lisa Crispin for recording and showing bugs: Quicktime. From Srinivas Kadiyala we have a few additional suggestions: SnagIt , qSnap, Paint (PrintScreen). Lalit has let me know that QTrace is now qTestExplorer! Click here to learn more! And thank you Lalit for the correction! From Kobi Halperin we have Defect Scribe that he hopes to explore and share information about.
Keep the suggestions coming and I will update this page.
How are you gathering your Testing Artifacts?
Do you record your testing?
Do you use screen cap tools?
Or do you have another method to provide your Testing Artifacts?
Throughout the “What is Your Legacy” series, I have mostly focused on the Testers in the trenches and I hope you enjoyed learning more about Testers that you may not be connected. Perhaps you found one or more Testers that you will start following on Twitter or their blogs. I love how the Testing community supports one another and how we share information within the community. Today I would like to talk about the book “How to Reduce The Cost of Software Testing”.
This book was written by many Testers who responded to a couple of questions posted on LinkedIn. It is an amazing book filled with the collective knowledge of more than 20-Testers from around the world sharing their thoughts and experiences on reducing the cost of Software Testing while still identifying the problems and bugs. A brief bio is provided on each of the authors with most of them having active websites. This is another great way to “meet” more Testers within the community.
This book also introduced me to Testers such as Selena Delesie who wrote an important chapter on “The Cost of Quality”. She presents different processes and approaches to reduce costs while improving quality through a fictional company case study. I think most of us are challenged to reduce the cost of testing but retain a high level of quality testing. We need to understand that cost as we make decisions since there are typically many different paths to choose from.
Another Tester I learned about through this book was Catherine Powell who wrote a chapter called “Opportunity Cost of Testing”. I love this chapter because as Testers every decision we make has a cost. We may select one set of tests over another set based upon the requirements and known risks. Testers are always fighting the clock; therefore understanding the opportunity costs of our decisions is fairly important.
This book is packed with valuable information; a great way to get acquainted with a lot of Testers; and the chapters can be read in any order. It does not get much better than that! Hopefully you will have an opportunity to read this wonderful book and apply what you are learning to your own department.
I hope you are enjoying this series as much as I am! Today I would like to share what I admire about Lalitkumar Bhamare and Teri Charles.
I admire Lalit for being a co-founder and Editor of Tea-time With Testers (TTWT). He works endlessly to publish an innovative, quality e-magazine. Think about how much work and time it takes to write one article that is worthy for publication. Now imagine working with authors to write articles, interviews, and other creative ways to keep TTWT fresh and interesting. I would like to share a few initiatives Lalit has been involved. If you go to their home page and scroll down, you will find “Our Story and Testimonials” video, which I was happy to be involved. I hope you take some time to watch it if you are not familiar with TTWT. The concept of the Women in Testing edition published in September 2013 was wonderful to read with so many women sharing their viewpoints on technology. Leah Stockley was the guest editor and she did a wonderful job!
Last year I got an email from Lalit asking if I wanted to participate in the State of Testing 2013 survey that TTWT was partnering with Joel Montvelisky. Joel was looking for information to write a post about the advances in the testing world in the last 5-10 years and he realized that there is no centralized set of information that provides visibility into what is happening and what are the trends in the world of testing today. That is when he turned to Lalit and they decided to turn this into a project. It is an important survey that in the short term you can use it to learn how you compare to other testers and where you might make changes. Over the upcoming years, we will have data to trend to better understand where we are headed as a Testing Community. You can learn more about Lalit at his LinkedIn profile.
Teri has so much enthusiasm for learning and encouraging others. I “know” Teri through Twitter and I have found she typically is the first person to congratulate or encourage someone. Through her blog Boulder Tester she shares her experiences in developing her testing skills and she is not afraid to get her hands dirty by learning how to write code. On her blog she discusses how coding never came as a natural skill. So what is her approach? Create a plan and then collaborates with a couple of other people to teach an “Introduction to Programming Concepts” course! You really need to read about her journey by going to her blog. Being a former programmer who studied many programming languages at college – I know this is not easy to do on your own. It is typically easier to learn another language once you have studied and understand the basics. So I have to say what Teri is doing is quite impressive!!
I love the article Teri wrote about her testing journey in the article Be a Student for Life for Tea-Time with Testers, September 2013 “Women in Testing Special”. I too am a learner and am always curious about how things work and why they work certain ways. Like Teri I think we both want to know everything and most likely get ourselves a little overbooked because we want to do everything and know everything. I look forward to see what Teri does next! You can learn more about Teri through her LinkedIn profile.
I am enjoying writing this series on “What is Your Legacy?” There are so many wonderful testers in the trenches that are making a difference in the Testing community whether it sharing testing approaches or encouraging others to develop and further their skills. In Part 1 and Part 2, I share a few testers and leaders that I admire – even though some are “famous” in their own right I am focusing more on the testers who have found a way to have their voice heard! We may not all write books or speak at conferences but you can make a difference. Find your voice and a medium for it to be heard. Remember Jerry Weinberg, Lisa Crispin, James Bach, and Michael Bolton had to start somewhere! None of them achieved their success over night – instead it was years of dedication and hard work. So much has changed with social media, blogging, Skype, and e-books that the world has become smaller. You can “chat” with people around the world opening up learning opportunities. This provides excellent opportunities to “meet” testers that were not possible at one time. Today I would like to share a couple more testers that I admire. Do not forget to read the first part and the second part of this series.
Jean Ann Harrison
Jean Ann has such a heart and passion for mobile testing and works endlessly in helping companies and individuals understand how mobile testing is different than testing a web-based application. You can often find her offering a free webinar and she is active in social media engaging in meaningful discussions on how we can improve testing. You can learn more about Jean Ann’s impressive background from her LinkedIn profile including her publications. I would like to share a couple of interviews from CAST 2012 that provides meaningful suggestions on Mobile testing that can also be used for other forms of testing.
Jean Ann on Mobile Testing. A short YouTube video where Jean Ann talks about the importance of gathering information to isolate a problem when a developer is not convinced the problem is related to his code. This is a good example of how gathering more intelligence to tell the testing story differently so the developer understands the impact.
Jean Ann on the Complexity of Mobile Testing. A short YouTube video where Jean Ann talks about organizing your exploratory testing for Mobile testing. Through a google search you can locate other webinars that Jean Ann has presented and freely distributes to the Testing community.
I became acquainted with Adam through a book review committee and was immediately impressed by his feedback in the discussion groups. Subscribing to his blog Software Testing – a Sisyphean Task? I learned more about thread-based testing which is similar to session-based testing. I had been using a similar approach and was glad to read how other testers are addressing testing problems with this approach. Adam shared his thread-based template and how he added a dashboard by using a MySQL database. I loved his approach and adapted it to how we work. So thank you Adam for sharing your approach with the Testing community! I used this article for one of my Lean Coffee Chats and the team really enjoyed the discussion – plus you can read James Bach article on why he came up with thread-based testing.
Adam provided this link within his article along with a lot of other valuable information providing an overall picture. He provides a lot of great research material with his articles and I would recommend that you follow his blog. Plus here is a short video from Adam on Sources of Learning from EuroSTARConference. You can learn more about Adam from his LinkedIn profile.