Recently, I want to learn more about how Testers are gathering their testing artifacts to help them make a decision on whether the code they are testing is ready to go to production. As Testers we know we will never be able to cover all of the potential tests. People outside of testing often have a hard time understanding why full testing coverage can be impossible especially for larger modules / functionality we are testing. This includes the sheer number of tests to perform, understanding the critical risks, what is important to the customer, among other considerations and then balancing all of that with time available for testing. Recording your testing may seem scary or intimidating at first, but it can have a lot of benefits. A recording can be used when working with a Developer on how to reproduce a problem; demonstrating to another Tester what you saw in testing to determine next steps; and providing physical evidence during an audit.
Recording your testing may or may not help defend your sampling strategy but if you add journal notes with your recordings you may be able to show the progression of your testing and why you made certain decisions. This can be important when you need to go back to review why you made certain decisions or review how you tested something. Most of us cannot rely on our memory to remember why we made certain decisions. Therefore, a certain level of documentation is important.
Understanding why and how we document information is important. A great starting point to understand more about how and why Testers collect journal notes is discussed in Michael Bolton’s article: An Exploratory Tester’s Notebook.
Another great article Have You Considered Evidence of Testing?
I have had a lot of informative conversations with Griffin Jones on the importance of gathering testing evidence. Please refer to his session talk at CAST 2013 What is good evidence? You can learn more about Griffin’s consulting business Congruent Compliance by clicking here.
You can find even more information from Software Testing Club where Testers shared their thoughts on recording tools by clicking here.
I gathered, through social medial, what video recording and documentation tools Testers are using to gather their testing artifacts. I would like to thank the following people for responding: Teri Charles, Raji, Gagan Talwar, Mohinder Khosla, Ajay Balamurugadas, Richard Bradshaw, Lisa Crispin, Srinivas Kadiyala, Adam Yuret, and Griffin Jones. As always, we have an awesome Testing Community that is always willing to share information.
Below is a summary of the findings. Some of these tools are open source and others you must purchase but they may have a short trial period. I would suggest identifying what is important to you in a recording tool before you start your evaluation to make a better decision.
- Debut video capture
- Rapid Reporter
- BB TestAssistant
- BB FlashBack Express
- CamStudio OpenSource
Another suggestion by Lisa Crispin for recording and showing bugs: Quicktime. From Srinivas Kadiyala we have a few additional suggestions: SnagIt , qSnap, Paint (PrintScreen). Lalit has let me know that QTrace is now qTestExplorer! Click here to learn more! And thank you Lalit for the correction! From Kobi Halperin we have Defect Scribe that he hopes to explore and share information about.
Keep the suggestions coming and I will update this page.
How are you gathering your Testing Artifacts?
Do you record your testing?
Do you use screen cap tools?
Or do you have another method to provide your Testing Artifacts?
Many of my readers of “The Testers Edge” may be pretty tech savvy when it comes to passwords and security. On my other blog “Realistic Cooking Ideas” I Previously wrote a posting about creating strong passwords for your accounts. Many people use “weak” passwords that are vulnerable for hackers. This time we have an Internet security hole for websites that start with “https” in the URL. The media is telling everyone to change their passwords; however do NOT change your password until the website provider applies the patch. Once they have applied the patch then you must change your password.
This is a very serious security hole that does not affect all Internet sites that have secured logins. There are a lot of sources to find out what sites are affected. This link provides a list of sites that are impacted that have applied patches and ones that are not impacted. Here is another link about Heartbleed that claims it affects 80% of websites! Here is a link I found regarding WordPress. And here is a tool to check a website heartbleed test. As a side note, I am in the tech field but on the software side. I do not specialize in security but I hope that sharing this information is helpful.
Once you know if a site you use is impacted, do NOT use it until there is a PATCH and then
CHANGE your password to a STRONG password!
I want to wish all my readers of “Realistic Cooking Ideas” and “The Testers Edge” a wonderful New Years! Instead of making yearly resolutions, I have a vision for my life. Typically they end up being 3-year visions. For those who are familiar with agile approaches my vision is similar to story mapping. I have a vision that can have multiple goals and over time different activities and tasks develop those goals and vision. For 2014 I would like to encourage everyone to dream big and find activities to work towards that dream. Plus get out of your comfort zone – when we stay in our comfort zone we reduce our learning opportunities. When we are afraid to fail, we never take the calculated risks that are necessary to grow our careers, our blogs, and other areas of our life. I believe in “failing quickly”. Try something new at a manageable level so you can quickly learn from it. For example, if you are trying a new test approach, try it with a smaller project where you can quickly learn what works and does not work. For those of us who cook, lets try a new ingredient that we are not real sure about in a small dish before tackling a more complex dish.
For my professional career I want to get deeper into different testing techniques and approaches to increase and broaden my testing toolkit. This includes understanding agile approaches and techniques to learn how to use them in environments that do not have self-managed teams. Plus I want to learn more about different context-driven testing techniques. From a quality viewpoint, I believe everyone, regardless of job title, has an important role to ensure quality from a product and service perspective. I would like to better understand how we could improve quality from all aspects within an organization.
When I started “Realistic Cooking Ideas for Busy People” in February 2011 it was to share recipes and ideas on how to make a quick dinner after work. Typically there is not much time and often we are tired. Too many times we go to a restaurant and pay too much money for a sub-par dinner. I have shared a lot of quick to make recipes and casseroles that can last a few days. When I can, I share tips on how to make dinner a bit easier to make. In 2014 I would like to find more recipes that you can make quickly or make in steps to reduce the time once you come home from work. For Christmas I got a pressure cooker – I hope to share a few recipes – and my thoughts on using it after a busy day at work.
I believe all aspects of our life are intertwined from our careers, home life, hobbies, etc. When you accomplish a challenge it makes you stronger in all areas of your life. Today I would like to share with you some beautiful pictures from Mt. Willard – a mountain in New Hampshire that Greg and I climbed this past September. Officially this is our first mountain that we climbed since hiking to New Hampshire waterfalls does not qualify. At one time hiking a mountain was way out of my comfort zone! It took three trips to New Hampshire and a lot of training to get to this point. The hiking in New Hampshire is much more rugged and difficult than in Western New York. They really do not have rocks – they have boulders and thick intertwining tree roots. Every step can twist an ankle! Hiking is more of a straight up climb where after a few minutes you wonder why in the world are you doing this? Imagine if after the first trip to New Hampshire we just gave up on climbing a mountain because it was too difficult. What a wonderful experience we would have missed with such fabulous views! I would encourage you to dream big and have patience to work towards achieving that goal. Go outside your comfort level but remember to always manage the risks. I would never recommend that you take dangerous risks to achieve a goal. There are many mountains in New Hampshire with a higher elevation than Mt. Willard. But for me climbing Mt. Willard is the maximum of my physical and mental capabilities after a three-year journey. Below are pictures from Mt. Willard and a video that I hope you will take a minute to watch, as it is a beautiful view as you arrive to the top!
One of the things I love about the Testing Community is how we share information and learn together. Before I move on with more postings on leadership, I wanted to share with you information I learned from Lalit Bhamare, one of the editors of Tea-time with Testers about a project he and Joel Montvelisky are embarking upon. You may be familiar with Joel’s blog QA Intelligence – a QABlog and the e-magazine Tea-time with Testers.
Background on this project
Joel was looking for information to write a post about the advances in the testing world in the last 5-10 years and he realized that there is no centralized set of information that provides visibility into what is happening and what are the trends in the world of testing today. That is when he turned to Lalit and they decided to turn this into a project.
They are going to launch a survey towards the end of the week and it will run for about 10 days. For this project to be productive the survey needs to reach as many testers as possible worldwide.
How you can participate
This is where you and I can be part of this project. Please click here and you will be directed to Joel’s blog where he shares more information about this survey and you can subscribe to get notified when the survey goes live! All you need to do for now is enter your email address to receive the notification. This is an excellent opportunity to be involved at the beginning as this is a survey that will be conducted each year. What a fabulous idea to start gathering information on testing trends and how they change over time.
What else you can do
After you sign-up to participate in the survey, let your friends and co-workers know about the survey. You can share this information through Twitter, Facebook, G+ and other social media platforms. The more testers around the world that participate the better the results. I do hope you will consider participating!
I will never forget the time when a release went to production and a business user thought he found a serious problem. He proceeded to tell everyone about the bug which made the developers and testers look bad that we missed something so serious. Long story short… the business user made a mistake and there was not a bug. But the damage was done. Of course no one went around telling everyone there was not a serious bug. Since that time I became rather sensitive about using the word “bug”. I prefer to say that we have a problem that needs to be researched. It may or may not be a bug.
I originally wrote an article “Guidelines on a Meaningful Problem Report” in the November 2011 edition of Testing Circus. Click here to read the original publication. I have revised this article to include the perspective “Is it really a bug?”
Providing the correct information to the development team on a problem discovered during testing is an important role of the tester. To gain an understanding of the functionality’s expected behavior the tester may talk to a business user or a business analyst. Through researching a problem we may discover it is a bug; we did not understand the expected behavior and it is intended functionality; or it might be an enhancement.
General Guidelines to Research a Problem
Below are guidelines in researching and writing a problem report. I do value communication with the development team over sending them problem reports. But sometimes they are busy and cannot stop what they are coding to review a problem. In those cases and when submitting to a bug tracking system it is important to write a concise, accurate report.
When writing a problem report or in your verbal discussions with a developer it is important to provide the correct steps to reproduce the problem. It is best to submit a report once the problem is verified with the expected behavior. False or incorrect problem reports can harm a tester’s reputation and relationship with developers. Make sure you can reproduce the problem. For example, it is important to reproduce the problem to determine if:
• you can reproduce it upon demand;
• the failure is related to specific test paths;
• the failure is associated with specific data;
• certain environments cause the problem (ie., client, browsers).
If you encounter a sporadic problem, work with the developer to determine what information will help him find the solution. I would document as much information as I can capture. Depending upon the type of problem, I might add what other programs are running at the same time. Providing a percentage of the failure can be helpful, such as: I ran test “x” 10 times and it failed 60% of the time and here is how it failed. Sporadic problems can be difficult to fix; therefore, it is important to provide as much information that you can gather. Sometimes I like to review the information I am gathering with a developer before I complete my tests in case he can identify additional details that will be helpful.
When appropriate, have another tester reproduce the problem to help identify additional details. If the problem cannot be reproduced on a different machine, document any differences between the two computers.
Reproducing a problem can also identify tester error and it is best to catch those prior to submitting a problem that does not exist.
Some problems will produce an error log with additional information that is valuable to the developer. Understand what additional technical information is valuable and be sure to capture that information.
I only provide screen shots if the problem is difficult to describe. In those situations, I would prefer to discuss the problem with the developer. If that is not possible or more documentation is needed, I will consider adding screen shots. However, I will crop them to show only the important details as to reduce the noise in the report. I use them as supplemental information as my written report will contain all the information without needing to review the screen shots.
Document the Steps
Remember that someone else needs to reproduce this problem on his machine and sometimes problems are fixed months after they are submitted. Therefore, take time in documenting the steps and review them by re-testing to ensure all critical steps are captured. Identifying at what point the problem occurs is important. For example, if the problem occurs when clicking on a create button or when clicking on the save button is valuable information to the developer.
Is it a Known Problem?
Before submitting a new problem, review the bug tracking system to ensure it has not been logged. If a problem has been logged, add any additional details if the information provide new insight. If a similar problem has been entered, work with the employee who can make help determine if a new problem should be submitted. You do not want to add too much noise to a bug tracking system. I have seen the results of a bug tracking system where everyone adds new issues without consideration of what was previously entered. It becomes difficult to find information because there are duplicate bugs and issues are entered that are not bugs. I always believe it is a good idea to first check your bug tracking system before you get too far in writing a problem report.
Discussions with the Developer
If there is an opportunity, review the problem with the developer to gather more information to add to the report. For example, a developer may ask for additional tests to be performed and those results can be part of the report. This conversation may help the tester understand if the problem has been logged or if there is a difference in opinion on how the functionality should perform. Any differences should be resolved with the business user before submitting the problem.
Discussions with the Business User
If the tester is unsure if it is a problem, or does not understand the intended behavior, review those questions with the business user. I prefer that the tester first review questions with a tester who may understand the business functionality. That way we are tapping into the business community to seek out new information for the department instead of continual training on the same areas. I use the same process with the development team. Before submitting the report, make sure it is a problem and that the correct expected behavior is defined.
How Much Information to Document
How much information should you document in the test report? The details can depend upon:
• the company’s standards;
• how difficult it is to reproduce;
• the complexity of the problem;
• how much information the development and testing team requires.
Writing the Report
When writing the report, I prefer to keep the sentences concise providing critical information at the beginning of the report. Additional information that may or may not be helpful to the developer should be placed at the end of the document with a descriptive heading. This ensures the developer does not have to read through the document to find the important details. Re-read your report to ensure steps are not missing, sentences are clear, unnecessary information and words are not included, and follows any company’s standards.
Submitting the Report
The tester will submit his report based upon the bug tracking system adopted by the company. Sometimes a developer may question a tester’s report especially when a developer is closely attached to the feature. It is best to handle these conversations with the facts, reproduce the problem with the developer if that is helpful, and do not get into an emotional discussion. If an agreement is not reached, then be sure to involve the business user who is responsible for the functionality for the final decision. It is best to minimize these situations by gathering sufficient upfront information and having conversations with the developer and business user (or Test Lead, Test Manager) prior to submitting the problem. Submitting a false or incorrect problem report can hurt the tester’s reputation and relationship between developers and testers. Refer to my posting on Crucial Conversations for a communication model that can help with bridging conversation gaps.