How to Perform Software Testing

 

 

How to Perform Software Testing

Colorado State University

Department of Electrical Engineering

JTC300 Professional and Technical Communication

Minh Anh Nguyen

Email: minhanhnguyen@q.com

Table of Contents

List of figures/illustrations                                                                                  3

Chapter One: Introduction of Software Testing                                      4

Chapter Two: Software or application requirements                                           5                     

2.1  Prerequisite to installation                                                                            5

2.2  Net framework                                                                                           5

2.3  Internet connection                                                                         5

2.4  Operating system                                                                                        5

2.5   Third party development tools                                                                    6

2.6  Hardware                                                                                                   6

           

Chapter Three: Testing                                                                          6

3.1  User requirements                                                                                       6

3.2  Functional testing requirements                                                                    6

3.3  Non-Functional testing requirements                                                            7

3.4  Test case or suite                                                                                        8

3.5  Test (execute test)                                                                                      8

3.6  Verified test case                                                                                        9

3.7  Ease of use                                                                                     9

3.8  Simple test report                                                                                       9

3.9  Scheduler                                                                                                   11

Chapter Four: Benefits of software testing                                                         11

Troubleshooting                                                                                                13

References                                                                                                       18

Glossary of Basic Terms                                                                                   19

List of figures

Figure1. Example of software test report template                                             10

Figure2. Example of software defect report template                                         16

Chapter One: Introduction of Software Testing

Software testing is the process of evaluating the developed system to assess the quality of the final product or computer software.  The purposes of software testing are to discover errors, mistakes, and flaws in the developed software and to ensure that software satisfies the specified requirements of customers. Software testing is a crucial part of the software development cycle [2, 4, 5,7]. It is closely linked to what has come to be known as structured analysis and design.  The software development cycle is a structure enforced on the development of a software product. During the development cycle, software engineers and test engineers are working together to find bugs and ensuring that the product must include all necessary functions and be compatible with the targeted environment [1,3,4,5]. Therefore, the software testing process consumes a lot of time during the software development cycle. The software development cycle begins when a software or product is released and made available for the public because there are suggestions for improvement from customers that can be taken into consideration in making the product or software better.

Software testing includes the development of procedural scripts that not only save time and resources when applications are updated, but also speed up the process of testing when regression testing is necessary. These scripts  also include the development of  test cases or suits, test reports, test results, analysis of test results, and instructions for setting up for each test case [1,4].

There are plenty of software testing books which will tell test engineers or users how to test well.  This user’s guide provides some tips, techniques, and guidance for developing software testing programs based on extensive practical experience.

Chapter Two: Software or application requirements

2.1 Prerequisite to installation

Before installing or updating any new software, users need to know some of the requirements for installation to avoid any unexpected difficulties during execution. This would be included uninstalling any previous or existing software, and identifying the Net framework, internet connection, operating system, etc.

2.2 .Net Framework

Net framework is a subject which is commonly bundled with different versions of Windows Operating System. Some frameworks such as Themis require .NET 4.0 runtime or higher to be installed in advance before launching the application. The .NET 4.0 program can be downloaded for free from Microsoft Download.

2.3 Internet Connection

If the software or application can be accessible through LAN or GPIB, then it will not require a faster internet line with a minimum of 1Mbps, since LAN connections typically come with at least 10Mbps.  However, if the application is accessible through the internet, then it will likely require a faster internet line with minimum of 1Mbps. This requirement is needed to avoid a high rate of failure.

2.4 Operating System

Some software or applications can only be installed and run on a 64-bit operating system. Users must check to see whether or not the software is compatible with their operating system such as Windows, Linux, Mac, etc, before they install the software.

2.5 Third Party Development Tool

Some applications or software require development tools which have been developed by other software companies such as Dell, Intel, etc.  The purpose of buying and using third party development tools is to facilitate getting the element values or results for specific tests in the HTML format which customers commonly request.

2.6 Hardware

Users also need to check the application/software compatibility with different hardware configurations. Hardware is mostly a computer, but sometime it is a test instrument. The computer hardware has a central processing unit (CPU); this CPU is the heart of the computer system and is responsible for executing, or running the software.  The CPU can only understand a series of binary numbers such as 0s and 1s, therefore the software must translate human language into series of codes which are made up of binary numbers [2, 3]. Each code has its own meanings and operations.  Hardware and Software are two main types of components required for a system to work.

Chapter Three: Testing

3.1  User Requirement

Defining user requirement is an important phase which helps software developers and testers in specifying system functions to make sure the program does what the user needs it to do.

3.2 Functional Testing Requirement

Functional testing addresses several requirements which are needed for successful software testing. It deals with requirement documentation and shows that any specific features will work. Therefore, function testing is done by using the design specifications provided by the design team or the functional specifications provided by customers. The goal of this type of testing is to verify that the software or application meets customer’s requirements, that it does not have any major bugs, and to ensure that the software is ready for release to the public. Functional testing is a black-box type of testing and is used to evaluate the software’s compliance with its specified requirements [2, 4, 5].  In actual or live mode, test engineers need to verify specific functions of the code.  This type of test would be executed before the non-function testing.  Four requirements of functional testing are: 1. Preparation of test data based on the specifications of functions. 2. Accurate representation and of customers’ requirements. 3. The execution of test cases. 4. Observation of the actual and expected test outputs. Some types or methods of functional tests are: unit testing, smoke testing, interface testing, regression testing and user-acceptance testing [2, 4, 3].

Positive and negative functional testing are two basic categories of functional testing.  Positive functional testing involves inputting valid inputs to see how the software application responds to these and also testing to determine if outputs are correct. Negative functional testing involves using different invalid inputs, unanticipated operating conditions and other invalid operations.

3.3 Non-Functional Testing Requirement

Non-functional testing is a type of testing of software for its non-functional requirements. It is designed specially to evaluate the readiness of a software or product to handle subjects or problems which are not covered by functional testing.  It is also used to measure and compare the results of testing attributes of software [3, 4, 5]. The goal of this test is to see whether or not the software stands up to customer expectations or behaves correctly and meets performance requirements.  Non-functional testing is used to determine whether or not the software will provide a good user experience.  This test can be started after the completion of functional testing. Non-functional testing has a great influence on customers.  This test demonstrates how much time the software will take to complete a task or responds to a request. Some types or methods of non-functional tests are: installation of software, loading of software, and configuration of software [2, 3, 5]. 

3.4 Test Case or suite

A test case is a sequence of steps that software test engineers or quality assurance engineers create to perform a certain test on the software or application. A good test suite is one that detects real faults because the set of faults in a program is unknowable. Test cases involve required inputs and are sometime referred to as test scenarios.  A test suite contains more than one test case and the successful completion of one test case must occur before the next test case can begin.  For example, test case 1: login hardware, test case 2: load software, test case 3: open test file, test case 4: run test file, test case 5: logout.  Test case 2 cannot start until test case 1 finishes.  A test suite is a collection of test cases, which is used to test a software or application to show how the software behaves.  A test suite also contains detailed instructions, descriptions or goals for each test case and information on the software configuration or prerequisite steps

3.5 Test (Execute test)

For any software, users can execute tests in two basic forms: actual or live mode and simulation mode. There are some differences between test runs in actual or simulation mode. Simulation mode only allows an application to perform a mock run on the defined test collection. The test results or values are dummy values that the users have created. The purpose of simulation is to avoid errors discovered during actual test runs. Any error messages which are received during the simulation must be fixed before a run in the live mode.   In actual test or live mode, the software allows users to define values but test results are real and generated by the system or instrument itself. Any mismatches or wrong values which are received during live mode testing are considered bugs.  Software test engineers must contact the software development engineers and explain the issues, and then both engineering teams must work together to resolve the issues before releasing the software to customers.

3.6 Verified Test Case

A verification process is needed to making sure that test case is setup correctly. The verification processes can be performed by eye-ball checking or simulation.

3.7 Ease of Use

Some software or applications are designed in such a way that navigation through them is user-friendly and lacks complexity.  Test engineers need to test all the new features which are embedded in the software to ensure that these features work correctly.

3.8 Simple Test Report

A test summary report is prepared at the end of testing and shared with all involved team members such as software developers, software testers, etc. The goal of a test report is to explain details and activities about the testing performed on the software to the project manager and team members.  Therefore, the test report must be clearly written, concise and easy to understand. The test report is extremely important because it is evidence for the testing performed and the testing conclusion.  During the test cycle, the test report is a communication sends out to the quality assurance team describing activities of the day or software behavior [2, 3].  The test report should include defect information, test description, and test case or requirement number.

Some companies have a test report template or special tool to generate test report, which makes work much easier for test engineers, because they only have to fill-in the test results.  If there is no test report template available, then test engineers or quality assurance engineers must create one. There are many types of test reports are available online.  A sample of a test report template that I created and presently used to submit test reports to my customers is shown in the figure below.

                        Figure1. Example of software test report template

Individual testers or test engineers will be sending their status on execution of the test cases or a test report to  the  lead  tester, project manager, and software engineers,  on  a  timely  basis  as  described  in  the  test  plan  document.

3.9 Scheduler

Careful logical scheduling is useful for implementing new ideas. Imagine that in large organizations, interaction occurs in a complex environment and each different host system has its own availability issues. Sometimes, test equipment and software require a special set up before engineers can perform any testing.  Therefore, in order to avoid wasting time to setup the equipment, software test engineers are often required to work in weird hours in order to test the software or application.

Chapter Four: Benefits of software testing

There are many different approaches and techniques to test software, with different benefits for software quality and the development process. Software Testing is a process of finding errors while executing a program. It is also the process of validation and verification of the software product, which in turn should deliver zero defects, and result in reliable and quality software products for users with more accurate and reliable results. Software testing provides a means to cut errors, reduce maintenance time or costs, and to reduce the cost of software development.  It provides a way of analyzing a software item to detect the differences between the software’s existing and required functionality.  Each piece of software or application needs to pass all of a testing program to ensure that consumers have a good experience with the product. Failure to release a working product that meets customer requirements can damage company’s reputation and reduce overall product sales.  During my experience in software testing, I have seen diverse business benefits such as improved quality, a gain in confidence for programmers, and earning trust from customers.

Testing techniques refer to different methods or ways of testing particular features of a computer program, system or product. Only the testing phase can ensure the quality of the software. Currently, there are many different software testing techniques and methods that software test engineers and software development engineers can use to improve software quality and reliability.

Troubleshooting

The goal of software troubleshooting is to filter out and resolve problems, and to restore the software to normal operation.  Software troubleshooting is the process of scanning, identifying, diagnosing and resolving technical or source-code-related problems, errors and bugs in software.  This can be done by using both functional and non-functional testing requirements.  Software troubleshooting can also be performed when software needs to be configured correctly, such as resolving issues due to incorrect installation or restoring software after corruption or file deletion caused by a virus.

There are many software troubleshooting techniques or steps that test engineers can use to find reasons for failure in software applications. Test engineers can use some of the troubleshooting tools to check basic connectivity issues, or when they set up a test environment. These tools are normally are normally called Third Party Development Tools such as  Agilent IO and NIMAX, and they can be brought from KeySight Technologies and National Instruments (NI) companies. Each test engineers would have their own methods and check list of troubleshooting procedures.  A sample check list of software troubleshooting techniques that I created and presently use to find the cause of failing tests and to confirm error messages is provided below. Following this check list will show whether a problem is not software related or not a real bug: 

1.      Check if software versions are compatible. If things don’t work in a particular environment the problem may be related to browser compatibility, the operating system version, product version, etc. This step will help avoid some invalid bugs being logged later by showing up front that the software is not supported in any way.

2.      If test engineers are using any configuration file for testing their application, then they must make sure this file is up to date as per the application requirements. Many times some global configuration file is used to pick or set some application flags. Failure to maintain this file as software requirements change will lead to malfunctioning of the application under test. This is not a bug.

3.      Check to see if a missing table is the main reason that the application will not work properly. Check to see if a database is a proper set.  This is not a bug.

4.      Check to see if there is any other hardware issue that is not related to the software application.

5.       Make sure that the application hardware and software prerequisites are correct.

6.      Check to see if all software components are installed properly on the computer or test instrument.

7.      Check whether registry entries are valid.

8.       For any failure, look into the ‘system event viewer’ for details. Test engineers can trace out many failure reasons from the system event log file.

The troubleshooting steps test engineers follow will be of great help during their test execution. Software troubleshooting will provide test engineers with answers to the following questions:

1.      What is not working?

2.      Why is it not working?

3.      How can you make it work?

4.      What are the possible reasons for the failure?

Test engineers need to perform software troubleshooting before they submit any bug reports.  This is to confirm that an error message is really a bug and to make sure that they did not miss any important flag or they did not configure their test setup properly.  The first step of the software troubleshooting process is identifying the problem.  In this step, test engineers will be checking on possible issues that can cause such problems and work on measures and then alternatives to find a solution.  Troubleshooting will help test engineers to decide whether the ambiguity they found in their software application under test is really a bug or any test setup mistake. 

If there is a bug or defect in the software application or product, test engineers must submit a bug or defect report so that software developers can fix the problem. Test engineers should try to get 100% of the bugs resolved. This requires bug reporting skill.

Most companies have a bug report template or special tool to generate a bug report. This template makes work much easier for test engineers because they only have to fill-in the test issue and error messages that they saw during the execution of the test and after troubleshooting steps.  If there is no bug report template available, then test engineers or quality assurance engineers must create one. There are many types of test bugs report template available online.  A sample of a test bug report template that I created and presently use to submit test bug reports to my software developers or engineers is shown in the figure below.

Figure2. Example of software defect report template

After any bug fix, test engineers must verify that the defect is fixed and they must retest all test cases to ensure that the fix will not cause any more issues for the software application. For example, say a test engineer is verifying a defect fix, and finds that there are 3 test cases failed due to this defect. Once the test engineer verifies the defect fix as resolved, the test engineer will retest or test the same functionality again by executing the same test cases that failed earlier. Test engineers should submit a closed bug report to close the issue, if they confirm that the defect is resolved.

References

1.      Javed, A. (2012). How To Improve Software Quality Assurance In Developing Countries. ACIJ Advanced Computing: An International Journal, 17-28.

2.      Limaye, M. (2009). Software testing: Principles, techniques and tools. New Delhi: Tata McGraw-Hill Education Private.

3.      Myers, G., & Badgett, T. (2004). The art of software testing (2nd ed.). Hoboken, N.J.: John Wiley & Sons.

4.      Patton, R. (2001). Software testing (1st ed.). Indianapolis, IN: Sams Pub.

5.      Patton, R. (2006). Software testing (2nd ed.). Indianapolis, IN: Sams Pub.

6.      PUMA- Publication MAnagement. (n.d.). Retrieved July 8, 2015, from http://puma.isti.cnr.it/rmydownload.php?filename=cnr.isti/cnr.isti/2004-TR-36/2004-TR-36.pdf.

7.      Tzmann, M., & Young, C. (2003). Software testing and internationalization (English ed.). Salt Lake City: Lemoine International.

Glossary of Basic Terms

·         Assigned To: The name of the person that is assigned to analyze or fix the defect.

·         Attachments: Attach any additional information like screenshots and logs.

·         Black box testing: is a functional testing.

·         Close Date: Date which the test engineer has checked the fixed issues and closed the issues.

·         Defect or Enhancement: Defect means the software is not working the way it says it will. Enhancement means the software is not working the way customers want.

·         Defect ID/Number: A unique ID number used to identify the issue in the issue tracking log.

·         Defect Priority: Priority of the Defect. How soon the problem report submitter requires a solution. The accepted values are:  1. Critical: issue will stop project progress if not resolved.  A solution is needed immediately. 2. High:  A solution is needed as soon as possible. 3. Medium: the problem should be solved in the next release. 4. Low: the problem should be solved in a future release.

·         Defect Severity: Severity of the Defect.

·         Defect Type: Type of the defect. It can be Functional, Installation or GUI issue

·         Descriptions: Any prerequisites or preconditions that must be fulfilled prior to executing the test.

·         Fixed by: Name of software engineer or developer who fixed the issue

·         GPIB (General Purpose Interface Bus):  is used to enable electronics test equipment to be controlled remotely.

·         Impact Summary: a description of the impact of the issue.  The impact may be expressed in terms of one or more of the following:  schedule, scope, resources, and space.  The impact description should also include a reference to any milestones.

·         Integration testing: is used to identify whether or not there is any bug in software.

·         LAN-local area network: it is a computer network

·         Load software testing: is used to check whether the application can load during develop software.

·         Mbps- Megabits per second: is referred to data transfer speeds and measured in megabits

·         Name of person doing test: The name of the person who executed the test

·         Open Date: Date on which the defect was initially reported and logged.

·         Planned Fix Build Version: Build version of the product where the defect was fixed

·         Problem Description: Detailed description of the defect.

·         Regression Testing is tested on the entire software and not just the new functionality and defect fixes.

·         Release/Build Version: Release/Build version of the product where the defect was detected.  

·         Reported by: The name of the person who reported the defect.

·         Requirement ID or test case ID: is the ID of the test case.

·         Smoke testing: is used to make sure the major functionality is working fine.

·         Software Build date: The date of build of the software.

·         Status: The status of the defect. 

·         Steps to Replicate: Step by steps to reproduce the defect.

·         Test Case Id: The test case that caused a bug to be raised.

·         Test date: The date of execution of the test.

·         Unit testing: It is useful to identify, determine bugs and errors

·         Validation: Step-by-step procedure to execute the test.