Software testing is any activity aimed at evaluating an attribute or capability of a program or system and determining that it meets its required results. Although crucial to software quality and widely deployed by programmers and testers, software testing still remains an art, due to limited understanding of the principles of software. The difficulty in software testing stems from the complexity of software: we can not completely test a program with moderate complexity. Testing is more than just debugging. The purpose of testing can be quality assurance, verification and validation, or reliability estimation. Testing can be used as a generic metric as well. Correctness testing and reliability testing are two major areas of testing. Software testing is a trade-off between budget, time and quality.

Introduction

Software Testing is the process of executing a program or system with the intent of finding errors.Or, it involves any activity aimed at evaluating an attribute or capability of a program or system and determining that it meets its required results.Software is not unlike other physical processes where inputs are received and outputs are produced. Where software differs is in the manner in which it fails. Most physical systems fail in a fixed (and reasonably small) set of ways. By contrast, software can fail in many bizarre ways. Detecting all of the different failure modes for software is generally infeasible.

Unlike most physical systems, most of the defects in software are design errors, not manufacturing defects. Software does not suffer from corrosion, wear-and-tear -- generally it will not change until upgrades, or until obsolescence. So once the software is shipped, the design defects -- or bugs -- will be buried in and remain latent until activation.

Software bugs will almost always exist in any software module with moderate size: not because programmers are careless or irresponsible, but because the complexity of software is generally intractable -- and humans have only limited ability to manage complexity. It is also true that for any complex systems, design defects can never be completely ruled out.

Discovering the design defects in software, is equally difficult, for the same reason of complexity. Because software and any digital systems are not continuous, testing boundary values are not sufficient to guarantee correctness. All the possible values need to be tested and verified, but complete testing is infeasible. Exhaustively testing a simple program to add only two integer inputs of 32-bits (yielding 2^64 distinct test cases) would take hundreds of years, even if tests were performed at a rate of thousands per second. Obviously, for a realistic software module, the complexity can be far beyond the example mentioned here. If inputs from the real world are involved, the problem will get worse, because timing and unpredictable environmental effects and human interactions are all possible input parameters under consideration.

In many if not most organizations, formal software testing can easily be tailored to the application being examined and has only two real prerequisites.

♦ Business requirements and design documents that allow development of a test plan

♦ People who understanding how to write and carry out a test plan

SOFTWARE TESTING—WHAT, WHY, AND WHO

WHAT IS SOFTWARE TESTING?

Software testing is a process of verifying and validating that a software application or program

1. Meets the business and technical requirements that guided its design and development, and

2. Works as expected.

Software testing also identifies important defects, flaws, or errors in the application code that must be fixed. The modifier “important” in the previous sentence is, well, important because defects must be categorized by severity.

During test planning we decide what an important defect is by reviewing the requirements and design documents with an eye towards answering the question “Important to whom?” Generally speaking, an important defect is one that from the customer’s perspective affects the usability or functionality of the application.

The quality assurance aspect of software development—documenting the degree to which the developers followed corporate standard processes or best practices – assuring quality is not a responsibility of the testing team. The testing team cannot improve quality; they can only measure it, although it can be argued that doing things like designing tests before coding begins will improve quality because the coders can then use that information while thinking about their designs and during coding and debugging.

Software testing has three main purposes: verification, validation, and defect finding.

♦ The verification process confirms that the software meets its technical specifications. A “specification” is a description of a function in terms of a measurable output value given a specific input value under specific preconditions. A simple specification may be along the line of “a SQL query retrieving data for a single account against the multi-month account-summary table must return these eight fields <list> ordered by month within 3 seconds of submission.”

♦ The validation process confirms that the software meets the business requirements. A simple example of a business requirement is “After choosing a branch office name, information about the branch’s customer account managers will appear in a new window. The window will present manager identification and summary information about each manager’s customer base: <list of data elements>.” Other requirements provide details on how the data will be summarized, formatted and displayed.

♦ A defect is a variance between the expected and actual result. The defect’s ultimate source may be traced to a fault introduced in the specification, design, or development (coding) phases.

WHY DO SOFTWARE TESTING?

Why test software? “To find the bugs!” is the instinctive response and many people, developers and programmers included, think that that’s what debugging during development and code reviews is for, so formal testing is redundant at best. But a “bug” is really a problem in the code; software testing is focused on finding defects in the final product.

Software testing answers questions that development testing and code reviews can’t.

♦ Does it really work as expected?

♦ Does it meet the users’ requirements?

♦ Is it what the users expect?

♦ Do the users like it?

♦ Is it compatible with our other systems?

♦ How does it perform?

♦ How does it scale when more users are added?

♦ Which areas need more work?

♦ Is it ready for release?

What can we do with the answers to these questions?

♦ Save time and money by identifying defects early

♦ Avoid or reduce development downtime

♦ Provide better customer service by building a better application

♦ Know that we’ve satisfied our users’ requirements

♦ Build a list of desired modifications and enhancements for later versions

♦ Identify and catalog reusable modules and components

♦ Identify areas where programmers and developers need training

WHAT DO WE TEST?

First, test what’s important. Focus on the core functionality—the parts that are critical or popular—before looking at the ‘nice to have’ features. Concentrate on the application’s capabilities in common usage situations before going on to unlikely situations. For example, if the application retrieves data and performance is important, test reasonable queries with a normal load on the server before going on to unlikely ones at peak usage times. It’s worth saying again: focus on what’s important. Good business requirements will tell you what’s important.

The value of software testing is that it goes far beyond testing the underlying code. It also examines the functional behavior of the application. Behavior is a function of the code, but it doesn’t always follow that if the behavior is “bad” then the code is bad. It’s entirely possible that the code is solid but the requirements were inaccurately or incompletely collected and communicated. It’s entirely possible that the application can be doing exactly what we’re telling it to do but we’re not telling it to do the right thing.

A comprehensive testing regime examines all components associated with the application. Even more, testing provides an opportunity to validate and verify things like the assumptions that went into the requirements, the appropriateness of the systems that the application is to run on, and the manuals and documentation that accompany the application.

Testing can involve some or all of the following factors. The more, the better.

♦ Business requirements

♦ Functional design requirements

♦ Technical design requirements

♦ Regulatory requirements

♦ Programmer code

♦ Systems administration standards and restrictions

♦ Corporate standards

♦ Professional or trade association best practices

♦ Hardware configuration

♦ Cultural issues and language differences

WHO DOES THE TESTING?

Software testing is not a one person job. It takes a team, but the team may be larger or smaller depending on the size and complexity of the application being tested. The programmer(s) who wrote the application should have a reduced role in the testing if possible. The concern here is that they’re already so intimately involved with the product and “know” that it works that they may not be able to take an unbiased look at the results of their labors.

Testers must be cautious, curious, critical but non-judgmental, and good communicators. One part of their job is to ask questions that the developers might find not be able to ask themselves or are awkward, irritating, insulting or even threatening to the developers.

♦ How well does it work?

♦ What does it mean to you that “it works”?

♦ How do you know it works? What evidence do you have?

♦ In what ways could it seem to work but still have something wrong?

♦ In what ways could it seem to not work but really be working?

♦ What might cause it to not to work well?

A good developer does not necessarily make a good tester and vice versa.

Key Players and Their Roles

Business sponsor(s) and partners –

♦ Provides funding

♦ Specifies requirements and deliverables

♦ Approves changes and some test results

Project manager –

Plans and manages the project

Software developer(s) –

♦ Designs, codes, and builds the application

♦ Participates in code reviews and testing

♦ Fixes bugs, defects, and shortcomings

Testing Coordinator(s) –

Creates test plans and test specifications based on the requirements and functional, and technical documents.

Tester(s) –

Executes the tests and documents results