The trend nowadays is to outsource software testing. Offsite software testing companies are everywhere; they're headquartered in the United States and in technology-rich countries such as India, Singapore and Japan. Company literature boasts the tremendous time and money saving features of outsourcing software testing while stressing high quality work you can count on. But is it really beneficial to outsource software testing? There's no standard answer to this potentially expensive proposition.

How can you decide if it makes sense to outsource software testing? You've got to begin by carefully analyzing your needs as they relate to this critical phase of software development. Software testing means many things to many people. The main objective of software testing is to ensure the software functions as specified before going into production. But it also means making sure the software is intuitive and easy to use. Let's face it – no matter how great the software product is, if it's too cumbersome to use, its usefulness will be short-lived.

Before you make the decision to outsource software testing, it's important to clarify what you want from the outsourcing company. As their popularity grows, so do the services these companies provide. Many also provide software design services. Or, they will plan, but not implement a software testing process for your organization. And they will even send individuals to your site to take care of integrating the software testing tools you purchase with your hardware. In general, regardless of your exact software testing needs, if you've got the money to pay for it, you'll find an outsourcing company willing to do it.

One reason companies decide to outsource software testing is to eliminate the learning curve. Software testing companies not only know about the different software testing tools, their employees already know how to use these tools. If you work in software development, you know how tight production schedules are. Anything that saves time helps keep projects on schedule. Software delays are costly so when weighing the costs vs. benefits of outsourcing software testing, determine how project delays affect your bottom line.

Oftentimes software development houses cannot afford to hire and maintain a full-time development team and an in-house software testing team. For these companies, the decision to outsource software testing is an easy one.

The decision to outsource software testing makes sense for many companies. If you do decide to outsource, get your expectations and your costs in writing before work commences. Agree on a method of reporting bugs back to you to ensure that the information you receive back is useful. Also discuss such topics as regression testing, script maintenance, and load tests beforehand.

When you outsource software testing, it's important to build a solid relationship with the company for this current project as well as for future projects. If possible, speak with clients who have given their software testing work to the outsource company. As tempting as it may be don't let cost be the deciding factor. Remember, cheaper isn't always better.

Bug tracking. It's the Achilles' heel of software development. It's easy to report bugs found in software, but unless the bugs are properly tracked, all the way through to their resolution, one can never be sure whether these nuances truly have been swatted. Like criminals on the run, without proper bug tracking procedures, those responsible for fixing bugs spend their days living on the edge and forever looking over their shoulders. For they know that at any time, those bugs that weren't tracked can reappear, and create chaos.

But exactly who is responsible for bug tracking? Is this the responsibility of only one person? If so, then is the person who reported the bug responsible? Or is the person who supposedly "fixed" the bug responsible? Or is the QA person responsible? Maybe all of these people are in some way responsible for bug tracking.

Assuming the software has not yet gone into production, most often the responsibility for bug tracking lies with the software tester. Software testers have the job of verifying that the keystrokes they push produce the desired outcome. When this does not happen, testers must report these findings back to the developers.

Note however, that software testers are not required to fix software discrepancies. But generally, they are responsible for tracking a known bug. Once a programmer reports back that the bug has been corrected, the software tester must confirm this. They must reproduce the keystrokes that initially led them to the software discrepancy, and confirm whether (a) the bug has not been corrected, or (b) it has been corrected, or (c) it has been corrected, however, now something else is not working correctly.

If choice (a) was the typical outcome, bug tracking would not be such an enormous task. However, more often than not, one of the latter two choices is the outcome. With all that programmers have to do, they cannot be expected to track the status of all of the software's bugs in their heads. Nor can a software tester be expected to maintain this amount of information in their heads, either.

Bug tracking involves more than just assigning the problem from person to person to person. You're not playing a game of "hot potato"; you're trying to get working software released to those who need it. Before a programmer can even begin to "fix" a bug, he needs to know why it occurred. He needs to know the steps which led to the error so he knows where to go in the code to look for the problem. When reporting bugs, it's also important for the tester to say what should have happened, and also what actually did happen.

A colored-coded sticky note bug tracking system might work conceptually, but not in reality. And don't even think about sending email messages back and forth as an effective bug tracking method. When you're involved with serious software development, bug tracking software is the only thing software testers and developers can rely on.

Is automated software testing really worth the time and effort? The simple answer is "yes;" it's a critical part of the software cycle. The more complicated answer is that it's only beneficial if properly implemented.

Most automated software testing tools available today won't work on your software right out of the box. They first need some configuration. If you're lucky, you'll properly configure the automated software testing tool quickly and without pulling too many of your hair follicles out. Oftentimes, however, more time is spent on the configuration step than would be necessary to manually test the software. By the time you're finished you easily could have built your own automated software testing procedure yourself, right? That depends.

Automated software testing is an effective and necessary part of the software development cycle. Interestingly, a growing group believes that focusing on software testing after source code has been written is inefficient and it's too late. Instead, this group believes focusing on the design phase to make sure all involved clearly understand how the software must work would significantly reduce the testing phase and result in better management of the development team.

This does make sense but until these ideas become mainstream, automated software testing will continue. Given that there are so many testing tools available and that each tests different aspects of the software and/or has its own implementation, developing an automated software testing strategy is one way to reign in this process.

An effective automated software testing strategy helps clarify several important objectives. First, a strategy helps determine what is to be automated and how. It can also clarify where and how the ever-growing collection of scripts will be maintained. A strategy also forces those involved to weigh the costs versus the benefits of automated software testing to make sure the time, effort and money are well-spent.

Automated software testing is itself a software development project. If the right people with the right skill set are not put on this portion of the project, expected results won't be achieved. In other words, a software "tester" cannot also be the software "developer". Don't be fooled into thinking that automated software testing is something to be tackled whenever there's time to spare; it's a full-time effort to be carried out by a full-time software tester.

What then, does automated software testing actually test? Many of these testing tools work by creating or generating "test cases". Test cases are run against the software to see if the end results are as expected. If the results are not as expected, there is a problem with the software. Identifying discrepancies and correcting them early on makes sense. Building source code onto code that is not functioning properly compounds problems and makes them more difficult to trace back and fix.

The methodologies utilized by automated software testing tools vary based on the platform and the complexity of the code; another reason why an automated software testing strategy is so important.

Do you know which bug tracking software is best for your development environment? Besides the obvious issues of compatibility, do you know what other features are important? Don't let the sales pitches confuse. Before you purchase bug tracking software, know what your software developers and software testers want and need. After all, they're the ones who will be using this tool most.

Reliable bug tracking software is itself bug-free and stable. Select a product that has a proven track record of success. If possible, talk with actual users to see whether or not the software is meeting their needs.

You want your bug tracking software to be customizable so that it works the way your development team works. But you don't want this task to be so difficult that you have to first take a course to learn how to accomplish this. A good interface is simple and straightforward, and at the same time, intuitive.

Select bug tracking software that fits your budget and works on the hardware configuration you already have in place. If it requires additional, costly hardware to operate, keep looking.

As for features, including the ability to prioritize a reported bug helps managers allocate resources more efficiently. It's important to know additional information about each reported bug, and effective bug tracking software facilitates adding notes. For example, it's important to reproduce the steps that led to the discovery of the bug, the actions that should have happened when the bug was discovered, the actions that actually did happen, and the version that was used at the time of discrepancy. As the bug works its way from team member to team member, the good software tracks that history and allows additional notes to be appended at any time.

Reliable bug tracking software doubles as a management tool. It produces useful reports including work that has already been assigned to each team member. Such a report helps management reprioritize and reassign work when necessary. Also important is a built-in method for notifying specific team members that new work has been assigned to them. As each team member completes work on the bug, they need the ability to update the status of the bug. The bug itself needs to be reassigned to another individual or marked as closed.

Be sure that the bug tracking software you select allows customization of the steps necessary to close an open bug report. Also make sure the software allows you to designate the person(s) who has the authority to close a bug report.

And if the bug tracking software does not allow an easy way to regularly back up its data, find another package. Bug reports are too vital to lose!

Once bug tracking software is installed, don't allow any member of the team to circumvent the software. Regardless of the features, bug tracking software can't keep proper track of bugs if it's not being used properly. Until everybody is using the same tracking system, bugs will continue to run rampant, something you can't afford.

Don't be fooled into thinking that software testing procedures are a waste of time and effort. Documented procedures for testing software must be in place before testing begins. Lack of software testing procedures or using procedures that are not clearly and completely defined often results in time delays and cost overruns; two things software marketers cannot afford, especially when staying ahead of the competition is crucial.

To be useful, software testing procedures must encompass all aspects of the software testing process. It's vitally important that the procedures define the people who will be involved in the testing process, the skill set of each team member, and their availability for the duration of the testing cycle. For the software cycle to remain on track, the software testing procedure must also delineate a carved-in-stone testing schedule including dates of important milestones.

To be effective and useful, software testing procedures require much more information. Procedures must define guidelines for creating test cases. Include in this section the processes for running these test cases, and the creation, testing and maintenance of testing scripts. Also include here a thorough explanation of all other approaches that will be used during the software testing phase. It addition to a discussion of the specific processes for software testing, be sure to define the specific parts of the software to be tested during the stated timeframe.

Software testing procedures must also define the hardware and software resources that are needed to keep the testing process on track and the length of time each will be needed. If the resources are not available, it won't be possible to meet project deadlines.

One aspect of software testing procedures that can be difficult to agree on is a definition of acceptable testing results. It's imperative, however that a firm decision be made here. No software testing procedure should ever begin without a clear understanding of what results are and are not acceptable.

Finally, there needs to be a process for reporting the results of the software testing phase, including the reporting of bugs. Software testers are not responsible for fixing software problems; only reporting them. However, they are responsible for verifying that previously reported bugs have been fixed. It's imperative that this process also be detailed in software testing procedures. Equally important is discussion of and procedures for regression testing. Software testers need to ensure that new fixes, and any software changes and enhancements do not adversely affected previously tested parts of the software.

Software testing is a critical component of the software development cycle. And software testing procedures are critical to the success of the testing phase. Software remains in a perpetual state of change which is why software testing, whether manual or automated, is so vital to a software product's success. Typically, 30 – 40% of the software engineers employed at larger software development companies work on software testing and each of these individuals needs to understand his or her role in this ongoing process. Software testing procedures help ensure that they do.

Software Testing Tools – Are they really useful? There's no easy answer to this question. Software testing is a crucial part of the development process, and tools definitely help with this oftentimes overwhelming task. But to be useful, software testing tools must support the testing process.

What does this mean? It means you need to start by understanding the different phases of software testing. For example, do you understand the difference between black box and white box testing? Black box confirms only that the software meets its stated requirements and functions accordingly. White box testing looks at the actual software code to ensure paths, conditions, code statements and branches are written properly.

Do you know which software testing tools work best for unit testing, integration testing or system testing? Each of these testing processes addresses a different aspect or view of the software. Starting to understand that software testing tools are not a "one size fits all" solution?

Some rounds of software testing are better accomplished using humans, not software testing tools. Do you know which ones? Functional testing, alpha testing, acceptance testing and usability testing fall into this category. To confuse matters even a bit further, you've got to make sure you're using the right type of human for the different "human" software testing processes.

Software testers work closely with developers throughout all stages of development and use software testing tools. End users are those individuals for whom the software has been created. Beta testers are humans with more technical backgrounds (generally) who get involved in software testing just before the software is ready to be released into production. They look for last minute bugs and functionality issues. User acceptance testing ensures the resulting software is "user friendly" and satisfies the end users' needs; also important before the software goes into production.

There are even more phases of software testing that go on during the software development lifecycle. Determining which of the phases are better accomplished using software testing tools and which are better left to human intervention takes effort. In larger development houses, the IT department has a pretty good grasp on this. And they're the ones who ultimately decide which of the hundreds of software testing tools on the market are best for their development/testing teams; definitely not an easy task.

Software testing tools are themselves software. As such, each of these testing tools undergoes the whole development, testing, maintenance and upgrade cycles that other types of software do. So before making purchasing decisions, it's best to try out these products, talk with existing users, research the product's track record, and know how much configuration you'll need to do to get the product up and running.

And remember, software testing tools can't work miracles. They cannot make poorly designed software better. They can't do anything about unrealistic development schedules. And most importantly, software testing tools will never work properly if management or others are allowed to continually change software specifications after development has begun.

The last thing busy IT departments need is more paperwork. But software testing white papers are a different type of paperwork. Designed to educate, not frustrate, you'll find that software testing whites papers have been written for almost all of the software testing products currently available for purchase.

Software testing white papers are technical documents that generally are written by those with technical backgrounds, not a company's marketing department. What does that mean to you? It means you can rely on software testing white papers to provide you with the factual information you need to make decisions without getting lost in all the marketing hype.

If you're involved with software testing, not only can white papers help you make informed purchasing decisions, they can help you better understand (and hopefully resolve) some of software testing's most troublesome issues.

You'll find software testing white papers to help you get your software testing process back together in one centralized location; a big problem in these days of outsourcing, high employee turnover and limited resources. Or perhaps a white paper that focuses on the prevention of software testing troubles before they emerge by providing an in-depth discussion of better ways to monitor the process and better ways to resolve those problems that do emerge could be useful to your organization.

Are you tired of the stress involved with reacting to problems that prop up after software has already gone into production? If you feel it's time for your IT department to switch from being reactive in nature to proactive, you're not alone! You'll find plenty of software testing white papers offering advice in this seemingly unmanageable area.

Software testing white papers that have been written about a particular product incorporate several important components. Most will begin by stating the problem the product helps correct as well as providing pertinent background information. The body to the white paper is devoted to how the product works to solve the stated problem.

Software testing white papers provide a great deal of information regarding the product testing process. The paper explains in detail how the tests were set up, and provides detailed information regarding the test results including information about how the results were verified or validated. Examples and/or lines of code are included as necessary to support product findings. Oftentimes, the writer's credentials are included in the white paper as are a list of resources used to compile the documentation.

Generally quite long, software testing white papers are not for the faint of heart. They're full of technical jargon, technical references, and other information that those with technical backgrounds will find particularly useful and enjoyable to read.

Software testing continues to grow more complex even as incredible advancements in coding and coding efficiency are being made. Look to software testing white papers to help you tame some of your software testing beasts. Written by technical experts for technical experts, software testing white papers can help you get your testing back under control.

The proper testing of software takes a lot of work, and therefor a lot of time. While I don't agree with this tactic, when projects get delayed, the time to test the system is one of the most likely candidates to take a hit when squeezing the schedule. „Two weeks delayed? We test two weeks shorter, and presto, back on schedule again."

When facing a short time frame available for testing purposes, you got to make the best the time and resources available. A software test strategy that takes this into account is risk and requirements based testing.

In this strategy we assume that it's undoable to test everything. From a economic point of view it doesn't even make sense; spending lots of time to parts of a system where the changes of having a bug are low, or even if a bug has been found, where the impact of it will be low. Risk and requirements based testing helps you to determine what to test first, in which sequence, so you spend the time you have to the parts that really matter.

The strategy starts with a risk analysis to determine the functions (requirements) with the highest risk, and plan your test activities guided by this analysis.

To help you identify the risks involved in all your requirements, consider the following aspects:

• Functions often used by the users


• Complex functions


• Functions that have a lot of updates or bug fixes


• Functions that require high availability


• Functions that require a consistent level of performance


• Functions that are developed with new tools


• Functions that require interfacing with external systems


• Functions with requirements with a low level of quality


• Functions developed by more programmers at the same time


• New functions


• Functions developed under extreme time pressure


• Functions that are most important to the stakeholders


• Functions that reflect a complex business process

Related links

SQAtester.com - Your Online Software Testing & Resource Center

To have a proper testing of software, the project manager, or test manager, should plan the test activities in advance. To assist with this task, I list on this page an outline for a software test plan.

Test organization

This section of the test plan describes the organization around the testing activities. Responsibilities and used resources should be named under this topic.

Communication & procedures

An overview of the communication during the testing activities should be provided in this section. Also the procedures during the software test for bugfixing, version control e.a. should be listed.

Test strategy

This section contains on overview of the test strategy used for the software testing, acceptance criteria and a statement to which level will be tested.

Test items

An overview of the functions to be tested and their priorities should be listed in this section of the software test plan.

Test deliverables

A description of the products used by testing.

• Test input


• Test reports


• Infrastructure to be used


• Progress reports

Test activities

An overview of the activities needed for testing, e.g.

• Installation infrastructure


• Writing of test scripts


• The actual performance of the test


• Monitoring progress


• Creation of reports

Schedule

The actual planning of the software test activities and resources used.

Software testing is probably the most complex task in the software development cycle. It often takes longer to test the software than it does to write the code. The problems discovered during the software testing phase add more time onto the coding phase, resulting in further delays in the product's release, and so this vicious cycle goes.

It's nearly impossible to attribute the problems that arise during the software testing cycle to any single factor. Before going any further, it's important to clarify one point. Software that works, but not the way it is supposed to work, is not considered an error in the coding (also known as a bug). Rather, this situation is the result of an error in the design phase. Software engineers refer to these differences as a fault vs. a failure. Faults can turn into failures, but that's a subject for another time.

So why is software testing such a time-consuming and frustrating process? One factor is the complexity of the software being developed. The more complex the project is, the more there is to test. Also, complex projects typically involve multiple development team members, including those working directly for the company and those working as sub-contractors.

Software testing troubles directly attributable to the human factor include poorly documented code and employee turnover. And if the person who is not properly documenting the project is the same person who leaves the company midway through the project cycle, the problem quickly compounds.

Another software testing difficulty arises when those developing the software are the same ones testing that software. These individuals have a much higher level of understanding about software and computer system. They're not likely to make the same types of mistakes during the software testing phase that end users might make using the finished software. For example, software engineers understand that you never press the power button before you properly close all applications. End users, in their rush to burst out of the office at 5:00 pm just want the computer off, and some won't wait until every application closes properly.

Plenty of software testing applications are available to help during the software testing phase. These products are designed to facilitate and enhance the software testing phase, provided they can be made to work with the software being tested.

The main purpose of software testing is to discover software failures. As failures are discovered, they can be corrected. However, software testing cannot guarantee that the software is problem-free. Testing finds many bugs, but even the most extensive testing processes can't fix failures that are not uncovered. Never rely on software testing as proof that no problems exist. That's considered software verification, an entirely different process.

Regardless of the difficulties involved with software testing, one truth remains. Software failures discovered early on cost far less to correct than those that occur in latter stages of software development, or worse, after the software has been released to the general public.