Testing and code coverage

More times than I can count, I have heard someone say “We’re trying for XX percent code coverage with our unit tests”. If XX is 100% are you guaranteed to have no defects?

The first thing that comes to my mind when people start rambing on about unit tests is: What about integration and system testing? There are a number of blog entries out there that talk about the difference between the types of testing. It suffices to say that pure unit testing, even at 100% code coverage, is only going to reveal a small percentage of an application’s defects.

So what are you actually trying to do?

Test for a reason — don’t just test to test. You can get into a cycle where you’re writing unit test apon unit test and still releasing applications that are defect riddled. First you need to define a quality metric (a way to measure the quality of the application). “As few bugs as possible” isn’t it! Ask yourself questions such as: What’s a bug or defect? and How is the priority and severity of a defect determined? (e.g. Are spelling errors bugs? They may not be to a developer, but they may be show stoppers to a client.) Everyone wants to jump in and test up the wazoo but the problem is, if you test the wrong things and you don’t know what a bug or defect is then your testing is potentially wasting time.

Let’s put this another way: let’s say you have 80% coverage with all of your tests. What’s preventing that 20% that you’re not testing from containing 90% of the defects and 100% of the P1 show-stoppers?

Your goal is to identify the cross product of the areas of highest quality risks and the areas that are most important to the user of the application. These areas must have enough testing to ensure that the desired level of quality is met. For example, what if the 20% that you did not test just so happens to be the login page? The client cannot log into the application! So what’s the point of testing or even writing the rest of the app? This is an obvious overtrivialization as the login page is an obvious element to test. But in reality, the show stoppers commonly fall into a trivial category. A number of projects that I’ve performed triage in the past suffered from this problem to an alarming degree. Months of work and testing when into components of the product that, at the end of the day, were not high on the clients list and were overshadowed by trivial defects.

Plan ahead. Identify what determines quality in your application. Test effectively.

Advertisements

10 comments

  1. hi
    sam mainly the testing life cycle is
    TEST PLANNING
    TEST DEVELOPMENT
    TEST EXECUTION
    RESULT ANALYSIS
    DEFECT MANAGEMENT
    SUMMARY REPORTS
    mr explaining each and every phase is very difficult during the time fact
    so some othertime i’ll explain
    in TEST PLAING we have to prepare test strategy plan,test schedule,effort estimation and many more factors.
    in TEST DEVELOPMENT identify test scenario,develop test cases,write test scenario
    u’ll search some other sites man it is to difficult to type all this things
    bye

  2. hi guys,
    how r u all? and i had a doubt in testing plz give me reply
    wat is meant by “pages per second”?
    ok bye

  3. Hi,
    1.Could u suggest tool which help to test HTTP/FTP, SMTP protocols. actuall we are testing manually, too fasten testing process.
    our product test just acts as proxy server, whatever the traffic from out network, we have validate through some polices an nd rules.
    2. i need help in QTP tool to learn and workout with tool.
    3. Currently i’m working on instalshield of our product, here the process is , every week we will get Build , and i test all the product to install and Uninstall , verify their funcationalities. i think it is time consuming. i ant to make some funtionalites to automate.
    please waiting for u r reply.
    regards,
    shanti kumar reddy

  4. Sir
    My question is purely related to Configuration Management. In the place where I started I was asked about this to explain to the other people in the group and I told about the check in and checkout of files on the server and also about the build of the files by the CM team.
    Can you let me know if CM is more than this or was my explanation on the right track.
    Waiting for answer.
    Thankyou

  5. I have made a new forum available that will take the place of this posting. I hope that everyone will find the interface much easier to use for asking and answering questions.
    http://www.realityinteractive.com/forum
    You will need to register with the forum to use it. I was not able to automatically transfer over the users from this site.
    There is no easy way to integrate the questions asked and answered here into the forum. For that I apologize.
    I have locked out this message to prevent further postings.


Comments are closed.