Guidelines on Testing

Different types of testing :

Black box testing - not based on any knowledge of internal design or code. Tests are based on requirements and functionality.
White box testing - based on knowledge of the internal logic of an application's code. Tests are based on coverage of code statements, branches, paths, conditions.

Unit testing: the most 'micro' scale of testing; to test particular functions or code modules.
Integration testing - Testing of combined parts of an application to determine if they function together correctly. The 'parts' can be code modules, individual applications, client and server applications on a network, etc. This type of testing is especially relevant to client/server
Functional testing - Black-box type testing geared to functional requirements of an application; This type of testing should be done by testers. This doesn't mean that the programmers shouldn't check that their code works before releasing it (which of course applies to any stage of testing.)

Regression testing - re-testing after fixes or modifications of the software or its environment. It can be difficult to determine how much re-testing is needed, especially near the end of the development cycle. Automated testing tools can be especially useful for this type of testing. Load Testing
Acceptance testing - final testing based on specifications of the end-user or customer, or based on use by end-users/customers over some limited period of time.

Load testing - testing an application under heavy loads, such as testing of a web site under a range of loads to determine at what point the system's response time degrades or fails.
Recovery testing - testing how well a system recovers from crashes, hardware failures, or other catastrophic problems.
Security testing - testing how well the system protects against unauthorized internal or external access, willful damage, etc; may require sophisticated testing techniques

Alpha testing - testing of an application when development is nearing completion; minor design changes may still be made as a result of such testing. Typically done by end-users or others, not by programmers or testers.

Beta testing - testing when development and testing are essentially completed and final bugs and problems need to be found before final release. Typically done by end-users or others, not by programmers or testers.
What's a 'test plan”?

A software project test plan is a document that describes the objectives, scope, approach, and focus of a software testing effort. The process of preparing a test plan is a useful way to think through the efforts needed to validate the acceptability of a software product. The following are some of the items that might be included in a test plan, depending on the particular project:
• Title
• Objective of testing effort
• Software product overview
• Relevant related document list, other test plans, etc.
• Relevant standards or legal requirements
• Overall software project organization and personnel/contact-info/responsibilties
• Testing priorities and focus
• Scope and limitations of testing
• Test outline - a test approach by test type, feature, functionality, module, etc. as applicable
• Test environment validity analysis - differences between the test and production systems and their impact on test validity.
• Test tools to be used, including versions, patches, etc.
• Test script/test code maintenance processes and version control
• Problem tracking and resolution - tools and processes
• Reporting requirements and testing deliverables
• Software entrance and exit criteria
• Test site/location

What's a 'test case'?
• A test case is a document that describes an input, action, or event and an expected response, to determine if a feature of an application is working correctly. A test case should contain particulars such as test case identifier, test case name, objective, test conditions/setup, input data requirements, steps, and expected results.


What should be done after a bug is found?
The bug needs to be communicated and assigned to developers that can fix it. After the problem is resolved, fixes should be re-tested, and determinations made regarding requirements for regression testing to check that fixes didn't create problems

What steps are needed to develop and run software tests?
• Obtain requirements, functional design, and internal design specifications and other necessary documents.
• Determine project-related personnel and their responsibilities, reporting requirements, required standards and processes
• Identify application's higher-risk aspects, set priorities, and determine scope and limitations of tests
• Determine test approaches and methods - unit, integration, functional, system, load, usability tests, etc.
• Determine testware requirements(bug tracking, etc.)
• Determine test input data requirements
• Identify tasks, those responsible for tasks, and labor requirements
• Set schedule estimates, timelines, milestones
• Prepare test plan document and have needed reviews/approvals
• Write test cases
• Have needed reviews/inspections/approvals of test cases
• obtain needed user manuals/reference documents/configuration guides/installation guides, set up test tracking processes, Obtain and install software releases
• Perform tests
• Evaluate and report results
• Track problems/bugs and fixes
• Retest as needed
• Maintain and update test plans, test cases, test environment, and testware through life cycle

How to Enter Bug – Registry
• Complete information such that developers can understand the bug, get an idea of it’s severity, and reproduce it if necessary.
• Current bug status
• The application name
• The function, module, feature, object, screen, etc. where the bug occurred
• Full bug description
• Description of steps needed to reproduce the bug if not covered by a test case or if the developer doesn't have easy access to the test case
• File or the data that would be helpful in finding the cause of the problem
• Tester name
• Test date
• Bug reporting date
• Name of developer/group/organization the problem is assigned to
• Description of problem cause
• Description of fix
• Date of fix
• Retest date
• Retest results
• Regression testing results

A reporting or tracking process should enable notification of appropriate personnel at various stages. For instance, testers need to know when retesting is needed, developers need to know when bugs are found and how to get the needed information, and reporting/summary capabilities are needed for managers.

Note: Tester has to check-in the Bug Registry at the end of the day and must see to that all the bugs recorded in the registry is debugged

Important Testing Checklist

Software Testingis the Process of
Systematically Running a Software System Or Program
To uncover errors or defects

Software Testing in Organizations aiming for SEI Level 3 +:
•An acknowledged competitive advantage contributing to business success
•Moved from ad hoc to an engineering discipline
•Carrier for faster product delivery, with version controlled test plans and test cases
•Led to reliability prediction through defect metrics
•Focussed on defect elimination at the stage of defect induction itself

Testing Phases:
–Unit Testing
–Integration Testing
–System Testing
–Acceptance Testing

All Test Plans and Test Cases shall be Reviewed and Approved before Testing starts.

Unit Testing
–Lowest-level component test–Key foundation for later levels of testing
–Detects 65% - 75% of all bugs
–Stand-alone test
–Ensures the conformance to each unit in DDD

Integration Testing
–An incremental series of tests of combinations or subassemblies of selected components in an overall system
–Integration testing is incremental in that successively larger and more complex combinations of components are tested in sequence, proceeding from the unit level to eventually the full-integrated system
–Ensures the conformance to each module in HLDD

System Testing
–Highest level of application functionality testing performed by the systems group
–Ensures the conformance to the Functional Requirements as specified in the SRS

Acceptance Testing
–Independent test performed by the users or QA prior to accepting the delivered system
–Ensures the conformance to the Functional Requirements as specified in the URD

Testing Activities
•Test Case Design
•Review of Test Case Design
•Testing
•Recording Testing Results
•Review and Sign-Off of Testing Results
•Defect Reporting

Testing Documents and Records

Documents (SCIs)
Test Plan for the Project
Test Cases Documents
Records
Test Report
Defect Log
Review Reports

Unit Test Plan (UTP)
•Plan for the entire Unit Testing in the project
•Identifies the Units/Sub-units covered during Unit Testing
•Each feature in the DDD mapped to a Unit Test Case ID
•To be prepared at DDD phase itself
•The test cases are designed and documented •References to the Test Case document(s) shall be given in UTP
•Deliverables at Unit Testing Phase shall be identified in UTP
•Resources required shall be mentioned in Project Plan
•Schedules shall be planned

Testing coverage shall be identified in the UTP:
•Path Coverage
•Statement Coverage
•Decision (Logic/Branch) Coverage
•Condition Coverage
•Decision/Condition Coverage
•Multiple-Condition Coverage
•Functionality testing
•User interface testing
•Regression testing

Unit Testing CoveragePath Coverage
-Test cases will be written to cover all the possible paths of control flow through the program.
Statement Coverage
–Test cases will be written such that every statement in the program is executed at least once
Decision (Logic/Branch) Coverage
–Test Cases will be written such that each decision has a true or false outcome at least once.

Condition Coverage
–Test Cases will be written such that each condition in a decision takes on all possible outcomes at least once.
Decision/Condition Coverage
–Test Cases will be written such that each condition in a decision takes on all possible outcomes at least once, each decision takes on all possible outcomes at least once, and each point of entry is invoked at least once.
Multiple-Condition Coverage
–Test Cases will be written such that all possible combinations of condition outcomes in each decision, and all points of entry, are invoked at least once.

Integration Test Plan (ITP)
•Plan for the entire Integration Testing in the project
•Identifies the Modules/Sub-Modules covered during Integration Testing
•Each component of HLDD mapped to a Integration Test Case ID
•Recommended to prepare at HLD phase itself
•The test cases are designed and documented
•References to the Test Case document(s) shall be given in ITP
•Deliverables at Integration Testing Phase shall be identified in ITP
•Resources required shall be mentioned in Project Plan
•Schedules shall be planned

Testing coverage shall be identified in the ITP:
•Functionality Testing•User interface testing
•Dependency (API) testing
•Smoke testing
•Capacity and volume testing
•Error / disaster handling and recovery
•Concurrent execution testing
•Equivalence partitioning
•Boundary-value analysis
•Cause-effect graphing
•Error guessing
•Equivalence Partitioning
•The test cases are partitioned into equivalence classes such that

–each test case should invoke as many different input conditions as possible in order to minimize the total number of test cases necessary
–if one test case in an equivalence class detects an error, all other test cases in the equivalence class would be expected to find the same error
Boundary-value analysis
–Test cases explore boundary conditions
–Test cases consider input conditions as well as output conditions
Cause-effect graphing
–Technique to identify test cases by translating the specifications into Boolean logic network.
–This is a systematic method of generating test cases representing combinations of conditions
Error guessing
–The test cases are written both by intuition and experience to uncover certain probable type of errors
–The test case design is by the knack of “smelling out” errors

System Test Plan (STP)
•Plan for the entire System Testing in the project
•Identifies the features covered during System Testing
•Each feature in the SRS mapped to a System Test Case ID
•Recommended to prepare at SRS phase itself
•The test cases are designed and documented
•References to the Test Case document(s) shall be given in STP
•Deliverables at System Testing Phase shall be identified in STP
•Resources required shall be mentioned in Project Plan
•Schedules shall be planned
•Functionality Testing
•User interface testing
•Usability testing
•Volume Testing
•Stress Testing
•Security Testing
•Performance Testing
•Installation and upgrade testing
•Standards conformance testing
•Configuration testing
•Network and distributed environment testing
•Forward / backward compatibility testing
•Reliability Testing
•Error / disaster handling and recovery testing
•Serviceability Testing
•Documentation testing
•Procedure testing
•Localization testing

Acceptance Test Plan (ATP)
•Plan for the entire Acceptance Testing in the project
•Identifies the features covered during Acceptance Testing
•Each feature in the URD mapped to a Acceptance Test Case ID
•Recommended to prepare at URD phase itself
•The test cases are designed and documented
•References to the Test Case document(s) shall be given in ATP
•Deliverables at Acceptance Testing Phase shall be identified in ATP
•Resources required shall be mentioned in Project Plan
•Schedules shall be planned
•Alpha Test•Beta Test
•All the tests covered under system testing can be repeated as per user requirements


Terminology

Functionality Testing
–Functionality testing is the determination of whether each functionality mentioned in SRS is actually implemented. The objective is to ensure that all the functional requirements as documented in the SRS are accomplished.
User Interface Testing
–Focus is on testing the user interface, navigation and negative user behavior
Concurrent Execution Testing
–Focus is on testing with simple usage, standard usage and boundary situations

Volume Testing
–Volume Testing is to ensure that the software
•can handle the volume of data as specified in the SRS
•does not crash with heavy volumes of data, but gives an appropriate message and/or makes a clean exit.

Example:
•A compiler would be fed an absurdly large source program to compile

Stress Testing
–Executes the software in a manner that demands resources in abnormal quantity, frequency and volume and verifies that the system either performs normally, or displays message regarding the limitations of the system
Usability Testing
–Usability testing is an attempt to uncover the software usability problems involving the human-factor
Example:
•Are the error messages meaningful, easy to understand?
•Security Testing
–Attempts to verify that protection mechanism built into a system will in fact protect it from improper penetration
Example:
•a database management system's data security mechanisms

•Performance Testing
–Tests the run-time performance of software within the context of an integrated system
Example:
•the response times under certain configuration conditions.Installation and Upgrade Testing
–Focus is on testing whether the user would be install and upgrade the system
Configuration Testing
–Configuration testing includes either or both of the following:
•testing the software with the different possible hardware configurations
•testing each possible configuration of the software
Network and Distributed Environment Testing
–Focus is on testing the product in the required network and distributed environment
Dependency (API) Testing
–Focus is on testing the API calls made by the system to other systems
Localization Testing
–Focus is on testing problems associated with multiple languages support, conversion and also hardware aspects
Reliability Testing
–The various software testing processes have the goal to test the software reliability. The "Reliability Testing" which is a part of System Testing encompasses the testing of any specific reliability factors that are stated explicitly in the SRS
Error / Disaster Handling and Recovery Testing
–Forces the software to fail in a variety of ways and verifies that the system recovers and resumes processing
Serviceability Testing
–Serviceability testing covers the serviceability or maintainability characteristics of the software
Example:
•service aids to be provided with the system, e.g., storage-dump programs, diagnostic programs
•the maintenance procedures for the system
Documentation Testing
–Documentation testing is concerned with the accuracy of the user documentation. This involves
•Review of the user documentation for accuracy and clarity
•Testing the examples illustrated in the user documentation by preparing test cases on the basis of these examples and testing the system
Smoke Testing
–Focus is on testing the system releases as it is being built to uncover critical and showstopper errors
Configuration Testing
–Configuration testing includes either or both of the following:
•testing the software with the different possible hardware configurations
•testing each possible configuration of the software
Procedure Testing
–If the software forms a part of a large and not completely automated system, the interfaces of the developed software with the other components in the larger system shall be tested. These may include procedures to be followed by
•The human operator
•Database administrator
•Terminal user
–These procedures are to be tested as part of System testing
•Standards Conformance Testing–Focus is on testing whether the product conforms to prescribed and published standards
•Alpha Test–Within a vendor, the last level of internal test prior to a limited beta release
•Beta Test
–The vendor’s equivalent of a pilot test (usually with a greater number of participating sites than for a pilot)
•Alpha Test–Within a vendor, the last level of internal test prior to a limited beta release
•Beta Test
–The vendor’s equivalent of a pilot test (usually with a greater number of participating sites than for a pilot)
•Equivalence Test
–A test using only a small sample of all possible test conditions, but ones which are chosen to uncover almost as many defects as an exhaustive test would have uncovered
– The key question is :
•What subset of all possible test cases has the highest probability of detecting the most errors ?
Regression Testing:
Comprehensive re-test of an entire system :
•after system delivery, when a modification has been made, or
•before delivery and at the end of system test , after all test cases have been passed, but not passed together against the final version of the system product
Regression Testing:
•Regression testing requires that a regression test bed (comprehensive set of re-usable system test cases) be available throughout the useful life of the delivered system
•Functional acceptance tests form the core of this regression test bed.
Regression Testing:
•The regression test must be maintained, to keep it aligned with the system as the system itself evolves . This maintenance may not be a trivial effort.
•If a careful determination is made that only portions or subsystems will be affected by a particular change (i.e., the subsystems are decoupled and insulated), then only a partial regression re-test of the affected portions is absolutely necessary
Regression Testing:
•If errors are detected during on-going system operation, then test case(s) should be added to the existing regression test bed to detect any possible further re-occurrences of the error or related errors
•The additional effort to build a regression test facility is relatively minor if it is done during system development : the attitude should be that test cases are designed and organized to have an on-going life after system delivery, not one-time throw-aways
•The decision to perform any particular regression test is based on an analysis of the specific risks and costs
Regression Testing:
•Regression tests are more manageable and cost-effective they are coordinated with scheduled releases, versus a piece-meal approach to system modification
•Full regression testing should always be performed when the overall system architecture has been affected
•A guideline for setting the boundaries of regression testing: include any interdependent, integrated applications; exclude tangential applications.
•Assessing Test Effectiveness•Techniques that provide means to assess the effectiveness, coverage and robustness of a set of test cases:
–Error Seeding
–Mutation Analysis

Note :
These techniques are talked about more than they are practiced. Many people see them as somewhat impractical because they can be difficult to apply well.

Mutation Analysis Approach:
Create numerous (minor) variations of the baseline system or program. Then test each variation with the original, unchanged set of test cases, and determine how many of the tests behave as if the product was not changed
Purpose:
Mutations which do not test differently from the baseline product are carefully examined, to determine whether in fact the tests are inadequate

Manual Web Testing Points to be Consider

Module Testing

Function Testing / Black box Testing

Add / Edit

Delete

Search

Sort

View/ paging

GUI (Graphical User Interface) Testing

Web page title

Navigation

Broken link

CSS

Menu

Focus / Tab order

Tool tip

Err messages

Spelling

Database testing

Check data added as per user submitted

Apply query

Performance testing

Total no. of user

Internet speed

Insert multiple data in database, check effects on UI side

Browser compatibility

Consider the layout, alignment, grid line, css, err msg

Integration testing

Check interaction, Interface between modules

Check CSS, Navigation

System testing

Test entire application before & after upload