Black Box Testing

What is Black Box Testing?

Unlike White Box Testing, which is focusing on the code or syntax stuff, Black Box Testing is focusing on functional requirements, and it compliments White Box Testing. it only concerns with testing the specification, and cannot guarantee that all parts of the implementation have been tested.

Thus black box testing is testing against the specification and will discover faults of omission, indicating that part of the specification has not been fulfilled. White box testing is testing against the implementation and will discover faults of commission, indicating that part of the implementation is faulty. In order to fully test a software product, both black and white box testing are required.

Black box testing attempts to find out: (1) incorrect or missing functions, (2) interface errors, (3) errors in data structures or external database access, (4) performance errors, and (5) initialisation and termination errors.

 

Advantages of black box testing?

  • The test will be unbiased because the developer and tester are independent of each other;
  • The tester does not need to know deeply the specific programming languages;
  • The test can be done from the perspective of end user, instead of developer.

 

Disadvantages of black box testing?

  • The test can be redundant if the developer has already run a test case;
  • The test cases are sometimes difficult to design;
  • Testing every possible input stream is unrealistic because it would take an inordinate amount of time. Therefore, many program paths will go untested.
  • Only test the performance or functions, sometimes the errors exist in the code cannot be found. Maybe will cause further problems which cannot be disclosed now.

 

Black Box Testing methods:

  1. Equivalence Partitioning
  2. Boundary Value Analysis.
  3. Cause Effect Graphing Techniques.
  4. Comparison Testing

 

Equivalence Partitioning:

is a typical black box testing technique which divides the input domain of a program into classes of data from which test cases can be derived. Equivalence partitioning method strives to define a test case that uncovers classes of errors, thereby reducing the total number of test cases that must be developed.

  • equivalence partitions are <10,000, 10,000-99,999 and >100, 000
  • Test cases would be selected from each equivalence class i.e. 00000, 09999, 10000, 99999, 100001

Choose the most representative one for each set.

Boundary Value Analysis:

is a black box testing method that where test cases are designed to test the boundary of an input domain. Studies have shown that more errors happen on the “boundary” of an input domain rather than on the “center area”.

So here, we can select the minimum and maximum values of an input and output range to test.

Example?

  • 10,000, 99,999,100,000

Cause Effect graphing Techniques

The basic idea of this method is that, a “cause” if an input condition, and an “effect” is a specific sequence of computations to be performed. A cause-effect graph is basically a directed graph that describes the logical combinations of causes and their relationship to the effects to be produced. It can help testers to select test cases to chekc if the program will produce right effect for every possible combination of causes.

Cause Effect Graph

(source: https://www.tutorialspoint.com/software_testing_dictionary/cause_effect_graph.htm)

Why we are using cause-effect graph/When should we use/What circumstances?

  • To identify the possible root causes, the rease for a specific effect, problem, or outcomes;
  • To relate the interaction of the system among the factors affecting a particular process of effect;
  • To analyse the existing problems so that corrective action can be taken at the earliest.

Benefit:

  • It Helps us to determine the root causes of a problem or quality using a structured approach.
  • It Uses an orderly, easy-to-read format to diagram cause-and-effect relationships.
  • It Indicates possible causes of variation in a process.
  • It Identifies areas, where data should be collected for further study.
  • It Encourages team participation and utilizes the team knowledge of the process.
  • It Increases knowledge of the process by helping everyone to learn more about the factors at work and how they relate.

Comparison Testing:

which comprises of comparing the content of multiple objects, e.g. files, databases, against actual results.

  • One can easily detect the underlying weakness and strengths of the software
  • It helps to realize the design structure of competing products as well as allows you to use them as a benchmark for future enhancements and improvements

 

(Some good sources and articles about black box testing and test cases writing:

http://www.cnblogs.com/Jackc/archive/2009/02/24/1397433.html

http://agile.csc.ncsu.edu/SEMaterials/BlackBox.pdf)

Root Cause Analysis (RCA)

How do you analyse the root cause for a defect?

As a software tester, we can use Root Cause Analysis.

Root Cause Analysis (RCA) is an approach used in software testing to analyse and identify the root causes of defects and address them, instead of treating the symptoms. If some problems happen, instead of just fixing it, we can also investigate and try to fix the underlying cause so that this kind of problems will be prevented from the next or further phases.

How to do the RCA?

Five whys technique: whose goal is to determine the root cause of a defect or problem by repeating the question “Why”. Each “why” forms the basis of the next question. For example:

6How to do RCA?
Why we've received an issue?
Why Save button is not working on users page?
Why does specific symbol affect...

(Source: https://www.slideshare.net/DavidGevorgyan1/root-cause-analysis-of-defects)

The answer to these questions will give us the exact phase, where the defect actually exists. Then, we can come up with “What” question:

What will you do to avoid this in future?

So now we can prevent the same defect to arise again by implementing the answer of this question.

Why doing this?

Doing the RCA accurately helps to prevent defects in the later releases or phases.

Examples?

Problem Statement: You are on your way home from work and your car stops in the middle of the road.

1. Why did your car stop?
– Because it ran out of gas.

2. Why did it run out of gas?
– Because I didn’t buy any gas on my way to work.

3. Why didn’t you buy any gas this morning?
– Because I didn’t have any money.

4. Why didn’t you have any money?
– Because I lost it all last night in a poker game.

5. Why did you lose your money in last night’s poker game?
– Because I’m not very good at “bluffing” when I don’t have a good hand.

(Source: https://www.isixsigma.com/tools-templates/cause-effect/determine-root-cause-5-whys/)

What is Software Testing Life Cycle (STLC)?

Just like developers follow the Software Development Life Cycle (SDLC) likewise testers also follow the Software Testing Life Cycle which is called as STLC. It is the sequence of activities carried out by the testing team from the beginning of the project till the end of the project.

Software Testing Life Cycle is a testing process which is executed in a sequence, in order to meet the quality goals. It is not a single activity but it consists of many different activities which are executed to achieve a good quality product. There are different phases in STLC which are given below:

  1. Requirement analysis
  2. Test Planning
  3. Test case development
  4. Environment Setup
  5. Test Execution
  6. Test Cycle Closure

Software testing life cycle (STLC)
Each of the step mentioned above has some Entry Criteria (it is a minimum set of conditions that should be met before starting the software testing) as well as Exit Criteria (it is a minimum set of conditions that should be completed in order to stop the software testing) on the basis of which it can be decided whether we can move to the next phase of Testing Life cycle or not.

Let us discuss about each phase in detail:

Requirement Analysis

This is the very first phase of Software testing Life cycle (STLC). In this phase testing team goes through the Requirement document with both Functional and non-functional details in order to identify the testable requirements.

In case of any confusion the QA team may setup a meeting with the clients and the stakeholders (Technical Leads, Business Analyst, System Architects and Client etc.) in order to clarify their doubts.

Once the QA team is clear with the requirements they will document the acceptance Criteria and get it approved by the Customers.

Activities to be done in Requirement analysis phase are given below:

  • Analyzing the System Requirement specifications from the testing point of view
  • Preparation of RTM that is Requirement Traceability Matrix
  • Identifying the testing techniques and testing types
  • Prioritizing the feature which need focused testing
  • Analyzing the Automation feasibility
  • Identifying the details about the testing environment where actual testing will be done

Deliverables (Outcome) of Requirement analysis phase are:

  • Requirement Traceability Matrix (RTM)
  • Automation feasibility report

Test Planning

Test Planning phase starts soon after the completion of the Requirement Analysis phase. In this phase the QA manager or QA Lead will prepare the Test Plan and Test strategy documents. As per these documents they will also come up with the testing effort estimations.

Activities to be done in Test Planning phase are given below:

  • Estimation of testing effort
  • Selection of Testing Approach
  • Preparation of Test Plan, Test strategy documents
  • Resource planning and assigning roles and responsibility to them
  • Selection of Testing tool

Deliverables (Outcome) of Test Planning phase are:

  • Test Plan document
  • Test Strategy document
  • Best suited Testing Approach
  • Number of Resources, skill required and their roles and responsibilities
  • Testing tool to be used

Test Case Development

In this phase the QA team write test cases. They also write scripts for automation if required. Verification of both the test cases and test scripts are done by peers. Creation of Test Data is done in this phase.

Activities to be done in Test Case Development phase are given below:

  • Creation of test cases
  • Creation of test scripts if required
  • Verification of test cases and automation scripts
  • Creation of Test Data in testing environment

Deliverables (Outcome) of Test Case Development phase are:

  • Test cases
  • Test scripts (for automation if required)
  • Test Data

Test Environment setup

This phase includes the setup or installation process of software and hardware which is required for testing the application. In this phase the integration of the third party application is also carried out if required in the project.

After setting up the required software and hardware the installation of build is tested. Once the installation of build is successful and complete then the Test Data is generated.

After the creation of Test data the Smoke testing is executed on the build in order to check whether the basic functionalities are working fine or not. This phase can be done in parallel with the Test Case Development phase.

Activities to be done in Test Environment Setup phase are given below:

  • As per the Requirement and Architecture document the list of required software and hardware is prepared
  • Setting up of test environment
  • Creation of test data
  • Installation of build and execution of Smoke testing on it

Deliverables (Outcome) of Test Environment Setup phase are:

  • Test Environment setup is ready
  • Test Data is created
  • Results of Smoke testing

Test Execution

Before starting the Test Execution phase the Test Environment setup should be ready. In Test Execution phase the test cases are executed in the testing environment.

While execution of the test cases the QA team may find bugs which will be reported against that test case. This bug is fixed by the developer and is retested by the QA.

Activities to be done in Test Execution phase are given below:

  • Execution of Test Cases
  • Reporting test results
  • Logging defects for the failed test cases
  • Verification and retesting of the defect
  • Closure of defects

Deliverables (Outcome) of Test Execution phase are:

  • Test execution Report
  • Updated test cases with results
  • Bug Report

Test Cycle Closure

In order to start the Test Cycle Closure activity the Test Execution phase should be completed. In Test Cycle phase the QA team will meet and discuss about the testing artifacts.

The whole intent of this discussion is to learn lessons from the bad practices. This will help in future projects.

Activities to be done in Test Cycle Closure phase are given below:

  • To evaluate the test completion on the basis of Test Coverage and Software Quality
  • Documentation of the learning from the project
  • Analyzing the test results to find out the distribution of severe defects
  • Test Closure Report preparation

Deliverables (Outcome) of Test Cycle Closure phase are:

  • Report of Test Closure

The below table briefly explains the Software Testing Life Cycle along with the Entry Criteria, Activity, Exit Criteria and Deliverable associated with each phase:

STLC phases Entry Criteria Activity Exit Criteria Deliverables (Outcome)
Requirement analysis Availability of Requirement document both Functional as well as non-functionalArchitectural document of the application or the product should be available

Acceptance criteria defined and duly signed by the customers

 

Analysis of System Requirement specifications to understand the different business modules and it’s functionalitiesTo identify the user profile, user interface and user authentication

Types of tests to be performed on the application or product should be identified

Should collect the details about testing priorities

Preparation of RTM that is Requirement Traceability Matrix

Test Environment details should be identified in order to do testing

Analysis of automation possibility if it is required

RTM should be signed offThe customer should sign off on the test automation feasibility Requirement Traceability Matrix (RTM)Report on Automation Feasibility if it is applicable
Test Planning Detailed requirement documentRequirement Traceability Matrix (RTM)

Automation Feasibility Report

 

Preparation of Test Plan documentPreparation of Test strategy document

To analyze the best suited testing approach for the application or product

To analyze the testing techniques and the types of testing to be carried out in order to maintain the quality

Selection of the testing tool

Estimation on the testing efforts

Resource planning as per the skill required for testing and also assigning roles and responsibility to them

Approved Test Plan documentApproved Test Strategy document

Document of Effort estimation

Test Plan documentTest Strategy document

Effort estimation document

Test case development Detailed Requirement documentTest Plan and Test strategy documents

Automation Feasibility Report

 

 

Creation of test cases for all the modules or features in the application or productCreation of automation scripts if required

Review of test cases and test automation scripts

Test data creation

Reviewed Test casesReviewed Test automation scripts

Test data creation ready for testing

 

Test casesTest automation scripts

Test data

Test Environment setup System design documents should be availableArchitectural document of the application should be available

Environment set-up plan document should be available

 

Understanding the design and architecture of the applicationSetting up the test environment

Installation of required hardware and software in order to start testing the application

Integration of any third party application (if required)

Installation of build

Creation of test data

Execution of smoke testing on the build

Accepting or rejecting the build as per the smoke test result

Environment setup is ready for testingAll the required software and hardware are installed

Build installation is complete and successful

Test data creation is complete

Smoke testing is done

Test environment along with test dataSmoke test result
Test Execution Documents like RTM, Test Plan, Test strategy, Test cases and Test scripts should be readyTest environment should be ready

Test data should be ready

Integration of third party application (if required) should be successful

Smoke testing of the application should be successful

 

Execution of test casesPreparation of test result document

Logging defects for the failed test cases

Mapping of defects with the test cases

To update the test cases and test strategy if required

Fixed defects should be retested

Closure of the defects if they are working as expected

Execution of regression testing of the application or product in order to ensure its stability post defect closure

 

All test cases are executedDefects are logged and tracked for closure Completed the test case executionUpdated the test cases wherever required

Defects reported

Test cycle closure All the test cases are executed and updatedTest results are documented

Defect logs are available

Evaluation of the test completion on the basis of Test Coverage and Software QualityPreparation of Test Closure report

Analyzing the test results to find out the distribution of severe defects

Signed off Test Closure report by the client Test closure Report

 

Source: http://istqbexamcertification.com/what-is-software-testing-life-cycle-stlc