Testing metrics can be classified as belonging to one or more of the
following categories:
·
Project metrics, which measure
progress toward established project exit criteria, such as the percentage of
test cases executed, passed, and failed·
Product metrics, which measure
some attribute of the product, such as the extent to which it has been tested
or the defect density·
Process metrics, which measure
the capability of the testing or development process, such as the percentage of
defects detected by testing·
People metrics, which measure
the capability of individuals or groups, such as the implementation of test
cases within a given schedule
Metrics related to product risks
Metrics related to product risks include:
·
Percentage of risks completely
covered by passing tests·
Percentage of risks for which
some or all tests fail·
Percentage of risk not yet
completely tested·
Percentage of risks covered,
sorted by risk category·
Percentage of risks identified
after the initial quality risk analysis
Metrics related to defects
Metrics related to defects include:
·
Cumulative number reported
(found) versus cumulative number resolved (fixed)·
Mean time between failure or
failure arrival rate·
Breakdown of the number or
percentage of defects categorized by the following:
o Particular test items or components
o Root causes
o Source of defect (e.g., requirement specification, new feature,
regression, etc.)
o Test releases
o Phase introduced, detected, and removed
o Priority/severity
o Reports rejected or duplicated
·
Trends in the lag time from
defect reporting to resolution·
Number of defect fixes that
introduced new defects (sometimes called daughter bugs)
Metrics related to tests
Metrics related to tests include:
·
Total number of tests planned,
specified (implemented), run, passed, failed, blocked, and Skipped·
Regression and confirmation
test status, including trends and totals for regression test and confirmation
test failures·
Hours of testing planned per
day versus actual hours achieved·
Availability of the test
environment (percentage of planned test hours when the test environment is
usable by the test team)
Metrics related to test coverage
Metrics related to test coverage include:
·
Requirements and design
elements coverage·
Risk coverage·
Environment/configuration
coverage
Metrics to monitor test planning and
control
Metrics to monitor test planning and control activities may include:
·
Risk, requirements, and other
test basis element coverage·
Defect discovery·
Planned versus actual hours to
develop testware and execute test cases
Metrics to monitor test analysis
Metrics to monitor test analysis activities may include:
·
Number of test conditions
identified·
Number of defects found during
test analysis (e.g., by identifying risks or other test conditions using the
test basis)
Metrics to monitor test design
Metrics to monitor test design activities may include:
·
Percentage of test conditions
covered by test cases·
Number of defects found during
test design (e.g., by developing tests against the test basis)
Metrics to monitor test
implementation
Metrics to monitor test implementation activities may include:
·
Percentage of test
environments configured·
Percentage of test data
records loaded·
Percentage of test cases
automated
Metrics to monitor test execution
Metrics to monitor test execution activities may include:
·
Percentage of planned test
cases executed, passed, and failed·
Percentage of test conditions
covered by executed (and/or passed) test cases·
Planned versus actual defects
reported/resolved·
Planned versus actual coverage
achieved
Metrics to monitor test completion
Metrics to monitor test progress and completion activities will include
a mapping to the milestones, entry criteria, and exit criteria (defined and
approved in test planning), which may include the following:
·
Number of test conditions,
test cases or test specifications planned and those executed·
broken down by whether they
passed or failed·
Total defects, often broken
down by severity, priority, current state, affected subsystem, or·
other classification·
Number of changes required,
accepted, built, and tested·
Planned versus actual cost·
Planned versus actual duration·
Planned versus actual dates
for testing milestones·
Planned versus actual dates
for test-related project milestones (e.g., code freeze)·
Product (quality) risk status,
often broken down by those mitigated versus unmitigated, major·
risk areas, new risks
discovered after test analysis, etc.·
Percentage loss of test
effort, cost, or time due to blocking events or planned changes·
Confirmation and regression
test status
Metrics to monitor test closure
Metrics to monitor test closure activities may include:
·
Percentage of test cases
executed, passed, failed, blocked, and skipped during test execution·
Percentage of test cases
checked into re-usable test case repository·
Percentage of test cases
automated, or planned versus actual test cases automated·
Percentage of test cases
integrated into the regression tests·
Percentage of defect reports
resolved/not resolved·
Percentage of test work
products identified and archived
In addition, standard project management techniques such as work
breakdown structures are often used to monitor the test process. In Agile
teams, testing is part of a user story’s progress on the burndown chart. Where
Lean management techniques are used, testing progress on a story-by-story basis
is often monitored by having the user story card move through a column on the
Kanban board.
Given a defined set of metrics, measurements may be reported verbally in
narrative form, numerically in tables, or pictorially in graphs. The
measurements may be used for a number of purposes, including:
·
Analysis, to discover what
trends and causes may be discernible via the test results·
Reporting, to communicate test
findings to interested project participants and stakeholders·
Control, to change the course
of the testing or the project as a whole and to monitor the results of that
course correction
