Software product certification is an independent assessment by an entity that is accredited (ISO/IEC 17065) which evaluates commercial-off-the-shelf (COTS) software. The testing is conducted by a software testing lab accredited with ISO/IEC 17025.
For the ICT project (considering to acquire and/or customise a COTS software), the QS Product Certification in an important quality evaluation method that offers The Agency the ability to:
- Ascertain AS-IS quality of the COTS software being proposed
- The Agency immediately knows the certification status of the COTS software it is evaluating, where a standardized approach was used by a trustworthy independent party to V&V the overall quality of the COTS software.
- The Agency has access to the lab report accompanying the certification where it verified and validated all the claims made by the COTS software owner.
- The Agency has an objective independent party measuring the size of the COTS software i.e. its Function Point count, based on claims made by the COTS software owner.
- Objectively compare COTS software vs COTS software
- The Agency, as it utilises the same software certification scheme i.e. QS Product Certification, can also objectively benchmark various COTS being evaluated by comparing against one another.
- The Agency can objectively estimate the percentage of Function Point count that fits its requirements for each of the proposed COTS software.
- Analyse the extent of customisation to be applied (Buy vs Build)
- The Agency can objectively estimate the percentage of functional customisation required against the overall COTS AS-IS functionality.
- Subsequently, an objective risk assessment of customisation effort and potential quality impact to the TO-BE COTS can be ascertained.
- A decision can follow on whether to recommend to “Buy COTS Software” or “Custom Build” the TO-BE software for The Agency.
- Re-ascertain quality of COTS after customisation
- Subsequent to a “Buy COTS software” decision and customisation performed by the Development Team, The Agency can re-ascertain that the overall quality of the software remains intact i.e. should there be quality issues, and defects reported on a certified AS-IS COTS software, they can be isolated to the customisation performed.
Figure 4.14 below summarizes the benefits derived from undergoing the QS Product Certification.
Quality Software (QS) Product Certification scheme evaluations measure a minimum of six (6) external quality characteristics, that are:
- Functionality,
- Reliability,
- Usability,
- Efficiency,
- Maintainability, including ability of the original COTS vendor to fix reported defects,
- Portability
Each of these characteristics can have one (1) or more metrics measuring that item following a defined measurement method and scale. The external metrics involves measurements of a software product derived from how well it works through testing, operating, and observing based on the applicable software product.
Once evaluated, The Agency will know how well the product performs across these characteristics. The result is then used as input to determine the amount of testing needed during the ICT project lifecycle.