oftware has assimilated itself into almost every aspect of our lives. It resides within our homes, vehicles, phones, workspaces, and so on. We find it in our televisions, speakers, light switches, and on and on and on. It is everywhere. Resistance to software’s assimilation is futile. It makes our lives easier.
Perhaps not coincidentally, the negative consequences of software-related incidents have drastically increased in the past few years. One of the most recently publicized incidents of software “gone bad” is software’s contribution to the Boeing 737 Max malfunctions, which led to two fatal crashes in 2018 and 2019. The Boeing 737 Max malfunction is a case where the software’s reported performance appears to have contributed directly to aircraft falling out of the sky. These and other incidents have not just gained the public’s attention. They have also served as a catalyst for changes within the industry, such as the use of quality management systems (QMS) to demonstrate that software does what it is designed to do.
But concerns about software performance pre-date the recent incidents. The National Institute of Standards and Technology (NIST) published a document nearly twenty years ago titled “The Economic Impacts of Inadequate Infrastructure for Software Testing” (available at https://www.nist.gov/system/files/documents/director/planning/report02-3.pdf). According to Table ES-4 in the document, the economic impact in the U.S. economy at that time of an inadequate software testing infrastructure carried with it a cost of nearly $60 billion annually.
In fact, I found a number of companies that track the cost of software incidents and their economic impact. One such company is Tricentis, a software testing company (https://www.tricentis.com). In their most recently available “Software Fail Watch” report from 2018, the company estimates lost revenue related to software failures in 2017 at about $1.7 trillion, certainly not a trivial amount!
But all of this leads to a single question, that is, how do you know whether your test software is actually behaving the way you think it should. So the objective of this article is to aid the reader in better understanding how to develop evidence that test software is performing within its intended design.
Software-based V/V processes can be as simple as a logbook recording a test case and its results, providing a source that you can reference during audits. The rigor of your V/V process could also be at the other end of the spectrum, in which you attempt to create every possible scenario and document the results. But, no matter how much you test your software, reaching 100% confidence is impractical and may well be impossible to achieve.
My wife, Bobbi, is convinced there are people out there that will always find a way to break something. (Of course, she’s not referring to me!) It may be some folks just have a knack for finding errors, but where is the dividing line between too little and too much? That question is beyond the scope of this article but is merely intended to show you where you can find the tools (sources) to create your own software V/V process. You get to decide where to draw the line.
If you’re thinking Deming Circle or Cycle Wheel, which championed the “Plan-Do-Check-Act” approach, you’re on the right path. However, I would add one more element to the “Plan-Do-Check-Act” approach. It would be “observe,” as in “Observe-Plan-Do-Check-Act” or OPDCA. You can find more information about the OPDCA model at Foresight University (http://www.foresightguide.com/shewhart-and-deming).
If you’re comfortable checking out this and other resources and proceeding on your own, you can stop reading this article. But let me provide my own brief guide to the V/V process.
To illustrate, let’s use as an example MIL-STD-461G, radiated emissions RE102 greater than 30 MHz. First, you replace the receive antenna in the system setup with a calibrated signal source. The inputs are test conditions, test limits, transducer correction factors, receiver measured data over frequency, and a known good signal from a calibrated source. RE102 requires the system check target amplitude to be the test limit minus six decibels (test limit – 6 dB). The actual calibrate signal source settings are a little different. The system check signal generator output target amplitude base equation is:
-
= (Test Limit-6 dB)-Antenna Correction Factor
-
= Receiver Recorded Value + Antenna Correction Factor
+ Signal Path Insertion Loss – External Preamplifier gain (if required)
I highly recommend using a “check early and check often” philosophy for MOTS and custom software. The difference between the black box and white box methods is accessibility. With the black box method, you do not have control (access) of the inner workings of the software but simply monitor the results of the operation and report your findings. With the white box methodology, you have access to virtually all aspects of the software and can test the test software’s inner operation and verify its performance.
MOTS V/V requirements pertain to functions or routines you’ve created or modified. There will be a point at which you won’t be able to modify the software since the software manufacturer is responsible for ensuring proper software operation and likely limits access to the software’s basic functions. Typically, this would include instrument drivers, basic EMC/EMI functions, and maintenance actions.
You could use the black box V/V method for any of MOTS functions you cannot change. Changes you make to the software should be verified and validated prior to release, always remembering that validation and verification are simply creating evidence that the software is adhering to your process and the applicable standard. The basic differences between the black box and white box methods include the level at which you are testing and the functions/routines you modified or created. You control the lower-level software functions and verify their performance. The software V/V is a process.
Let me offer an example in which you create a limited selection routine within a MOTS software. You would open the routine, operate the function, and verify the results. It takes no more effort. It sounds simple until you have a few hundred or more modifications to observe and validate. The complexity is within the sheer volume of the items you may need to verify.
You could take it one step further by creating different test cases where the user intentionally enters incorrect information to see how the software responds. Good software should provide error handling routines. And don’t forget that you have some of the same tools available to you that you do when applying the black box method. The standard required system checks are useful tools to prove that the software is doing what it is supposed to do.
I must reiterate that testing within the design development is not part of the V/V process. It is part of maturing the product, which is part of the design process. The development test results should also be documented for future prosperity. It could be stored and used within the “lessons learned” database, which may help you meet other QMS standard requirements.
I recommend performing a risk analysis and creating a test case table for custom software based from the results of the risk analysis then V/V testing each test case. Seed the test cases with intentional errors as well as known good variables. Remember that the goal is to ensure the product is performing within expectations and that some people are geniuses when it comes to breaking things. Conduct the test cases and record the results while keeping in mind any noncompliance results require a failure analysis and corrective actions.