Algorithm Testing is a necessary part of both FIPS 140 validations as well as some Common Criteria (CC) certifications. This testing isn’t without risk, and failing to properly prepare for algorithm testing can cause missed milestones, strain development resources, and introduce unnecessary uncertainty in the process.

Here are some steps vendors can take to help manage the risks of algorithm testing:

1) Start Early

It’s never too early to start planning. Algorithm testing has its own challenges, and discovering those late in the process can cause missed milestone and unnecessary risk. Find a consultant who can help you with the process, or work with a lab that can provide the type of assistance you’ll need to prepare for testing. Not all labs are equally skilled at providing the engineering support necessary to properly prepare for algorithm testing.

When you’re planning milestones, you should figure that algorithm testing is going to take 2-4 weeks. That time might be somewhat front-loaded with preparation work, but at least 2 weeks of it will come late in the process. You’re going to be submitting the final FIPS build of your product to the lab for testing, and that build might be hot off your internal QA lab.

There will naturally be a tension between the delivery of the final build and the urgency to get the FIPS testing process started as soon as possible. Most labs will only do functional testing after algorithm testing. If any problems are discovered in algorithm testing, it typically means some changes to the module are needed to remediate the problems. Those changes would invalidate and functional testing, causing delays and repeated work.

You’ll need to make sure that you account for the estimated 2 weeks for algorithm testing between the delivery of the final build and the start of functional testing.

2) Bring engineering resources

Vendors should assign a point-person to oversee algorithm testing. That person should be familiar with the cryptographic provider within the product, as well as the uses of cryptography in the larger product.

This point-person will be the direct interface between the consultant or lab that’s assisting with testing. The consultant or lab will need to determine the scope of testing, the complete list of algorithms to test, and even the modes and limitations of the module.

3) Know your module

You’ll need to know all of the algorithms implemented by your module, as well as the modes and limitations of those algorithms. Does your RSA Signature Verification handle both 2K and 3K keys? Is your SHA implementation bit- or byte-oriented?

For most Open Source libraries, such as OpenSSL and Bouncy Castle, these capabilities are well known. If you’re using a custom solution, or a closed-source library, then you will need to be able to work with your consultant or lab to help determine those capabilities.

4) Pesky Data

Algorithm testing occasionally requires the demonstration of capabilities that are outside of the normal usage of the cryptography. For example, to test DRBG, the entropy is specified in the test and must be the exclusive entropy provided to the module. Most “live” DRBGs will automatically pull entropy from the system for their initial state, and might additionally pull entropy when certain reseed thresholds are met. This default behavior must be prevented under test circumstances.

The full details of these instances of “pesky data” required for testing are beyond the scope of this blog. Your consultant or lab can help you identify these problem areas, and give you advice on how to address this requirement. A typical solution is to produce a “testing” build of the module, or to provide a system of runtime settings that can set the module into a testing mode. There is no right way to do it — it depends on your specific circumstances.

5) Document for next time

Have you point-person everything they can about the process and the methods that were undertaken to complete testing. This documentation will be invaluable should you decide to re-validate your module in the future. Your staff may change over time, and that knowledge could be lost. Save yourself the time and effort of re-discovering all the valuable information and lessons you learned through the process.