Open security testing received a boost with the announcement last week that the University of New Hampshire's InterOperability Lab (UNH-IOL) would become the first university-based testing facility to work with NetSecOPEN, a vendor-based organization that aims to create an open framework for testing next-generation firewall products.
With few exceptions, security testing has been a closed affair — a situation that has often been a sore point for both vendors and security-equipment consumers. Over the past five years, businesses have become more knowledgeable about the type of performance needed to do security at speed, making a stronger argument for open testing, says Timothy Winters, senior executive for software and IP networking at the UNH-IOL.
"Open testing is definitely up and coming. The argument is, 'Open it up so people on both sides can see what was tested, and then you can do apples-to-apples comparisons,'" he says. "Anyone can see whether they are covering the issues that they need covered and, if not, can request changes, and that can feed back into the open testing."
Along with similar efforts in anti-malware systems and penetration testing, NetSecOPEN marks a growing movement toward open security testing standards. While impetus for the organization came from vendors, both open testing labs and testing-equipment vendors have signed on as well. To date, the list of members include large network-security firms — such as Check Point Software, Cisco Systems, Fortinet, and WatchGuard — and testing firms such as UL and the European Advanced Networking Test Center (EANTC).
"Hearing from a lot of vendors, they want options because other programs are closed," the UNH-IOL's Winters says. "They have issues with the lack of openness of not know what was being tested and how it was being tested. Often, the results come out and people would say, 'I got a different number,' and vendors would not know why they got a specific score."
The testing methodology has been submitted to the Internet Engineering Task Force (IETF), the organization that sets the standards for Internet technologies. The focus on getting the specification accepted as a standard demonstrates the commitment to openness, says Brian Monkman, executive director of NetSecOPEN.
"The big thing is the transparency aspect," he says. "So much performance testing is done in various labs that don't share their testing methodology. The goal here is to ensure that there is a lot more information presented with these tests that will provide the enterprises with the ability to reproduce things and understand the philosophy behind the tests themselves."
The effort will initially focus on quantifying the performance of next-generation firewall technology in a small number of realistic environments. In addition, the group will curate different sets of network data, including attack data that can be used to test the efficacy of products.
Initially, however, the focus will not be on the capabilities of products to detect the latest attacks, but on their performance while checking traffic under realistic scenarios, says the UNH-IOL's Winters. Those scenarios will be different depending on customer needs.