Here at the University of New Hampshire’s Interoperability Lab (UNH-IOL), the collaborative efforts of Plugfests, also known as group test events, have been hosted and developed since the late 1980s. The lab’s dedicated space for these types of events holds up to 60 companies in a neutral, third party environment. Common among the UNH-IOL’s group testing efforts, are the multi-vendor Non-Volatile Memory Express (NVMe®) and NVMe over Fabrics (NVMe-oF™) Plugfests which are held two times per year.
The NVMe and NVMe-oF Plugfests test device’s conformance and interoperability against NVMe standards in order to ensure their qualification for the NVMe Integrator’s List. Each NVMe Plugfest contains a new set of test cases, which are based on the latest developmental aspect originating from the technical committees representing the NVMe. In most cases there will be existing tests in the initial For Your Information/In Progress (FYI/IP) Status that take 1-3 Plugfests to be modified to the Mandatory Status test, then in which all NVMe and NVMe-oF devices must pass to attain a spot on the NVMe Integrator’s List.
In early 2020, due to the Covid-19 pandemic, the UNH-IOL made the decision to transition these events to virtual and allow member companies the first opportunity to have their products tested and qualified for the NVMe Integrator’s List. Since this time, the lab has hosted two virtual Plugfests to run testing for both NVMe and NVMe-oF devices. During these events the IOL teams are running tests for multiple companies and their devices.
When preparing for testing, the IOL releases the conformance software, IOL INTERACT™, to participating members 60 days ahead of the plugfest to ensure customers have enough time to work out any bugs ahead of the event. Member companies then ship their products and setup instructions to the IOL Datacenter team, from there the team performs the setup, runs the test(s), and reports the results. In the event of an issue that requires debugging of the product, the member is contacted to send new firmware, discuss over Zoom, or in some instances, the customer can be granted access through a remote desktop session in order to access their product through the lab’s network. However, the majority of debugging has been as simplified as a configuration change that is most easily communicated through email.
The process for overall testing and data collection is very similar for both in person and virtual. In both scenarios the data logs are shared with customers during and after the testing process via email.
Although there were some hiccups during the configuration processes, the lab did not see any real changes in the number of passing devices at the two 2020 events. There were 5 PCle companies which tested a total of 8 products, and 3 companies that tested 8 NVMe-oF products with 14 different combinations. These combined for a total of 8 companies, across 6 countries, testing 16 different products throughout the two events..
A big part of the Plugfests is the ability for companies to come together in a shared space and brainstorm ideas and solutions while testing their products, and although the virtual events allow for device certification, the removal of in person Plugfests would take away the collaboration between fellow engineers.