NIST plans to hold a two-week LTS Test and Evaluation Event, henceforth called LTS T&E Event, where interested companies bring their LTSs for T&E with a two-fold purpose:
Assessment and validation of the ISO/IEC 18305 standard with the intention of improving the next version.
LTSs are very complex and have many features and operational constraints that must be evaluated. Therefore, ISO/IEC 18305 is a complex standard that must be assessed and validated before it can be widely used. It is through the process of validation that NIST will discover any problems associated with applying the standard and answer questions, such as the following:
Are the performance metrics appropriate? Do they address all relevant performance aspects?
Is each T&E scenario appropriate for the intended purpose? Is each T&E scenario well-described?
Are reporting instructions sufficient and complete?
Are there any inconsistencies within the standard?
During each of these events, commercially available systems will be tested according to the procedures of ISO/IEC 18305. Lessons learned from the LTS T&E Event will be used to make modifications to the testing procedures and corresponding revisions in ISO/IEC 18305.
Assessment of commercially available technologies using the standardized test methods of ISO/IEC 18305 for the dual purposes of comparing technologies and further development of the technology.
Comprehensive test results for each system will be made available to the respective manufacturer. In addition, summary test results for all systems tested, with the names of the companies anonymized, will be made available to the general public. A benefit of having data from the LTS T&E Event is that minimum performance requirements can be set for indoor localization and tracking systems for particular applications. It is not possible to set performance requirements without having standardized test procedures or an understanding of current system capabilities.
In preparation for the LTS T&E Event, NIST has instrumented five buildings on its Gaithersburg, MD campus, selected according to the guidelines specified in ISO/IEC 18305, with 1,300+ rugged, one-inch, circular floor markers whose locations have been surveyed by a professional surveying company using precision laser surveying equipment. These floor markers will constitute points at which each prospective LTS will be tested.
A limited number (8-10) of LTS products will be tested at the LTS T&E Event. Products deemed suitable for testing based on level of maturity and commercial availability will be accepted on a first come, first served basis during an open registration period. This will be followed by a pre-event workshop at NIST to inform the participating companies about the logistics of the event and answer their questions. The LTS T&E Event will be held in each of the buildings mentioned above according to several testing scenarios. Prior to administering the suite of tests in a given building, any equipment needed for the operation of an LTS under test needs to be installed in the building. The equipment must be removed when testing of that system concludes. Most of the test scenarios involve one or more test subjects, each carrying a localization device, walking in a given building following a pre-determined path and pressing a button on the localization device once the test subject is on top of a pre-installed test point. This will prompt the LTS under test to generate an estimate for the location of the test subject while he/she was at the test point.
Comparing the ground truth 3D coordinates of each test point with an LTS’s estimate of the 3D location allows the estimation error to be computed and statistically analyzed using the performance metrics specified in ISO/IEC 18305. In addition to accuracy, Clauses 8 and 9 in ISO/IEC 18305 include many non-accuracy performance metrics for LTSs that will be used to characterize the performance of any LTS under test.
There will be some scenarios involving multiple test subjects to measure the scalability of the LTS in terms of how many users it can locate simultaneously. Some test scenarios will involve crawling on the floor, walking backwards, running, and sidestepping during certain parts of the scenario, because first responders often have to engage in such activities, and the performance of any inertial measurement unit used in the LTS is affected by how the test subject moves or whether he/she is stationary. There will also be a scenario where one or more localization devices are placed on a push cart that will be moved around the building. This corresponds to movements of merchandise in warehouses on cart or forklift. It is a different from human mobility.
The LTS T&E Event will be publicized through a Federal Register Notice (FRN) several months prior to the event to give companies equal opportunity to take advantage of participating in this voluntary T&E event.
Sponsors:
