PerfLoc: Performance Evaluation of Smartphone Indoor Localization Apps
With billions in everyday use throughout the world, the smartphone is obviously a very important personal communication, data processing, and entertainment tool. It is also true that there exists a strong demand for a personal navigation capability inside buildings and in other indoor environments to complement the availability of GPS/GNSS for outdoor navigation, which is already very popular. It is anticipated that a wide variety of Location Based Services (LBS) will be enabled if/when effective indoor localization and tracking solutions are developed.
The main goal of the PerfLoc project is to facilitate the development of the best possible smartphone indoor localization apps. We achieve this goal by making the following available to the R&D community:
An extensive repository of annotated smartphone sensor and RF signal strength data to enable researchers to develop smartphone indoor localization apps.
A web-based performance evaluation tool based on the international standard ISO/IEC 18305 to evaluate the performance of the apps developed based on the PerfLoc data repository.
The data is "annotated" in the sense that all the measurements are time-stamped and, more importantly, the data includes timestamps at which the test subject that collected the data was at one of 900+ surveyed test points deployed in four large buildings. Therefore, if an app developer uploads the location estimate generated by the app for the time instance the test subject was at a particular test point, the location estimate can be compared with the ground truth location of the test point. This makes it possible to evaluate the performance of the app over the web without any need to physically test the app in the four buildings.
Four Android phones were used in our data collection campaign. We collected a total of ~16 hours of data with each phone over 38 Test & Evaluation (T&E) scenarios in four large buildings with a total area of 30,000 square meters based on guidance from ISO/IEC 18305. We collected data from all sensors available in each phone (accelerometer, gyroscope, magnetometer, barometric pressure, step counter, light, temperature, humidity, etc.) and the strength of RF signals received from Wi-Fi access points and cellular base stations. We also collected GPS data whenever available, but it is well known that GPS availability inside buildings is spotty at best. To get more information about the data we collected as well as some tests we ran to validate the data, please consult a paper we published at the 2016 IEEE PIMRC.
It is not easy for any organization to deploy 900+ test points in four large buildings, have the test points professionally surveyed, and spend months to collect a large amount of smartphone data. The PerfLoc data set, which includes both training and test data, facilitates development of smartphone indoor localization apps. It relieves individual organizations from having to collect their own data sets. The PerfLoc data set along with our over-the-web standardized performance evaluation methodology provides a level playing field for development and performance evaluation of smartphone indoor localization apps. It makes it possible to compare the performance of such apps as opposed to the situation prior to PerfLoc where there was no way of comparing indoor localization techniques / apps that had appeared in a large body of scholarly publications. It has also made it possible for NIST to organize a prize competition based on the PerfLoc concept. After the release of the PerfLoc data and the launch of its performance evaluation portal, NIST will allow the R&D community to develop their apps and have them evaluated through the PerfLoc web portal for six months. At the end of the six-month period, NIST will invite the developers of the top three apps to make a live demo of their apps and present their work at an appropriate scientific conference. NIST will award cash prizes to the top three apps at the conference.
This web page and the pages linked to it provide all the information you need to use the PerfLoc data and participate in PerfLoc Prize Competition. This includes an FAQ page, where NIST will publish any questions about PerfLoc that might be of interest to a wider audience and answers to those questions. You may find the answer to your question in the FAQ page.
Any question? Please email LocSprt@nist.gov.