Overview
In the last decades, interconnected devices of several types started to be used in the most diverse contexts, creating intelligent environments, and improving the population’s life. New experiences that merge real-world reality with objects and virtual realities have become available through many applications and devices, and with application in several areas, such as entertainment, industry and manufacturing, education and training, medicine and health care, and militarism, among others. On the other hand, the growing use of devices that monitor and collect data about their users and the surrounding environment, the interconnection and sharing of data by such devices, and the rapid dissemination of information through the Internet increased the concerns on maintaining privacy and confidentiality of individuals and companies. Then, several data protection and privacy legislations were created, such as the European General Data Protection Regulation. But identifying risks to privacy and data confidentiality in images captured by devices that freely use cameras to scan the environment around them – to identify objects and people – is a challenging and open issue.
In this project, we intend to provide automatic tools for guaranteeing the privacy and confidentiality of individuals and companies while maintaining seamless, real-time, and persistent physical-digital Augmented Reality (AR) experiences. We will identify the main classes of privacy and confidentiality risks in the AR context and apply machine learning to automatically identify occurrences of such issues in raw data captured by AR applications. For each class, we will research the adequate countermeasure. The identification methods and countermeasures must consider the specific requirements (e.g., response time) and resource limitations of AR applications. To validate our proposals, we will build two use AR applications. The first scenario is on a location-based AR game (LBARG), while the second evaluation scenario is on using AR to aid training and manufacturing.
Source code available:
- https://github.com/CIIC-C-T-Polytechnic-of-Leiria/SafeAR – A repository containing the source code of SafeAR implemented as a service to run on a server. Users may use YOLOv5, YOLOv8, and YOLOv9. Instructions for installations in conda and docker.
- https://github.com/CIIC-C-T-Polytechnic-of-Leiria/SafeARUnity – A repository containing the source code of SafeAR implemented as library in Unity. It uses Sentis and YOLOv8.
- https://github.com/CIIC-C-T-Polytechnic-of-Leiria/LootAR – LootAR is a Unity-based AR game that combines geolocation with augmented reality. Explore a real-world map, find virtual items, and collect them in AR mode. Features include item collection and obfuscation functionalities.
- https://github.com/CIIC-C-T-Polytechnic-of-Leiria/InPainTor – Real-time Semantic-aware Inpainting Model.
The support presentation in the seminar “Machine Learning and Privacy-Maintenance in Augmented Reality” by Tiago Ribeiro on May 15, 2024.
Contacts
For additional information, please send us an email to:
rogerio.l.costa AT ipleiria DOT pt
Acknowledgements
This work is funded by FCT – Fundação para a Ciência e a Tecnologia, I.P.,.
Reference: 2022.09235.PTDC
Funding: 49 888,90 € (Polytechnic of Leiria)
