A Potential of Neural Networks in NFT Marker Recognition in AR Apps (PoC)

VideoBomb's WebAR platform transforms every brand interaction into a dynamic AR experience. Brands have a unique opportunity to promote, inform, and educate their customers through this platform. It allows them to showcase their products, deliver key messages, and establish stronger connections with their target audience, driving business growth and leaving a lasting impression.

The primary objective of this project was to explore the potential of neural networks in simulating ARToolKit's functionalities, specifically for NFT marker recognition in augmented reality applications on mobile devices. The target was to achieve at least 20 FPS in video frame marker localization with precise detection of marker corner coordinates. However, it is important to note that the project served more as a proof of concept (PoC) than a final, polished solution.

In simpler terms, we tried to design a neural network that could easily learn new NFT markers, recognize them, and locate them in near real-time on any image/video, almost in real-time, allowing it to run on mobile devices. 

Eventually, our goal was to enable the neural network to return a mesh representing the exact position of the marker on any surface (such as bottles, shirts, or waving flags). This mesh could then be used to map a video or image onto the marker, replacing it in an augmented reality (AR) scenario.

Neural Network Selection and Adaptation:

• Chose AlexNet and MobileNet based on their potential for mobile application deployment.
• Applied transfer learning, customizing the last few layers and fine-tuning the entire network to suit specific project needs.
 

Training and Fine-Tuning:

• Approximately 5,000 images per marker were used to train the neural networks.
• Emphasis was placed on adapting both custom and pre-existing layers of the networks to the project's dataset.

Dataset Generation and Augmentation:

• Both Unity and a Python-based tool were employed to create and augment datasets, ensuring a rich variety of real-world conditions like blur, noise, and marker coverage.
• This approach was intended to enhance the robustness of the neural networks against diverse environmental factors.

Training Infrastructure:

• Utilized NVIDIA RTX 3080 and RTX 3070 graphics cards for efficient dataset processing.

Optimization for Mobile Deployment:

• ONNX was chosen for its effectiveness in deploying neural networks on mobile platforms.

• AlexNet and MobileNet, adapted through transfer learning, showed promise in NFT marker recognition.
• However, the project encountered challenges with the accuracy of marker localization, further research and development would be needed.
• The project functioned as a PoC, demonstrating the feasibility of using neural networks for NFT marker recognition, but did not reach a stage ready for production.