AR Calibration

The project solves the virtual-to-real calibration problem using $AX=YB$ formulation.

Note: The project once tried particle filters and looked into different noise configurations, making the repository redundant. You can ignore those parts.

Components

• Unity project: Should be deployed on HoloLens 2 or used with Holographic Remoting Feature.

• Python scripts: Used for tracking the optical marker and sending the data back to the Headset.

• Matlab codes: Used to simulate the algorithms.

  Only the main3.m script works, the other main scripts involve particle filters and noise configuration comparison and didn't show significant improvement.

  The DataAnalyze.m and DataAnalyze2.m are used to analyze the evaluation data.

How to use

• Open CalibrationOnDevice Scene and go to Main.

• Set the parameters you want in Calibration.cs on the inspector. Uncheck Is Simulation and only change two parameters.

• Num_samples: The number of measurements. 100 measurements would take 3-4 minutes to collect.

• K_factors: The filter range. the Slider UI script controls the division, and the default is 60.

• Open ndi.py in the Python folder and choose the right rom file path.

  romfile_path_Head = "Tracker_files\APPLE01.rom"
  romfile_path_Tooth = "Tracker_files\Cube_Reference.rom"
  SETTINGS = {"tracker type": "polaris", "romfiles": [romfile_path_Head, romfile_path_Tooth]}
  

  Note: This Python file somehow can only accept up to 2 marker arrays.

• Run both the Python script and the Unity project.

• Move HoloLens 2 (attached with optical markers) before the NDI tracking system at a slow speed. Three different audio feedbacks will show up after collecting every 1, 10, and 100 measurements, respectively.

• After finishing the collection, initial hand-eye calibration will be carried out. To fine-tune the calibration result, open your hand and make sure it's tracked by the headset and maintain it for 3 seconds, then the UI will appear on your hand.

- Put the Cube reference in front of the NDI tracking system and adjust the filter until it returns a good overlay.

Note: The calibration results are debugged out on the console. I would suggest you use Holographic Remoting and thus view the console information. However, I would add an output pannel UI in the future version.

Citation

Ai, L., Liu, Y., Armand, M., & Martin-Gomez, A. (2024, October). Calibration of Augmented Reality Headset with External Tracking System Using AX= YB. In 2024 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) (pp. 210-218). IEEE.