MR Spatial Scenes PainPoints

There are difficulties and obstacles of MR spatial scenes that prevent it from being commercialized on a large scale due to involving complex technologies such as computer vision, artificial intelligence, and distributed system engineering.

1) Efficiency of Producing MR Spatial Scene

The traditional procedure of producing MR spatial scenes requires professional surveying-grade devices with expensive Lidar and requires professional operators. This procedure costs too much for large-scale commercial applications.

2) Reconstruction Performance of MR Spatial Scene

The application scenarios of MR spatial scenes include scenic spots, museums, business streets, malls, city parks, and so on(except for restricted countries or regions, including the US, China, private properties, etc), which involve people's everyday lives. Considering the scale, visual condition, lighting condition, and dynamic visual interference in different scenarios, the model needs intensive testing and optimization involving large amounts of specific areas's data. For example, training a deep neural network for extracting local and global features from images to adapt to different lighting conditions and different space scales; Image matching is the basis of 3D reconstruction, and it is necessary to adjust the image matching strategy according to the multiple influence factors to filter out false matches.

Besides, to support large-scale construction, GPU-based optimization should be involved in the majority of steps in the whole pipeline, such as feature extracting, image matching, image recalling, bundle adjustment etc.

3) Optimization for Visual Positioning

The performance of MR spatial scenes also needs to be synchronized and optimized for relocation (the process of collecting image data through the camera of electronic devices such as mobile phones/Pads/VR/MR glasses, which is used to calculate the current 6DOF pose). For example, GPU-optimized efficient image retrieval, which can quickly retrieve correct candidate keyframes from thousands of frames on the mobile terminal; automatic scale correction means that the scale of the tracking system on the mobile terminal is determined by the inertial measurement unit (IMU). However, the VIO algorithm on the mobile terminal has undergone a lot of optimization and careful calibration, there will still be an error of 1-5%. If the scale is inconsistent, it will lead to systematic errors in the relocation calculation.

4) MR Spatial Scene Needs Constant Updating

During the practical application of MR spatial scenes, the visual features of the space change frequently, for example, new stores, new billboards, renovation or reconstruction of buildings, and so on. After the visual features exceed a certain percentage, the MR spatial scene needs to be updated, especially when it happens out-of-town or abroad, the MR spatial scene update work will become difficult and pay higher labor and transportation costs.

After many years' research and development, the DeMR Network core development team has achieved many technical breakthroughs regarding spatial intelligence and highly concurrent distributed systems. The problems above are well solved.