LingBot-Depth 2.0 – A Depth Completion Model Introduced by Lingbo Tech, Ant Group

1. What is LingBot-Depth 2.0

LingBot-Depth 2.0 is a depth completion model introduced by Lingbo Tech, Ant Group, aimed at real-world scenarios. It also simultaneously open-sourced the visual foundation model LingBot-Vision. This model employs a two-tier collaborative architecture of "foundation model + task head," achieving high-precision 3D reconstruction of transparent or reflective objects such as glass and mirrors through explicit boundary structure pre-training. The training data has expanded from 3 million in the previous generation to 150 million, securing first place in 12 out of 16 depth completion benchmark evaluations. The RMSE error for large-scale depth missing scenarios indoors has been reduced from 0.132 to 0.062. The model has passed the professional certification by Orbbec Depth Vision Lab, establishing a complete capability chain for robots to transition from "understanding" to "precisely perceiving."

Image source: official article

Technical Positioning and Domain: Belongs to the field of depth estimation and completion in computer vision, specifically focused on generating high-precision dense depth maps for scenarios such as robotics and autonomous driving. Unlike traditional monocular depth estimation, LingBot-Depth 2.0 takes sparse depth (e.g., from low-resolution ToF or structured light sensors) as input and combines it with RGB images for completion. It excels at handling scenarios where traditional depth sensors fail, such as transparent objects and reflective surfaces, filling the technical gap in depth perception for complex materials in the real world.

Development Background: Developed by the Lingbo Tech team under Ant Group, which has long been dedicated to embodied intelligence and 3D vision technology. The motivation for development stemmed from the problem of large-scale depth data missing in scenarios such as service robots and industrial inspection, particularly with materials like glass and mirrors. The team started from the visual foundation model and pioneered the use of "boundary structure" as an explicit pre-training objective, enabling the model to achieve sub-pixel-level edge localization capabilities and providing a solid geometric prior for depth completion.

Core Value: Solves the pain points of traditional depth completion models in transparent/reflective objects, such as "depth overflow" and blurry edges, reducing the RMSE error in large-scale depth missing indoor scenarios by half, reaching 0.062. By open-sourcing the visual foundation model LingBot-Vision, it lowers the R&D threshold for downstream tasks (e.g., object detection, segmentation). At the same time, through collaboration with Orbbec, it has launched commercial SDKs and integrated cameras, achieving a complete closed-loop from algorithm to hardware, accelerating the deployment of embodied intelligence in home, warehouse, and industrial scenarios.

Technical Features: Uses a dual-model collaborative architecture, with LingBot-Vision at the lower level focusing on boundary structure extraction, and LingBot-Depth at the upper level performing depth completion based on boundary features. The decoupled design allows each model to specialize in its most proficient area. LingBot-Vision requires only 160 million images for pre-training and achieves clearer boundary recognition than general visual foundation models that rely on billions of data points. The depth completion network introduces explicit boundary constraints and temporal feature alignment, effectively preventing depth overflow and ensuring consistency between video stream frames.

2. Key Features

• High-precision depth completion: Based on RGB images and sparse depth input, the model employs an encoder-decoder structure to perform multi-scale feature fusion, generating high-precision dense depth maps. In large indoor scenes with significant depth missing, the RMSE error is only 0.062, halved compared to the previous generation, significantly outperforming existing methods.

• Transparent object perception: For transparent or reflective objects such as glass, mirrors, and faucets, the model uses boundary structure features as depth propagation constraints to achieve complete 3D structure reconstruction. Traditional depth cameras often completely fail or produce incorrect depth in these scenarios, whereas LingBot-Depth 2.0 can accurately reconstruct object contours and spatial positions.

• Edge fine recognition: Leveraging the pre-trained boundary structure capabilities of LingBot-Vision, the model has sub-pixel level boundary localization ability. It can precisely identify object contours, surface intersections, and small structures (such as wires and gaps), achieving sharp truncation at edges and avoiding depth blurring.

• Temporal stability: In video stream input scenarios, the model ensures temporal consistency and spatial accuracy of depth estimation across frames through a temporal feature alignment mechanism. It eliminates common flickering and jittering issues in single-frame inference, resulting in smooth and continuous depth map sequences suitable for real-time robot navigation and operation.

• Cross-scenario generalization: The training data covers real-world scenarios with indoor and outdoor environments, complex lighting conditions, and various materials (glass, metal, fabric, plants, etc.), totaling 150 million samples. The model maintains stable performance in unseen scenarios, with generalization capabilities surpassing those of models trained solely on synthetic data.

• Hardware co-optimization certification: The model has passed professional certification by Orbbec Depth Vision Lab, significantly improving edge clarity and robustness. It can seamlessly integrate with the Orbbec Gemini 330 series cameras, directly calling the commercial version of LingBot-Depth through the SDK, enabling a unified "sensor + algorithm" deployment.

• Open-source visual foundation model: The LingBot-Vision model (ViT-G/L/B/S versions) is simultaneously open-sourced, supporting researchers in training their own downstream visual tasks such as depth completion, segmentation, and detection based on this foundation model, thereby reducing the cost of developing new models.

3. How to Use

1. Obtain model weights: Download the model weights from HuggingFace (link to be updated after official release).

2. Clone the code repository: Execute git clone (link to be updated after official release) to get the full code, technical report, and inference scripts. The repository includes model definitions, preprocessing tools, and example data.

3. Set up the runtime environment: Install dependencies based on the requirements.txt file in the repository. It is recommended to use Python 3.10+, PyTorch 2.0+, and CUDA 11.8+. In terms of hardware, it is advised to use NVIDIA A100/RTX 4090 or higher GPUs. The LingBot-Vision ViT-G model requires approximately 24GB of GPU memory. If using an Orbbec Astra camera, you must also install the corresponding SDK.

4. Prepare input data: Prepare RGB images and corresponding sparse depth maps (which can be obtained from ToF sensors, structured light cameras, or SfM algorithms). The sparse depth can be a point cloud projection or a low-resolution depth map, and the model will automatically handle the sparsity. If no real sparse depth is available, you can generate pseudo-sparse depth using a monocular depth estimation model (e.g., Depth Anything) as input.

5. Invoke inference: Run python infer.py --rgb_path ./rgb.png --depth_path ./sparse.png --output ./dense.png to generate a dense depth map. You can select different model sizes such as ViT-L, ViT-B, or ViT-S using the --model_size parameter. Inference speed: ViT-S takes approximately 20ms per frame on an A100, and ViT-L takes approximately 80ms per frame.

6. Edge-side integration (commercial version): If you need to run the model in real-time on a robot, you can purchase the Orbbec Gemini 330 series camera and integrate the LingBot-Depth commercial version through its SDK. The commercial version provides an optimized inference engine, supporting real-time frame rates (≥30fps) on edge devices such as Jetson Orin.

Notes: The open-source version only includes the base model of LingBot-Vision. The complete depth completion network (LingBot-Depth) must be obtained through commercial collaboration. It is recommended to first validate the results on a desktop before deploying them on edge devices. The temporal stability feature is currently fully supported only in the commercial version.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations: If the application scenario involves a large number of transparent or reflective objects such as glass and mirrors (e.g., home service robots, industrial quality inspection), LingBot-Depth 2.0 is the optimal choice, as its ability to perceive transparent objects far exceeds that of other competitors. However, note that the depth completion model requires commercial collaboration to obtain and must be used with an Orbbec camera. If only monocular depth estimation is needed and fast deployment is a priority, the Small version of Depth Anything V2 (Apache 2.0 license) offers fast inference and strong generalization capabilities, making it suitable for general scenarios. For dense prediction tasks in academic research (e.g., joint training of semantic segmentation and depth estimation), DPT and MiDaS remain valuable mature baselines, with flexible open source licenses. Overall, LingBot-Depth 2.0 has an irreplaceable advantage in the robotics field, but its closed ecosystem is a drawback.

6. Editor's Take

LingBot-Depth 2.0 makes insightful innovations in its technical approach. Rather than blindly following the common paradigm of "larger models + more data," the team started from the real pain points of robot perception and pioneered the use of "boundary structure" as an explicit pre-training objective. This design directly addresses the core contradiction of depth completion—depth overflow and blurring at object edges. Pre-trained on only 160 million images, LingBot-Vision outperforms general-purpose foundation models (e.g., DINOv2) that rely on billion-scale data in boundary recognition clarity, proving that domain-specific prior knowledge is more effective than simply scaling up data volume. The decoupled design of the dual-model collaborative architecture allows visual representation and depth estimation to perform their respective roles, avoiding the feature coupling issues of traditional end-to-end models in complex scenes.

From a practical value perspective, the model's breakthrough in transparent object perception is landmark. Materials such as glass and mirrors are abundant in scenarios like home service robots, warehouse logistics, and autonomous driving, where traditional depth sensors are almost completely ineffective. LingBot-Depth 2.0 reduces the RMSE for indoor large-area missing scenes to 0.062, halving the error, which directly improves robot safety and operation success rates in complex environments. The deep collaboration with Orbbec enables the algorithm to be quickly deployed as a product-level solution, shortening the distance from paper to industry.

In terms of target audience, this model is most suitable for R&D teams in the field of embodied intelligence, especially engineers working on service robots, industrial quality inspection, and autonomous driving perception. Academic researchers can leverage the open-source LingBot-Vision for downstream tasks related to boundary structure. Commercial users should note that the full depth completion capability requires a commercial partnership, and hardware binding limits flexibility.

Regarding future development potential, as embodied intelligence continues to demand higher perception accuracy, LingBot-Depth 2.0's technical approach is expected to become an industry standard. If the next step can open-source the depth completion network weights and support more hardware platforms, it will greatly promote ecosystem prosperity. Additionally, extending boundary structure pre-training to more vision tasks (e.g., 3D reconstruction, SLAM) is also worth anticipating.

7. Application Scenarios

• Home Service Robots: In environments such as kitchens and bathrooms, where glass, mirrors, and faucets are present, robots require precise navigation and manipulation. LingBot-Depth 2.0 can complete depth information for transparent objects, enabling robots to avoid collisions and accurately grasp glass cups or faucets, thereby enhancing safety and reliability in home scenarios.

• Industrial Quality Inspection: For three-dimensional size measurement and defect detection of transparent packaging (e.g., plastic bottles, glass bottles) and reflective metal parts. Traditional visual solutions struggle with stable imaging, whereas this model, based on boundary structural features, can accurately reconstruct surface profiles, achieving sub-millimeter-level defect identification. It is suitable for applications such as pharmaceutical bottle filling lines and mobile phone casing inspection.

• Warehousing and Logistics: In warehouses with complex lighting conditions, object edges are often obscured by shadows or reflections. LingBot-Depth 2.0's sub-pixel-level edge localization capability can assist robotic arms in precisely identifying object contours and small items (e.g., labels, cables), enabling stable grasping and sorting, and reducing the rate of incorrect grasps.

• Autonomous Driving: Completing depth information in road scenarios to enhance perception of transparent obstacles (e.g., glass facades, fully transparent bus shelters). In complex urban intersections, the model can provide reliable depth maps for planning modules, preventing misjudgments caused by missing depth data and improving driving safety.

• 3D Data Acquisition Devices: Integrated into RGB-D acquisition devices such as Orbbec EGO, providing precise real-world data for embodied intelligence training. Researchers can directly use this device to collect datasets with high-quality depth labels, which can be used to train their own grasping, navigation, and other downstream models, accelerating algorithm iteration.

8. FAQ

Q: What is the difference between LingBot-Depth 2.0 and Depth Anything V2?
A: They use different technical approaches. Depth Anything V2 is a monocular depth estimation model that can predict depth using only a single RGB image, but it cannot handle transparent objects and has blurry edges. LingBot-Depth 2.0 is a depth completion model that requires RGB + sparse depth as input, focusing on filling in missing areas. It significantly outperforms Depth Anything V2, especially in handling transparent objects and edge sharpness. In terms of application scenarios, Depth Anything V2 is suitable for general depth estimation, while LingBot-Depth 2.0 is more appropriate for scenarios with high precision requirements, such as robotics, and for handling special materials.

Q: Which versions of LingBot-Vision are open-sourced? Can it be used directly for depth completion?
A: We have open-sourced the weights and inference code for the ViT-G/L/B/S versions. LingBot-Vision itself is a visual foundation model that outputs boundary structure features (such as edge maps and object contours), and cannot directly output depth. To complete depth estimation, a Depth Head (i.e., the LingBot-Depth network) must be trained on top of it. This network is currently only available through commercial partnerships. Researchers can use LingBot-Vision as a backbone network to train their own depth completion, segmentation, or detection models.

Q: What hardware configuration is required to use LingBot-Depth 2.0?
A: For the open-source version, we recommend NVIDIA A100 or RTX 4090 (24GB VRAM) to run the ViT-L/G versions. If using the ViT-S/B versions, VRAM requirements can be reduced to 8–12GB. The commercial version can achieve real-time frame rates (≥30fps) on edge devices such as Jetson Orin, but it must be used with Orbbec Gemini 330 series cameras. Consumer-grade GPUs (such as RTX 3080 10GB) can only run the ViT-S version, with an inference speed of approximately 50ms per frame.

Q: How can I obtain the commercial version of LingBot-Depth? What is the pricing?
A: The commercial version must be applied for through official channels of Ant Lingbo Technology or Orbbec. It is typically provided in the form of an SDK, which includes an optimized depth completion model, a temporal alignment module, and camera drivers. Pricing depends on deployment scale and license duration, and specific details should be obtained by contacting the business team. Individual developers or small teams can first use the open-source LingBot-Vision for prototyping and then evaluate whether the commercial version is needed.

Q: How does the model perform in outdoor scenarios? Can it be used for autonomous driving?
A: The training data includes outdoor scenes, and the model performs well in conventional outdoor environments such as roads and buildings. It also shows significant advantages in scenarios involving transparent objects (e.g., glass facades). However, autonomous driving requires extremely high real-time performance (<10ms per frame), and the open-source version runs at approximately 80ms (ViT-L) on an A100, which cannot directly meet this requirement. The commercial version, through model quantization and edge optimization, can achieve 20–30ms, making it suitable for low-speed autonomous driving (e.g., logistics vehicles in a campus). Further optimization is still needed for high-speed scenarios.

9. Project Links

• Official Website: https://technology.robbyant.com/lingbot-vision
• GitHub Repository: https://github.com/robbyant/lingbot-vision
• HuggingFace Model Hub: https://huggingface.co/collections/robbyant/lingbot-vision
• Orbbec Official: https://www.orbbec.com (for obtaining commercial SDK and Gemini 330 camera information)