NVIDIA Drags AI from Virtual World into Reality

On June 16, NVIDIA released the XR AI library.

The core logic of this library is to feed real-world signals captured by AR/XR devices—video, audio, depth images, sensor data—into AI large models, enabling AI to perceive environments, invoke tools, retrieve enterprise knowledge bases, and then output auxiliary information on-site.

To put it more bluntly: give AI a pair of eyes to see the real world.

Engineers wear lightweight AR眼镜to troubleshoot PLC equipment failures, and AI links digital twins and automated production lines in real-time, pushing maintenance guidance directly before their eyes. Researchers conduct stem cell experiments, and AI automatically records standardized processes without needing to manually check operation manuals. Surgeons perform operations, and AI intelligently avoids visual field obstructions and pushes diagnostic information at key positions.

These are not sci-fi scenarios, but real cases already implemented.

NVIDIA's library integrates four core capabilities:

Receive real-world signals—the eyes and ears of AR/XR devices, digitizing the physical world.

Connect professional tools—NVIDIA Metropolis for visual AI, VSS for video retrieval, NeMo retriever for enterprise knowledge bases.

Compatible with large model ecosystems—NVIDIA's own Nemotron, Cosmos Reason, and other mainstream foundational models.

Intelligent agent scheduling engine—one-stop solution from prototype development to commercial deployment.

This technology is not just paper talk. Siemens uses XR AI and DGX Spark to build workshop auxiliary systems, and engineers can repair equipment by wearing glasses. AutoBio's Rana launched LabOS research operating system, relying on spatial agents to assist stem cell and CRISPR gene editing experiments, compatible with Meta, Rokid, VITURE AI glasses. Pittsburgh University Medical Center used XR AI and DGX workstations to create surgical auxiliary schemes, with systems intelligently avoiding surgical field obstructions and pushing diagnostic information.

NVIDIA calls these AI agents "new digital workforce."

Workers in factories, doctors in hospitals, researchers in laboratories don't need to become AI experts. Just put on AR眼镜, and AI agents can provide professional assistance in real-time.

The business logic behind this is clear: computing foundation (DGX workstation) + development platform (XR AI library) + terminal device (AR/XR glasses) = complete commercial closed loop.

NVIDIA has already established a moat in computing power, and now through the XR AI library extends this moat to every corner of the physical world.

In the past few years, AI has evolved mainly in the digital world—chatbots, image generation, code assistants. These are all pure digital interactions.

But the emergence of the XR AI library marks the beginning of AI entering the stage of physical interaction.

Factory workshops, operating rooms, laboratories—these physical scenes will become the next battlefield for AI.

And AR/XR glasses are the entry devices for AI to enter the physical world.

When AI can see, understand, and intervene in the physical world, it's no longer a tool, but a digital workforce with environmental perception, tool invocation, enterprise knowledge base retrieval, and human-machine collaboration capabilities.

At that time, we might no longer ask "what can AI do," but "what can't AI do"?

NVIDIA's move drags AI from the virtual world into reality. And the real world is much larger than the virtual world.