Vibe Coding XR: Google’s Playground for AI-Powered Spatial Prototyping

I’ve been watching the “vibe coding” trend with a mix of amusement and genuine curiosity. The idea that you can just describe what you want in plain English and have an LLM spit out working code is neat for 2D web stuff, but XR has always been the walled garden where that magic stops. Until now.

Google Research just dropped Vibe Coding XR, and honestly, it’s the first time I’ve felt excited about AI-assisted prototyping for spatial computing. Here’s the gist: they’ve wired Gemini Canvas into their open-source XR Blocks framework. You type something like “create a beautiful dandelion” and in under 60 seconds, you get a fully interactive, physics-aware WebXR app that runs on Android XR headsets like Samsung’s Galaxy XR. No Unity, no Unreal, no hand-coding perception pipelines.

How it works (the short version)

You open the XR Blocks Gem in Chrome on your Android XR headset or desktop. Type or speak your prompt. Gemini, armed with curated code templates and system prompts tuned for XR Blocks, figures out the scene layout, interactions, and physics. It generates the app on the fly. You pinch to enter XR mode, and there it is—your dandelion, animated, responsive to hand gestures. If you want to iterate, you tweak the prompt and regenerate. Share button creates a public link.

What’s clever is the “simulated reality” mode on desktop Chrome. You can test interactions before deploying to the headset. Depth sensing, hand tracking, physics—those work best on actual hardware, but for quick UI layout and spatial logic, the simulator saves a ton of headset-on/headset-off cycles.

Why this matters more than you think

XR prototyping has been a mess. You need to stitch together perception SDKs, game engine logic, and low-level sensor integration. That’s days of work for a single interaction test. Vibe Coding XR collapses that to minutes. It’s not meant for shipping production apps—it’s for rapid validation. “Does this UI feel right in 3D?” “Can users grab that object naturally?” Those questions can now be answered before you commit to a full build.

I’ve seen similar approaches before—OpenAI’s Codex demos, various AI-for-3D tools—but the XR-specific tuning here is what sets it apart. The system handles spatial reasoning automatically. You don’t need to tell it where to place objects or how to handle occlusion. Gemini’s long-context reasoning plus the XR Blocks templates handle that.

The technical bits worth knowing

The framework is built on WebXR, so it’s cross-platform by nature, but the Android XR optimizations are where the magic happens. Hand interactions, depth sensing, and physics are first-class citizens. The team is presenting this at ACM CHI 2026, and they’ve made the code open source on GitHub. You can also try the live demo right now.

One thing I appreciate: they’re not pretending this replaces traditional XR development. It’s a prototyping accelerator. If you’re an experienced XR dev, this saves you the grunt work of setting up basic interactions. If you’re a newcomer, it lowers the barrier to understanding what’s possible in spatial computing.

What could be better

I’d love to see more control over the generated output. Right now, you’re at the mercy of Gemini’s interpretation. For simple scenes it works great, but complex multi-object interactions might need manual tweaking. Also, the 60-second generation time is impressive for XR, but it’s not instant—you’ll wait a bit for each iteration.

Still, this is the kind of tool that makes me want to dust off my XR headset and start playing. The demo video shows a dandelion that blows away when you pinch it—simple, but it communicates the interaction model instantly. That’s the whole point: rapid, tangible feedback on spatial ideas.

Go try it. The link is live, the code is open, and the future of spatial prototyping just got a lot more accessible.