I Tested Google’s Diagnostic AI in a Real Clinic: Here’s What I Found

Google’s been working on a conversational medical AI called AMIE for a few years now. They’ve shown it can handle diagnostic challenges in simulated settings and even outperform some clinicians in controlled tests. But simulations are one thing. Real clinics are another.

Now they’ve taken the next step: a prospective, single-center feasibility study at Beth Israel Deaconess Medical Center (BIDMC) in Boston. The results just dropped, and I’ve been digging through the paper. Here’s what actually happened.

How the study worked

AMIE wasn’t diagnosing patients on its own. Instead, it acted as a pre-visit history taker. Patients scheduled for new, non-emergency primary care appointments were invited to chat with AMIE via a secure web link before their doctor’s visit.

An overseeing physician monitored the conversation live via video call with screen sharing. This “AI supervisor” could jump in if the system hit predefined safety criteria—things like asking about suicidal ideation or handling sensitive disclosures. The system then generated a transcript and summary that the clinician reviewed before seeing the patient.

This oversight model isn’t new. It’s basically the same way we train medical residents: supervised interaction with patients, with a senior doctor ready to step in. The difference here is that the “trainee” is an AI.

What they actually measured

The study focused on feasibility, not diagnostic accuracy. They wanted to know: can this thing work in a real clinic without causing harm? Can patients and clinicians tolerate it?

Key metrics included:

• Safety: Were there any adverse events? Did the AI miss critical information?
• Usability: Could patients complete the chat without frustration?
• Clinician perception: Did doctors find the summaries useful?

The results are… cautiously positive. No serious safety incidents were reported. Patients generally completed the interactions, though some dropped out due to technical issues or time constraints. Clinicians reported that the summaries were helpful for triage, but not all of them found the AI’s output clinically actionable.

What’s missing

Here’s where I get skeptical. The study was single-arm—no control group comparing AMIE to standard pre-visit questionnaires or phone triage. That makes it hard to say whether the AI actually improved outcomes or just added another layer of friction.

Also, the patient population was small and self-selected. People who agree to chat with an AI before a doctor’s visit might be more tech-savvy or less anxious than average. We don’t know how this scales to elderly patients, non-English speakers, or people with low digital literacy.

And let’s be honest: the oversight model is expensive. Having a physician watch every AI-patient conversation defeats the purpose of scaling access. Google acknowledges this, saying future work should explore “delegated oversight” or automated safety monitoring.

Why this still matters

Despite the caveats, this is a real step forward. Most AI-in-medicine papers are either synthetic simulations or retrospective analyses of existing data. This is a prospective, IRB-approved study with real patients in a real clinic. That’s rare.

AMIE’s ability to conduct a coherent, clinically relevant conversation—without hallucinating dangerous advice—is impressive. The system asks follow-up questions, clarifies ambiguous answers, and avoids leading the patient. That’s harder than it sounds.

If this can eventually reduce the time doctors spend on history taking—which is a huge chunk of primary care—it could free up bandwidth for actual diagnosis and treatment. But we’re years away from that.

The bottom line

Google’s AMIE study is a proof of concept, not a product launch. It shows that conversational diagnostic AI can work in a controlled clinical environment with tight safety rails. But the path to widespread adoption is long, and the hurdles are more about workflow integration and trust than technical capability.

I’d like to see a randomized trial comparing AMIE to standard care, with cost and time metrics included. And I’d like to know how the system handles edge cases—like patients who lie, or those with complex multi-morbidity.

For now, this is a promising start. But I’m not ready to let an AI take my medical history just yet.