Google’s Vantage Experiment: Using GenAI to Score ‘Future-Ready’ Skills

Google Research dropped something interesting this week: Vantage, a research experiment that uses generative AI to assess so-called “future-ready” skills — critical thinking, collaboration, creative thinking. The pitch is straightforward: standardized tests are terrible at measuring these competencies, so why not let an AI simulate a team scenario and score how you handle it?

The study, done with New York University, claims the AI’s scoring is on par with human experts. That’s a bold claim, and honestly, I’m a bit skeptical. But let’s dig into what they actually built.

The setup: AI avatars, not multiple choice

Vantage drops students into conversations with AI avatars. Think preparing for a debate or pitching a creative idea. The avatars are steered by an “Executive LLM” that dynamically introduces challenges — pushing back on ideas, creating conflict, that sort of thing. It’s like an adaptive test, but instead of picking A, B, or C, you’re talking your way through a simulated mess.

This is clever. Traditional assessments for soft skills are either laughably rigid (multiple choice on “how would you handle a disagreement?”) or impossibly resource-intensive (trained observers, standardized scenarios, hours of grading). Vantage tries to split the difference: controlled environment, but open-ended interactions.

The key insight here is that the AI doesn’t just grade your final answer. It watches how you navigate the conversation, how you respond when the avatar disagrees, whether you build on ideas or shut them down. That’s genuinely harder to fake than a written response.

Does it actually work?

The research team ran a study comparing AI scoring against human expert scoring. They claim parity. The tech report is linked in their post, and I’d want to dig into the specifics — what was the inter-rater reliability? How many scenarios? What’s the variance? — but the direction is plausible.

Here’s the thing: LLMs are surprisingly good at evaluating structured conversations. They’ve been trained on millions of examples of human dialogue, and scoring rubrics are essentially pattern matching. If you define “good collaboration” as specific behaviors (acknowledging others’ points, asking clarifying questions, proposing compromises), an AI can absolutely spot those patterns.

But there’s a catch. The AI is also generating the scenarios and steering the conversation. That creates a feedback loop that could reinforce whatever biases are baked into the model. If the Executive LLM thinks a certain communication style is “better,” it might steer toward confirming that bias. The researchers say they controlled for this, but I’d want to see independent audits.

The bigger problem: Are we measuring the right thing?

My real concern isn’t whether the AI can score these skills. It probably can, at least to some degree. The question is whether these “future-ready” skills are actually what we think they are.

The OECD and WEF frameworks that Google cites are political documents as much as pedagogical ones. They reflect a particular vision of what workers should be — adaptable, collaborative, creative. That’s fine as far as it goes, but it also conveniently aligns with the needs of tech companies that want employees who can pivot quickly and work in teams without pushing back too hard.

More to the point: if you define “critical thinking” as what an LLM can reliably score, you might end up teaching students to perform critical thinking in ways that are legible to AI, rather than actually developing the skill. It’s the same problem we’ve seen with standardized tests for decades — teaching to the test, except now the test is a chatbot.

What Vantage gets right

To be fair, Vantage is a research experiment, not a product. It’s on Google Labs, which means it’s explicitly experimental. The team is being transparent about the methodology and the limitations. That’s more than most AI education tools can say.

The sandbox environment is genuinely useful. Even if the scoring isn’t perfect, the practice of navigating simulated team dynamics has value. Students can experiment with different approaches, see how the AI responds, and reflect on their own patterns. That’s a lot better than reading a textbook chapter on “collaboration.”

I also appreciate that they’re not pretending this replaces human interaction. The goal is scalable, consistent assessment — not better assessment. If you need to evaluate 500 students on collaboration skills, Vantage is probably more reliable than a single overwhelmed teacher. But it’s not a substitute for real mentorship or coaching.

The bottom line

Vantage is a solid research effort from a team that clearly understands the limitations of traditional assessment. The AI scoring is probably good enough for formative feedback — telling a student “you tended to interrupt when the avatar was explaining their idea” is actionable and useful. Whether it’s good enough for high-stakes assessment is another question entirely.

I’d love to see this open-sourced or at least made available for independent researchers to poke at. The methodology is interesting, but the real value will come from understanding where it fails. For now, it’s worth signing up if you’re curious, but don’t expect it to replace your hiring manager or your professor just yet.