Artificial intelligence is learning to speak the language of health, trading bedside manner for bot-side insight.
OpenAI has introduced HealthBench, a new benchmark designed to assess how well artificial intelligence models perform in real-world medical scenarios, part of a broader effort to ensure such technologies are useful and safe in high-stakes environments related to health.
As the company noted in its blog post announcing HealthBench, improving human health will be “one of the defining impacts of AGI.” If developed and deployed responsibly, OpenAI said, large language models could expand access to health information, support clinicians in delivering high-quality care, and empower individuals to better advocate for their own health and that of their communities.
To build a tool grounded in real-world medical expertise, OpenAI collaborated with 262 physicians across 60 countries. The result is a benchmark that features 5,000 realistic health conversations simulating interactions between AI models and individual users or clinicians.
“The conversations in HealthBench were produced via both synthetic generation and human adversarial testing,” OpenAI said. “They were created to be realistic and similar to real-world use of large language models: they are multi-turn and multilingual, capture a range of layperson and healthcare provider personas, span a range of medical specialties and contexts, and were selected for difficulty.”
HealthBench evaluates the interactions across seven core themes, from emergency scenarios to global health, each designed to test how language models perform under varied and complex clinical conditions. Within each theme, model responses are scored using physician-authored rubrics that, in total, include 48,562 unique evaluation criteria assessing factors such as accuracy, communication quality and context awareness. Each response is scored by GPT-4.1, which determines whether the model meets the defined expectations.
For example, the emergency referrals theme tests whether a model can accurately identify urgent situations and recommend timely escalation of care. Other themes evaluate communication skills—such as inferring if a user is a medical professional and adjusting language accordingly—and the model’s ability to navigate uncertainty. HealthBench also examines whether models can interpret health data, recognize when key details are missing and seek clarification, and respond appropriately in global settings.
While the company highlighted notable progress, it acknowledged there is still room for improvement.
“Our findings show that large language models have improved significantly over time and already outperform experts in writing responses to examples tested in our benchmark,” OpenAI said. “Yet even the most advanced systems still have substantial room for improvement, particularly in seeking necessary context for underspecified queries and worst-case reliability. We look forward to sharing results for future models.”
The HealthBench evaluation framework and dataset are now publicly available on GitHub.
“One of our goals with this work is to support researchers across the model development ecosystem in using evaluations that directly measure how AI systems can benefit humanity,” OpenAI said.
Beyond healthcare, fitness and wellness companies are increasingly weaving AI into every aspect of the user experience, from smart equipment and recovery tools to member scheduling, health tracking and advanced personalization.
