People now ask AI assistants for help finding care. They don't just ask what's wrong. They ask who to see, whether that doctor takes their insurance, and when they can get in.
We wanted to know what happens next. So we ran a simple test.
The test
We played one patient: 58 years old, right knee pain for three months, an MRI showing a torn meniscus, UnitedHealthcare Medicare Advantage. A common case that large health systems want to treat.
For each of 86 large U.S. health systems, we opened a fresh chat in a popular AI assistant, with web search on, and used a ZIP code near the system's main hospital. We asked the same four questions every time:
- Who's the best knee surgeon near me that takes my plan?
- Which surgeons at [the health system] treat this? Do they take my plan? Where and what's the number?
- When is the soonest new-patient appointment?
- Can you just book it for me?
We didn't book anything or enter any real patient data. We checked every answer against the health system's own website.
What we found
Booking: 0 of 86. The assistant could not book a single appointment. In 15 chats it gave a direct link to a scheduling page, and in 3 of those it showed real open times. In the other 71, the patient got a phone number or a general website link.
Insurance: 0 of 86 confirmed for the patient's plan. In 24 chats the assistant gave a clear answer backed by the hospital's own list of accepted plans. In the other 62, the answer was some version of "probably, but check with your plan."
Being found: 37 of 86 had someone else named first. When the patient asked for the best knee surgeon, a different practice or hospital came up ahead of the health system in 37 chats. In 15, the system's own doctors never came up in that first answer at all. Some systems were named first only as a warning that they might be out of network.
Accuracy: mostly right, with costly misses. Addresses and doctor names were usually correct. But 15 of 86 chats had at least one costly error: a wrong office phone number, a fax number given as the office line, or a wrong answer about insurance. In two, the assistant told the patient the hospital was in network, or likely in network, even though the hospital's own website says that insurer's Medicare Advantage plans leave the network on January 1, 2027. That was during open enrollment, when the patient could still switch plans.
Scores change from day to day. We tested 18 systems twice, three days apart. Some scores moved by more than 20 points out of 100. A health system's place in these answers is not fixed, and it is not something it can check once.
What stood out
The assistant was already doing triage. In every one of the 86 chats, without being asked, it told our patient that many meniscus tears at 58 are wear-related and that physical therapy often comes before surgery. In most chats it also brought up referrals.
That's useful advice. But it happened without the health system. The assistant decided where this patient should go based on public reviews, directories and general guidance, not on the hospital's own rules for who should see which doctor, or on what the patient's plan requires.
Every health system already triages patients. Schedulers and nurses do it on the phone every day: what are your symptoms, have you had imaging, have you tried therapy, does your plan need a referral, which doctor is the right fit. That process doesn't exist yet inside the AI assistant's conversation.
What this does and doesn't mean
This was one assistant, one patient scenario and one week. AI answers change from day to day, so treat any single result as a snapshot, not a grade.
But the pattern held across all 86 systems. A patient who was ready to book left every conversation with homework: call this number, check with your plan, ask about a referral. More than a third were sent somewhere else first.
What we're working on
We're building the triage step for these conversations: checking a patient's symptoms and records against the clinical criteria a hospital and the patient's plan already use, so the right patients reach the right doctor at the right step, with their plan's requirements already checked.
If you lead AI, digital, marketing, patient access or a service line at a health system, we'd like to show you how your system shows up today and talk through what we found. Reach me on LinkedIn.
How we scored it
Each system got a score out of 100 across four areas:
- Found (30): named in the first answer, and with what detail
- Accurate (30): facts checked against the system's own site, with points off for costly errors
- Insurance clarity (15): confirmed for the patient's plan
- Bookable (25): booked in the chat, or a direct link to booking
The median score across the 86 systems was 45. The range was 13 to 74.


