New research from Dartmouth College reveals that correcting medical errors made by artificial intelligence, along with AI forgetting to schedule follow-up appointments, drains doctors' time. However, a new training method may help solve this problem, as written by Lucia Auerbach*.

Patient confusion due to AI responses

The spread of AI is almost as fast as the spread of the flu in emergency room waiting areas; perhaps the adoption of this technology in healthcare is why you have to wait longer for an appointment.

A new study from Dartmouth College has found that AI errors can cost doctors valuable time when filling out medical records. Errors and irrelevant details force doctors to spend time correcting AI responses, time that could have been spent treating or talking to patients.

Comparing AI-generated responses with actual responses

The study was presented at the 2026 annual meeting of the Association for Computational Linguistics and published in the conference proceedings. It was the first large-scale study of a patient portal using AI to draft responses to them. The researchers developed a tool that compares AI-generated responses with a dataset of real responses written by healthcare professionals from Dartmouth Health. They analyzed 146,000 conversations between 10,105 patients and primary care doctors in a large rural healthcare system. Additionally, they used the tool to evaluate doctor responses using Claude, Gemini, ChatGPT, Llama, Aloe, and Qwen.

Dr. Sarah Preum, the lead author of the study, said in a press release: 'We found that AI may sound like a doctor, but it doesn't think like one.'

AI and mismatch

The results indicate that AI-generated responses are often inconsistent with what doctors actually write. This includes responses that are too long, contain irrelevant or inaccurate medical details, or lack follow-up questions. In one case, the AI on the portal suggested a 32-year-old woman taking heartburn medication, who was worried about persistent nausea, adjust her diet. The doctor ignored that suggestion and asked whether there was any chance she could be pregnant.

Patient questions and risks to the elderly and pregnant women

Among all the gaps identified by the researchers, the lack of clarifying follow-up questions stood out. This is a problem, Preum tells Inc. magazine, because a follow-up question often guides care in the right direction.

This is especially true for symptom reporting messages; asking the wrong question, or not asking any question at all, may lead the patient down a wrong diagnostic or treatment path.

Preum added that the risks are greater for the most vulnerable groups, including the elderly, patients with multiple chronic conditions, people receiving immunosuppressive or cancer treatment, and pregnant women.

Preum added, 'The model can always generate an answer without asking any questions first, and that's not a flaw. But the real flaw is that a real doctor, upon receiving the same message, would have asked a clarifying question before responding. When the model skips this step, it's not working efficiently; it's just guessing.'

Generating personalized messages

Nevertheless, there are some potential benefits to this new technology in healthcare. The researchers found that by adapting AI to individual doctors' communication styles, accuracy can be improved by 33 percent, and editing can be reduced by up to 26 percent. The study concluded that AI responses can be useful when tailored to the doctor's needs.

The researchers created a technique called TADPOLE, which stands for Thematic Agentic Direct Preference Optimization for Learning Enhancement, that trains AI platforms using a hybrid model composed of doctor responses and AI responses. When they integrated TADPOLE with six commercial learning management systems, they found that pre-written responses matched the doctor's standards for accuracy and information quality better, saving busy doctors one to two hours of work daily.

Risks of deploying AI without evaluation

Deploying these tools widely before evaluating them for safety, bias, and other responsible AI practices is a real risk, especially for the most vulnerable patient populations. It also poses a potential risk to healthcare providers and systems; any incorrect or misleading AI-generated response could lead to significant legal liabilities.

* Inc., Tribune Media Services.