A RAG system can look great in a demo and still fail in real use. Demo questions are easy. The documents are clean. The people watching are friendly. Real users ask unclear questions about messy records. In healthcare, a wrong answer can hurt someone. This is why a team must test for hallucinations before launch.
Research gives a good reason to be careful. A team at Yale asked 18 medical experts to check the answers of RAG systems. The experts made more than 80,000 notes. The team found that RAG often did worse than the same model with no retrieval at all. This was a preprint, and it tested only two models, so its numbers will not match every system. Still, the message is clear. RAG in healthcare needs to be checked one step at a time, not only by looking at the final answer.
Check the Documents First
Many wrong answers start before anyone asks a question. They start when documents are turned into text.
Prompt-based extraction is one way to do this. A language model gets a prompt or a form and pulls out set fields, such as a drug name, a date, or a lab result. Several vendors offer this, including ABBYY, Mistral, and Docsumo.
The test is simple. First, a person types the correct values by hand for a set of sample documents. The sample should include faxes and poor scans. Then each value that the system pulled out is compared with the correct one. Two numbers matter. One is how many values were wrong. The other is how many wrong values came with no warning.
The prompt should also let the system say “not found.” The test should check that the system does say it, and does not guess. Each value should link back to the page it came from. Then a reviewer can check it fast.
Build a Test Set With Doctors and Nurses
A test set is a list of real questions with correct answers. Clinicians should write these answers or approve them.
The Yale team used “must-have facts.” These are the facts that a good answer has to include. Any team can copy this idea. For each question, a clinician lists the facts that must appear. Later, the scoring checks if they did.
The Yale team used 200 questions. A first test set can be smaller. It should still include common questions, rare cases, and questions that the documents cannot answer. That last group is important. A safe system should say “I do not know” when the answer is not there.
Check Each Step on Its Own
One score for the whole system hides the real problem. The Yale team split the work into three steps. Any team can do the same.
The first step is finding the right text. In the study, only 22 percent of the top 16 passages were useful. For 31 percent of the questions, none of the 16 passages was useful. Together, the passages held only about one third of the facts that the answers needed.
The second step is choosing which text to use. The model often picked the wrong passages. Only about 41 to 43 percent of its choices were right.
The third step is writing the answer. Experts checked each statement to see if it was true. They also checked whether the answer included all the must-have facts.
Free tools such as Ragas measure similar things. “Faithfulness” asks if the claims in an answer are backed by the text that was found. “Context precision” asks if the best text was ranked first. “Context recall” asks if all the needed text was found. Recall needs a correct answer to compare with, and the clinician test set provides it.
Test the Known Weak Spots
Overall scores can miss special traps. Each trap needs its own test.
Numbers. The Yale team saw that found text can pull a model toward the wrong number. In one case, a limit from a different setting changed how a lab result was judged. Tests should include doses, units, and ranges from sources that do not match the question well.
Words with two meanings. One question about poison ivy returned text about a comic book character. Medical terms, short forms, and brand names can cause the same mix-up.
Old or clashing documents. Other research warns that errors can enter when documents are out of date or disagree. A test should include such pairs on purpose. This shows which one the system trusts.
Questions with no answer. The system should say it cannot answer. It should not make something up.
Made-up sources. In the study, some models named sources that were not in the text they had found. For the smaller model, about 77 percent of these sources could not be checked. In every test, each source should be checked to make sure it is real and supports the claim.
Use Both Machines and People to Score
A judge model can score thousands of answers quickly. But it has its own faults and needs to be checked too. A good plan is to use automatic scoring on every new version. Then clinicians review a sample each time.
Reviewers will not always agree. In the study, agreement on whether a statement was true was only fair. For risky items, two reviewers should score on their own. Their differences should be written down and talked over.
Decide What “Ready” Means
The team should write down its standards before testing starts. Then nobody can argue about the results later. Examples are a limit on how often unsupported claims may appear, a minimum share of must-have facts covered, and no wrong doses at all in the test set. Clinical, legal, and privacy staff should set these limits with the technical team.
The research also shows that things can get better. The Yale team removed useless passages and rewrote the questions. This raised scores on harder tests by up to 12 percentage points. The team also noted that RAG helped less when the model was already strong. A small test can show if RAG adds value for a given job.
Keep Testing After Launch
Going live is a checkpoint. It is not the end of the work. The team should save real questions, review a sample every month, and add every failure to the test set. Over time, this set becomes a useful record of how safe the system is.
