A Federal Agency Published Its Own Test Queries. Almost Nobody Ran Them

 When a federal agency releases its own test queries, this provides researchers, developers, and organizations the ability to test their system against real-world queries.  However, the real benefit comes from how many people actually test, analyze, and use these queries to identify potential issues. This helps to identify some interesting points such as how effective public testing can be, how hard it can be to access a reliable testing tool, and the difference between providing a testing tool and using it.

What Was Actually Published

The US Department of Health and Human Services Office of Inspector General released a toolset describing how it identifies diagnosis codes at high risk of being baseless by the medical records behind them. It included the selection logic and the database queries, released so institutions could run the same checks on their own data.

For circumstances on why this matters commercially: US insurers covering Medicare patients are paid more for covering an ill person, and illness is instituted through diagnosis codes drawn from clinical notes. Those codes carry money. The toolkit told everyone precisely which ones the regulator would come looking at.

Then It Ran The Test

Across its audit work using that logic, the department reports that approximately 70 percent of the high-risk codes it analyzed were not supported by the connected medical records, with some categories baseless more than 90 percent of the time.

One national audit sampled 97 enrollee records containing an acute stroke diagnosis. None were supported as acute strokes by the substantiation behind them.

The breakdown of those 97 is the part engineers find entertaining. In 68 cases, the patient had truly had a stroke, properly documented in their medical history. The submitted code described it as an active event instead of a historical one. Twenty-two records comprised no exact stroke documentation at all, four could not be located, one documented a different condition, one carried a written name from a source not accepted for the purpose, and one was unclear.

So the controlling failure mode was a temporal attribute, correct at one point, never updated, and read subsequently as current.

Why Nobody Ran The Queries

This is the part significantly taking seriously if you run systems anywhere.

The software these institutions used was built to find diagnoses that had been missed. It expected that given a record, estimate which codes probably apply, rank by confidence, and hand a human the list. Finding more created revenue, so finding more is what got engineered.

A ranked applicant list has no element meaning “this existing record is wrong.” You can score something low. You cannot emit a negative resolution, because the output type does not contain one.

Which means running the controller query would have produced findings that the present stack had nowhere to put. You would get a list of problems and no workflow, no owner, and no restoration path. That is a much bigger enterprise problem than a SQL job, and it is why a free, produced, self-administered test went largely unutilized for thirty months.

The Pattern, Stated Generally

Having a test is not the same as being able to act on its output. An institution will quietly refuse to run a check when the result would land somewhere nothing is built to accept it.

Three questions that surface this in your own environment.

• Is there an issued standard in our sector that we could measure ourselves against today and have not? If yes, what expressly is stopping us, and is the honest answer that we would not know what to do with the output?
• Can our systems signify a negative finding at all? Not score it lower. Mark it wrong, and route it somewhere.
• Who receives a failure when one is created? If no role owns that, your detection expertise is decorative.

What Changed Afterwards

The institution that responded properly reconstructed the review layer so it can return a negative result, then ran the published test on itself before the external one. The formal name for that outermost check is a risk adjustment data confirmation audit, and a purpose-built radv audit solution now exists specifically to manage it: tracking what was analyzed, when, and on what confirmation, so a finding has somewhere to go.

The wider lesson costs nothing to apply. If someone has published the test you will ultimately be graded against, the only reason not to run it is that you are not ready to see the answer.

FAQs