The Research and Results are Published in Preprint on arXiv, Continuing the Company’s Commitment to Provide Transparency on its Methodology
PALO ALTO, Calif.—July 2, 2024— Today, Atropos Health, a pioneer in translating real-world clinical data into personalized, real-world evidence and insights, published a whitepaper outlining how ChatRWD™ beta outperformed other LLMs on five quality measurements.
For the study, five measures were developed to judge performance:
-
Was an answer provided?: This measures the percent of time the LLM was able to generate a response – any answer at all.
-
Were there limitations to the answer?: There were two main areas where answers were not complete
-
Was the question that was asked answered?: Sometimes when LLMs provide an answer, the response does not answer the question that was asked. This measures the percent of time the response generated answered the questions that was asked.
-
Did the LLM hallucinate?: An LLM hallucinated if it answered the question but the scope of the answer was not correct (example: answer was about breast cancer and the question was about prostate cancer). Another form of hallucination comes from the data the LLM used. Common hallucinations in this study were: not-relevant citations and non-existent citations.
-
-
Was the answer supported by credible evidence?: This occurred when an answer was provided that answered the question that was asked with evidence. This was the highest level of performance.
Two subjective measures were also used to evaluate the impact and quality of the answers provided by each LLM:
-
Trust: For an answer to be “trusted” an independent physician had to deem the answer to be high quality enough to inform their practice.
-
Best Answer: Independent physicians provided a qualitative interpretation on which answer was the best.
>
“As generative AI technologies move from hype to utilization in healthcare, it is critical that their relevance, reliability, and actionability are rigorously measured and independently verified,” said Atropos Health Chief Medical Officer and co-founder Dr. Saurabh Gombar. “High-quality clinical evidence must remain the cornerstone of value in healthcare. We put our technology to the test, and the results were clear: ChatRWD has no alternative. Independent physician evaluators trusted ChatRWD’s results, and while other LLMs were complementary when answering well studied questions, only ChatRWD consistently was able to create evidence for novel questions.”
Learn more
About Atropos Health
Atropos Health is the developer of GENEVA OS™, the operating system for rapid healthcare evidence across a robust network of real-world data. Healthcare and life science organizations work with Atropos Health to close evidence gaps from bench to bedside, improving individual patient outcomes with data-driven care, expediting research that advances the field of medicine, and more. We aim to transform healthcare with timely, relevant real-world evidence.
To stay up-to-date with Atropos Health, connect through LinkedIn or follow on X @AtroposHealth.