The core of the research is the SM3-Text-to-Query Benchmark, which integrates multiple database architectures and query languages for the first time. The benchmark's name is derived from the three "Ms" (multi-modal and medical). Text-to-Query means that a text in natural language triggers a query in the database, e.g., "Find patient names with symptoms ABC in hospital XYZ." Based on synthetic, i.e. automatically generated and therefore data protection-compliant, patient data and a comprehensive health terminology (SNOMED-CT taxonomy), this benchmark supports various database architectures, enabling a comprehensive evaluation of the performance of different systems.

The researchers developed 408 question templates, which were expanded into a comprehensive benchmark set of 10,000 natural language/query pairs for each query language. These query languages include four well-known languages (SQL, MQL, Cypher, and SPARQL). The goal of the experiments with these data was to determine how well large language models (LLMs) can translate natural language into these four different query languages. Two open (both LLama from Meta) and two closed language models (Gemini from Google and ChatGPT from OpenAI) were analyzed.

The study's results show significant differences in the performance of the various database models and query languages. In the future, the benchmark is expected to be adapted to additional query languages or real, standard-based patient databases, making it a valuable tool for future research and development in the field of digital healthcare. With their work, the researchers set new standards in the evaluation of text-to-query systems. This standardised and privacy-preserving benchmark provides the scientific community and the general public with valuable insights and solutions to the challenges of modern healthcare.