Google Cloud has debuted at HIMSS25 new generative AI capabilities in its Vertex AI Search for healthcare, including a new feature called Visual Q&A that searches tables, charts and diagrams, and the availability of Gemini 2.0 as one of the models now within the offering.

Vertex AI Search for healthcare's new, multimodal search capabilities are designed to help organizations give doctors, nurses and others a more comprehensive view of patient health.

Vertex AI Search for healthcare is a specialized tool within Google Cloud's Vertex AI platform that helps developers build better assistive technology for searching and retrieving information from complex medical data within health records, medical documents and other healthcare data sources.

Now with Visual Q&A and Gemini 2.0, the search technology can receive images such as tables, charts or diagrams directly as an input – instead of taking the image and first converting it into text. For example, it could receive a diagram of a patient's feet indicating where they have an injury, alongside a checkbox from a form the patient filled out indicating if they are wearing therapeutic shoes or orthotic inserts and any other symptoms indicated on the form.

Visual Q&A can also analyze medical forms where information is entered into tables – and when this information is a relevant part of a search, it will include it in the findings for the clinician.

Gemini 2.0 is capable of understanding native image and audio output and high-quality speech generation, among other features. For example, a medical researcher could input a large dataset of patient brain MRIs. Gemini 2.0 could then analyze these images directly, identifying subtle patterns and anomalies that might be missed by the human eye, and then generate a detailed report summarizing the findings. This allows for faster analysis of large imaging datasets.

Healthcare IT News sat down with Aashima Gupta, global director, healthcare, at Google Cloud, for a deep dive into the new technologies.

Q. What will the new generative AI capabilities in Vertex AI Search for healthcare mean for hospitals and health systems?

A. Both new generative AI capabilities – Visual Q&A and Gemini 2.0 – represent a significant step forward in how hospitals and healthcare organizations can access, view and use critical patient information. They allow providers to more easily find information within a patient's medical record and importantly, paint a more complete picture of that patient's health than previously possible.

Both new genAI capabilities are what we call multimodal, meaning they can process and integrate information from diverse sources of patient data – not just text, but also medical imagery and genetic information.

This is meaningful, because in healthcare nearly 90% of healthcare data is in image form, for example, X-rays and scans. What Visual Q&A and Gemini offer – by combining data from images, patient history, genetic information, etc. – is a comprehensive view of a patient's health that not only saves providers time, but also helps clinicians make accurate diagnoses and design personalized treatment plans.

Q. What do the Visual Q&A and Gemini 2.0 technologies do to make Vertex AI Search provide better results?

A. Visual Q&A and Gemini 2.0 enhance the capabilities of Vertex AI Search for healthcare by improving both the input process for providers and the output provided to those providers.

Visual Q&A allows Vertex AI Search to directly analyze images like tables, charts and diagrams, which are common in medical records. Instead of converting these images to text, which can lose important context, Visual Q&A understands the information contained within the image itself.

This means clinicians can search using the visual medical content itself, leading to more comprehensive and relevant results.

Gemini 2.0 improves the speed and accuracy of the search process by providing faster, more accurate answers through its improved ability to process multimodal data, including those same visual elements. This means clinicians get faster, more accurate answers to their queries, even when those queries involve complex images.

Q. Please offer a detailed example of what Gemini 2.0 is capable of, and how a provider can make use of it.

A. Imagine a physician is treating a patient with a complex medical history, including multiple imaging scans and lab reports. The physician needs to quickly understand the progression of a specific health condition, including any relevant details from the imaging scans.

With Gemini 2.0 powering Vertex AI Search for healthcare, the physician can simply ask a question like, "Show me the trend of the patient's blood glucose levels over the past six months and highlight any areas of concern in their recent abdominal CT scan." Gemini 2.0 can then quickly analyze both the lab data and image data to provide a comprehensive answer.

It could, for example, display a graph of the blood glucose levels with annotations indicating periods of instability, alongside a marked-up version of the CT scan highlighting potential areas of concern. This allows the physician to quickly see the key information they need – saving them time and allowing them to make the most informed treatment decisions.

Q. How does Google Cloud assure hospital and health system users of its newest genAI capabilities that they can trust the genAI to be accurate?

A. When it comes to AI in healthcare, accuracy and trustworthiness are paramount. At Google Cloud, we are committed to the highest standards for our customers and their patients. To ensure accuracy and build trust with our customers, we take the following approach.

We subject our models to rigorous testing and validation using diverse datasets and benchmarks. This includes measures like the OSCE (Objective Structured Clinical Examination) model, which helps evaluate the model's clinical reasoning abilities.

We strive to make our AI models as transparent and explainable as possible. This helps users understand how the model arrived at its answers, building trust and allowing for better human oversight.

We work closely with healthcare providers and organizations throughout the development and deployment process. This ensures our AI tools are aligned with real clinical needs and meet the highest standards of accuracy and reliability.

We continuously monitor the performance of our AI models and actively seek feedback from users to identify areas for improvement. This iterative approach ensures that our AI tools remain accurate and reliable over time.

We are committed to developing and deploying AI responsibly, with a focus on fairness, privacy and safety. We actively participate in initiatives like the Health Care AI Code of Conduct and Standing Together to ensure our AI tools meet the highest ethical standards.

We have a comprehensive approach to HIPPA compliance, including security and privacy design. The security and compliance measures that allow us to support HIPAA compliance are deeply ingrained in our infrastructure, security design and products. Because of this, we can offer a cloud infrastructure where our health customers can securely store, analyze and gain insights from health information, without having to worry about the underlying infrastructure.

AI is continuing to reshape the healthcare landscape. By adhering to these practices, we aim to build AI solutions that healthcare providers can trust to be accurate and reliable, and most importantly, beneficial for patient care.

Follow Bill's HIT coverage on LinkedIn: Bill Siwicki
Email him: bsiwicki@himss.org
Healthcare IT News is a HIMSS Media publication

WATCH NOW: Seattle Children's Chief AI Officer talks better outcomes through the technology