AI and Human Doctors Collaborate to Enhance Decision-Making

On October 23, 2025, the Whiting School of Engineering’s Department of Computer Science hosted a talk by Aaron Roth, a professor of computer and cognitive science at the University of Pennsylvania. The presentation, titled “Agreement and Alignment for Human-AI Collaboration,” delved into the complexities of how artificial intelligence can assist humans in making critical decisions across various sectors.

Roth discussed findings from three key papers: “Tractable Agreement Protocols,” presented at the 2025 ACM Symposium on Theory of Computing, “Collaborative Prediction: Tractable Information Aggregation via Agreement,” from the ACM-SIAM Symposium on Discrete Algorithms, and “Emergent Alignment from Competition.” His work addresses a pressing question: how can AI effectively support human decision-making?

A prime example Roth provided was the use of AI in healthcare, specifically in assisting doctors with patient diagnoses. The AI system would analyze a range of factors, including previous diagnoses, blood types, and symptoms, to generate predictions. The human doctor then evaluates these predictions, utilizing their own insights, such as the patient’s overall appearance.

In instances of disagreement, Roth explained that both the doctor and the AI can engage in a dialogue, reiterating their viewpoints for a set number of rounds. This iterative process allows both parties to integrate each other’s unique perspectives until they reach a consensus. This collaborative agreement is grounded in what Roth refers to as a “common prior,” where both the AI and the physician start with shared assumptions about the world, despite possessing different pieces of evidence.

Roth introduced the concept of Perfect Bayesian Rationality, which describes this process where both parties understand the knowledge the other holds, yet do not have access to the specific details. While this framework is promising, Roth acknowledged its limitations, particularly the challenge of establishing a common prior. The complexity of real-world scenarios, such as hospital diagnostic codes, can make achieving agreement particularly difficult.

To navigate these challenges, Roth discussed the role of calibration. He likened this to testing a weather forecaster to determine the accuracy of their predictions. Effective calibration involves designing assessments that can evaluate whether the forecasts align with true probabilities. He also introduced the idea of conversation calibration, where claims made by one party are influenced by the prior statements of the other party. For example, if an AI estimates a 40% risk associated with a treatment and the doctor assesses it at 35%, the AI’s subsequent claim will reflect this input, leading to a more refined risk assessment.

Roth pointed out that while this discussion assumes aligned goals between human and AI, misalignment can occur, particularly when the AI is programmed by external entities, such as pharmaceutical companies. In these cases, the AI may prioritize certain medications over others, potentially compromising the treatment’s objectivity.

To counteract potential biases, Roth advised that doctors consult multiple language models (LLMs). By engaging with various AI models, healthcare professionals can evaluate differing recommendations, ultimately selecting the most suitable treatment for their patients. This competitive dynamic among AI providers could lead to enhanced alignment and reduced bias in their recommendations.

Roth concluded by emphasizing the importance of understanding real probabilities—those that accurately describe how the world operates. While these probabilities provide the best insights into reality, their precise application is not always necessary. Often, accurate and unbiased estimates can be derived from data without needing the full complexity of perfect reasoning. This collaboration between doctors and AI, Roth suggested, could pave the way for more informed decisions regarding treatments and diagnoses, ultimately enhancing patient care.