Background and Aims: Large language models (LLMs) are machine learning-based systems designed to process human language and answer users' questions. Today, the growing use of these conversational tools in various fields, including healthcare and medicine, can significantly impact users' awareness. Despite their considerable benefits, concerns persist regarding the quality and reliability of their answers and guidance. Therefore, this study aimed to conduct a comparative analysis of three
AI tools based on LLM in answering questions related to breast cancer and evaluate the reliability of their responses and the references. Methods: In this study, the free versions of three
AI tools ChatGPT-۴, Microsoft Copilot, and GROK۳ (beta) were evaluated. First, ۱۰ key questions about breast cancer were identified through interviews with female students aged ۲۰ to ۲۵. These questions were then refined through group discussions among the researchers and multiple reviews, leading to the final formulation of prompts. Next, the finalized prompts were used simultaneously to query all three tools. The accuracy and reliability of the responses were assessed by clinicians. Additionally, two health information management specialists reviewed the sources referenced by these tools. Results: According to clinicians, the responses provided by these tools were somewhat reliable. Furthermore, a comparison of the answers revealed a relative similarity among these tools, with no significant contradictions observed. However, an analysis of the references showed that while ChatGPT-۴ and GROK primarily relied on English-language sources—mainly websites of international organizations—Microsoft Copilot used unreliable Persian-language websites for ۷۰% of the questions. Additionally, the evaluation of references indicated that these tools rarely referenced evidence-based information. Conclusion: This study provides further insights into the role of conversational
AI models as supplementary counseling tools for addressing common questions about breast cancer. The findings suggest that while these tools can be useful for initial education, they are not entirely reliable, raising concerns about the completeness, accuracy, and timeliness of their information. Therefore, they should be used solely as adjunct resources for preliminary education and counseling. If clinical symptoms are present, seeing a specialist is the best solution.
