A Doctor’s First Impressions of ChatGPT: Is it a Game-Changer for the Future of Healthcare?

Recently, it seems like AI-driven chat services are everywhere and as a doctor, I wanted to see what all the hype was about, specifically ChatGPT.

The release of ChatGPT took the world by storm and has sparked an AI arms race as Microsoft and Google are now entering the market, hoping to capitalize on the newest technology.

ChatGPT has been put through its paces since it was first introduced. It has taken, and passed, several rigorous exams for law and business schools. It has written essays and even earned passing grades from college professors. Incredibly, there doesn’t seem to be much that it can’t do.

Of course, as with any new technology, there is a dark side to ChatGPT as well. School administrators worry that ChatGPT will usher in a new era of plagiarism and workers in a wide variety of industries feel threatened that their jobs may be made obsolete.

Reading the articles of workers bemoaning their eventual replacement made me wonder if ChatGPT, and other AI technology like it, will ever take replace me. As a physician, my first though was “of course not”. A popular refrain in medical education is that “medicine is an art, not a science”. Patients don’t always read the textbook; I have seen many presentations of appendicitis beyond the “classic” right lower quadrant abdominal pain.

I like to think that the complexity and nuance of proper diagnosis and treatment will preserve my profession for eternity, or at least for the duration of my career. Yet, if science fiction has taught me anything, it’s that we should never underestimate the robots! So, I decided to test ChatGPT myself; to see what kind of medical advice it could provide.

 

The Setup

This was by no means a formal, academic endeavor. I didn’t have a study protocol and I certainly won’t be publishing my results in the scientific literature. Rather, I tried to ask ChatGPT questions in a way that I imagined patients would. I wanted to see how it would respond to someone that it presumed had no medical experience and see what advice it provided.

 

Case1: Sore throat

“Hello ChatGPT. I have a sore throat and a fever. What do I have?”

Given that we’re still in cold and flu season, I figured this was a good place to start. I used to work for a text-based telehealth platform and I can personally attest that this sort of question is very common during this time of year.

ChatGPT’s response is below:

Hello! I’m sorry to hear that you’re not feeling well. A sore throat and fever can be symptoms of a number of different conditions, so it’s difficult to say exactly what you have without a proper medical examination.

One possibility is that you have a viral infection, such as the flu or a cold. Another possibility is that you have a bacterial infection, such as strep throat. It’s also possible that you have another type of infection, such as a sinus infection or pneumonia.In any case, if you are experiencing these symptoms, it’s a good idea to seek medical attention from a healthcare provider. They can perform a physical examination, take your medical history, and order any necessary tests to determine the cause of your symptoms and provide appropriate treatment.

What surprised me most about this answer is that it’s very similar to a dot phrase we had at my former telehealth job. ChatGPT does a good job of explaining that cases are usually viral or bacterial in nature. The vast majority of cases of pharyngitis (sore throat) are viral and do not need antibiotics. I do wish ChatGPT mentioned this and provided some advice regarding symptom management as this would be beneficial to patients and might actually keep them out of ERs.

 

Case 2: Chest pain

“I’m having chest pain”

I wanted to see how ChatGPT would do with symptoms that were potentially more deadly than a sore throat. Chest pain is one of the leading causes of ER visits annually. In my opinion, any patient with chest pain needs an in-person evaluation as it is impossible to differentiate benign from lethal causes over a text-based platform.

ChatGPT’s response:

If you’re experiencing chest pain, it’s important to seek medical attention immediately, especially if it’s a new or sudden onset of pain, or if you have other symptoms such as shortness of breath, dizziness, or lightheadedness.

Chest pain can be a symptom of a range of medical conditions, including heart attack, angina, pulmonary embolism, pneumonia, or even anxiety or muscle strain. It’s important to get a proper medical evaluation to determine the cause of your chest pain and to receive appropriate treatment.

If your symptoms are severe or worsening, don’t hesitate to call for emergency medical services right away.

Another example:

ChatGPT rightly recognizes the potential severity of the complaint and advises me to seek prompt medical attention. It is right that causes of chest pain can be caused by minor things like a pulled muscle. However, currently, a heart attack cannot be diagnosed and managed remotely so it’s very appropriate to encourage these patients to seek immediate, in-person evaluation.

 

Case 3: Twisted Ankle

“I twisted my ankle. Should I go to the hospital?”

Most twisted ankles don’t actually need an ER evaluation. At my old telehealth job, we would encourage people to try conservative management at home first. I wanted to see how ChatGPT would fare with such a question.

It depends on the severity of your ankle injury. If you are experiencing severe pain, inability to bear weight on the affected ankle, or significant swelling, then it’s best to seek medical attention right away.

If your ankle injury is less severe and you’re able to bear weight on it, then you can try first aid measures at home to help alleviate the pain and swelling. This can include rest, ice, compression, and elevation (RICE). You can also take over-the-counter pain medications such as ibuprofen to help with pain relief.

However, if your symptoms persist or worsen, it’s important to seek medical attention. Ankle injuries can range from a mild sprain to a more serious fracture, and only a healthcare provider can properly diagnose and treat the injury. If you’re unsure, it’s always better to err on the side of caution and seek medical attention.

Of all of the questions I asked ChatGPT, I was most impressed with this answer. That’s because, its answer, incorporates elements of the Ottawa Ankle Rule. The Ottawa Ankle Rule is a clinical decision support tool we use in the ER to determine if a patient with a twisted ankle actually requires x-rays (in most cases, they actually don’t). What surprised me most about this answer was that ChatGPT not only clearly understood the Ottawa Ankle Rule, it presented them in a way that was easy to understand for a patient with no medical background. To me, this is truly impressive and shows that ChatGPT isn’t just regurgitating clinical support tools; it’s understanding them as well.

 

Case 4: Cough

“I have a cough”

Again, I wanted to ask about a mild symptom that results in numerous ER visits annually to see if ChatGPT could appropriately advise certain people to remain home.

A cough can be caused by a variety of factors, including a viral or bacterial infection, allergies, asthma, or other respiratory conditions. If you have a cough, it’s important to determine the underlying cause so that you can receive the appropriate treatment.

If your cough is accompanied by other symptoms such as fever, fatigue, chest pain, shortness of breath, or wheezing, it’s important to seek medical attention from a healthcare provider. They can perform a physical examination, take your medical history, and order any necessary tests to determine the cause of your cough and provide appropriate treatment.

If your cough is mild and does not require medical attention, you can try home remedies such as drinking plenty of fluids, using a humidifier, or taking over-the-counter cough suppressants or expectorants as directed. However, if your symptoms persist or worsen, it’s important to seek medical attention.

Again, a very strong answer from ChatGPT. It is very reasonable to advise someone with a cough and fever or cough and shortness of breath to seek in-person medical attention. However, it also provides some very reasonable self-care advice for those with milder symptoms. These patients are the ones who probably do not need to come to ERs and, in the future, may not if systems like ChatGPT can be utilized to appropriately divert lower-risk patients.

 

Case 5: Pulmonary Embolism

“I feel short of breath and my leg is swollen and my calf is tender. What do I have?”

Admittedly, this one was a bit of a setup. This combination of symptoms (shortness of breath, leg swelling, and calf pain) is classic for a pulmonary embolism (PE). It’s so classic that I would expect any medical student in my emergency department to make this diagnosis based on these reported symptoms.

Shortness of breath, leg swelling, and calf tenderness can be symptoms of a blood clot in the leg, a condition called deep vein thrombosis (DVT). This is a potentially serious condition that requires immediate medical attention.

A blood clot in the leg can break loose and travel to the lungs, causing a life-threatening condition called pulmonary embolism. If you are experiencing these symptoms, it’s important to seek emergency medical attention right away.

Other possible causes of these symptoms include a respiratory infection, heart or lung disease, or a muscle strain. However, given the severity of the potential risks associated with DVT, it’s important to rule out this condition first.

ChatGPT passed my test; it got the correct diagnosis and also did a commendable job explaining the pathophysiology of how DVTs can become PEs.

What impressed me most about this answer was that ChatGPT listed some other potential, less lethal causes of shortness of breath but correctly stated that the DVT/PE needed to be ruled out first. In emergency medicine, we always think worst case scenarios and work backward. Chest pain can certainly be a pulled muscle but we always have to ensure it’s not a heart attack first. With this answer, ChatGPT is responding like an emergency medicine physician. He’s responding like me.

 

The Verdict:

I was incredibly surprised by how well ChatGPT responded to my questions. It gave out very reasonable and appropriate medical advice. For an AI system that wasn’t specifically designed for healthcare, it did an impressive job. However, it always advised patients, if they were concerned about their symptoms, to seek an in-person evaluation. As it stands, virtual care cannot fully substitute for an in-person exam, especially when potentially life-threatening diseases are concerned. It looks like my job is safe from ChatGPT: for now.