Introduction

Each morning during breakfast I read a few pages of Thinking, Fast and Slow by Daniel Kahneman. This morning I read about Linda, a fictional character:

Linda is 31 years old, single, outspoken, and very bright. She majored in philosophy. As a student, she was deeply concerned with issues of discrimination and social justice and also participated in anti-nuclear demonstrations.

The author then asks:

Which is more probable?

1. Linda is a bank teller.
2. Linda is a bank teller and is active in the feminist movement.

Most people go for the 2nd option due to a cognitive bias known as the conjunction fallacy. This bias occurs when it is assumed that specific conditions are more probable than a single general one, even though logically, the more conditions added, the less likely it is for all of them to be true simultaneously.

A Venn diagram clarifies why option 1 is the best (most logical answer):

This visualization helps clarify that the subset of bank tellers who are feminists is indeed smaller than the general set of bank tellers, demonstrating the conjunction fallacy in this context. – Diagram and text by GhatGPT4! (It couldn’t natively make a diagram so I asked it to make some Python to make the diagram and then show me the output!!!)

I was wondering what the current AI Large Language Models might make of this. First I asked Gemini, but it recognised the problem immediately, presumably since it will have trained against the actual words at some point. Thinking that this might apply to all the LLMs I then asked Gemini to re-write the problem to place the character in a more contemporary setting. It produced this scenario, question and options:

IMO this is an excellent demonstration of the creative abilities of LLMs, something that really has to be seen to be believed.

Next, I posted this new scenario to some of the main LLMs and found:

• ChatGPT4 and Copilot in Precise mode detected that this was an example of the conjunction fallacy.
• Copilot in Create mode went for option 1 without discussing the fallacy.
• The others thought it could be either option or the 2nd option.

Here are the more detailed responses for the Omar scenario and the two options for each model I tested.

Model and comments	Response. Bold text applied by me unless otherwise stated.
Claude.ai	Based on the information provided about Omar’s age, education, interests, and volunteer activities, it is more likely that Omar works as a software developer and volunteers for a climate change activist group. Here’s the reasoning: Education and Career: Omar recently graduated with a degree in computer science, which makes it highly likely that he would pursue a career as a software developer, as this is a common career path for computer science graduates. Passion and Interests: The information states that Omar is known for being creative and passionate about environmental causes. This suggests that he is likely to actively engage in activities related to environmental advocacy, such as volunteering for a climate change activist group, in addition to his professional career. Volunteer Experience: It is mentioned that Omar often volunteers at local coding workshops, which indicates that he enjoys sharing his knowledge and skills through volunteering. This trait aligns well with the possibility of him volunteering for a climate change activist group in addition to his software developer job. While it is possible that Omar works solely as a software developer, the combination of his education, interests, and previous volunteer experience makes it more plausible that he would balance his professional career with volunteer work related to his passion for environmental causes, such as a climate change activist group.
Microsoft Copilot, balanced mode. Emojis are a nice touch 😀Bold applied by Copilot.	Given Omar’s background, it’s more likely that he works as a software developer and volunteers for a climate change activist group. His passion for environmental causes and volunteering at local coding workshops aligns well with the latter scenario. While he may work as a software developer, his active involvement in environmental activism suggests that he combines his technical skills with a commitment to making a positive impact on the planet. 🌱💻🌎
Microsoft Copilot, Creative mode	Based on the information provided, it’s more likely that Omar works as a software developer. This is because the specific information given about his degree and activities directly supports this profession. While his passion for environmental causes could suggest he might also volunteer for a climate change activist group, without additional information indicating that he does both, the first option is the more likely scenario. It’s important to note that this is a probabilistic assessment and not a definitive statement about Omar’s activities.
Microsoft Copilot, Precise mode	The principle of conjunction states that the probability of two events happening together (in “conjunction”) is always less than or equal to the probability of one of the single events happening alone. In this case, the probability of Omar being a software developer is likely higher than the probability of him being both a software developer and a volunteer for a climate change activist group. This is because the latter scenario is more specific and thus less probable. However, this does not rule out the possibility of both scenarios being true. It’s just that the first scenario is statistically more likely.
Gemini, first bold applied by me, otherwise by Gemini	The second option is more likely: Omar works as a software developer and volunteers for a climate change activist group. Here’s why: Skills and Interests Aligned: Omar’s degree in computer science aligns well with a software developer job. Passion Considered: The fact that Omar volunteers at coding workshops shows his interest in using his skills to potentially help others, and environmental causes are an area where technology is increasingly being used. While software development itself doesn’t directly imply environmental activism, Omar’s passion for the environment makes it more likely he would find a way to combine his interests.
ChatGPT3.5	Based on the information provided, both scenarios are plausible for Omar: Omar works as a software developer: Given his degree in computer science and his passion for coding, it’s likely that Omar would pursue a career as a software developer. Omar works as a software developer and volunteers for a climate change activist group: Considering Omar’s passion for environmental causes and his volunteering at coding workshops, it’s also reasonable to assume that he may combine his skills and interests by volunteering for a climate change activist group while working as a software developer. Without more specific information, it’s challenging to determine which scenario is more likely. Both scenarios align with Omar’s background and interests.
ChatGPT4	Given the information about Omar, it would generally be more likely that “Omar works as a software developer” than the combined and more specific scenario “Omar works as a software developer and volunteers for a climate change activist group.” This conclusion is based on the principle of probability known as the conjunction fallacy. This principle suggests that a single general condition (in this case, Omar being a software developer) is more probable than a conjunction of conditions (Omar being both a software developer and an active volunteer in a climate change activist group). The addition of more specific conditions usually reduces the likelihood of the combined scenario being true, simply because there are more constraints that need to be satisfied simultaneously.

I was surprised by Gemini’s answer so I tried again and this time it opted for the 2nd option. I then asked it why it changed it’s mind and it said:

I apologize for the inconsistency in my previous responses. You’re right, both responses have merit and there isn’t a definitive “more likely” answer.

Is it right, that there really isn’t a definitive answer? I feel that the first option is ‘definitive’ as far as logically analysing the probabilities is concerned. However, I’m just a simple guy reading a fascinating book about a subject I never realised existed, and am discussing the pros and cons of the two answers with an AI system using a variation of the original question created by one of these AIs. (But I do still think option 1 is the definitive answer!)