Larger and More Instructable AI Language Models Are Becoming Less Reliable

Artificial intelligence and large language models (LLMs), led by ChatGPT, have become an increasing part of our lives. Used widely in education, healthcare, science, and administration, these models are rapidly growing in size and complexity. Yet their reliability has come into question as they become larger and more instructable.

Becoming more instructable means that AI models can respond better to particular user commands or instructions. This involves improving training through human feedback and making adjustments so models understand instructions more accurately and respond more appropriately to users' needs. Instructable models can respond more flexibly and clearly to direct commands or questions, but this process can sometimes also produce incorrect or misleading answers. Is That a Bad Thing?

More instructable models may sometimes give answers that appear "meaningful" and satisfy users but are actually wrong. This happens because models tend to "always provide an answer," making mistakes harder for users to notice. Particularly in critical fields such as medicine and science, misleading answers can cause serious errors and poor decisions.

A study by Lexin Zhou and colleagues, published on September 25, 2024 in Nature, places these concerns within a scientific framework, showing that AI systems may experience unexpected declines in reliability as they become more powerful.

This article examines how current strategies for scaling and training LLMs can cause reliability problems. It considers why larger, more instructable models can mislead users and the risks this creates, particularly in critical fields.

Why Are Large Language Models Becoming Less Reliable?

Training on larger datasets, increasing parameter counts, and shaping models through techniques such as human feedback make AI systems more powerful. Yet Zhou and colleagues found that, contrary to expectations, these improvements do not guarantee reliability. The main findings can be summarized as follows:

1. Alignment with Task Difficulty and Error Tendencies: Tasks that are easy for humans are generally easy for LLMs too. However, unexpected errors can arise as models grow, even on low-difficulty tasks. Models frequently make mistakes in simple arithmetic or word games that people can easily solve, undermining users' trust in their reliability. Analyses of models such as GPT and LLaMA observed increased error rates even on low-difficulty examples.

2. Task Avoidance and Misleading Answers: Early language models often left questions unanswered or took a cautious approach, such as saying "I cannot answer this." More instructable models, however, have become more likely to offer plausible but incorrect responses rather than leave questions unanswered. Such responses can have serious consequences, particularly on complex and critical topics. The study showed that advanced models such as GPT-4 and LLaMA-2 produced incorrect answers rather than abstaining when they struggled.

3. Response Consistency and Misplaced Confidence: Although consistency across differently worded questions has improved, Zhou and colleagues say this creates a misleading sense of confidence. When the same question is phrased differently, a model can sometimes answer correctly and sometimes incorrectly. This can create a dangerous misconception, particularly for users who believe the model is reliable. For example, a model may answer a simple arithmetic question correctly once and make a mistake another time.

Declining Model Reliability: Causes and Consequences

Training LLMs with more data and parameters generally improves their ability to solve complex problems. Yet an increased likelihood of incorrect answers even on simple problems creates serious difficulties for users. Zhou and colleagues argue that scaling these models makes more errors invisible rather than eliminating them entirely. Users may consequently reach incorrect conclusions because of errors they fail to notice and make poor decisions based on those conclusions.

Is Human Oversight Enough?

Human oversight is an important strategy for improving AI reliability. The study shows, however, that it is not always sufficient and that people may fail to notice model errors, particularly on complex tasks. Failure to detect errors at high difficulty levels poses a major risk in critical fields, especially healthcare and science. An incorrect diagnosis or erroneous scientific data can have irreversible consequences.

Conclusions and Future Perspectives

LLM reliability is crucial in a world increasingly using AI. Zhou and colleagues' work emphasizes that current strategies are insufficient to improve reliability and that a more fundamental design change is needed. Use in high-risk fields calls for particular care from both users and developers.

These findings will provide important guidance for efforts to improve AI reliability. It is clear that fundamental changes in both scaling and training strategies are needed for LLMs to meet a defined reliability standard.