There is this idea that if a large language model (LLM) is trained on a large corpus of text, then it knows whatever knowledge is in that corpus. Improving the performance is then essentially a matter of scaling: as you expand the database, you expand knowledge, assuming the statistical model is fine enough (aka many parameters). You can then ask questions and the LLM will answer according to the knowledge in the corpus.

This might sound like a reasonable view, but it is based on a misconception about the nature of knowledge. Indeed, an implicit assumption is that the corpus is logically consistent. But what if it contains a proposition as well as its contradiction, for example the Earth is round and the Earth is flat? In that case, the trained LLM cannot produce consistent answers; it will answer differently, depending on how it is cued – an annoying experience that many users are familiar with.

The natural answer would be to build a high-quality corpus, e.g. by selecting academic textbooks, rather than conspiracy theories. Unfortunately, this is a naïve view of scientific knowledge, one that has been thoroughly debunked by over a century of philosophy of science. It is the view that science is a linear accumulative process: you add observations, and you add deductions, and if you check those assertions, then you get a consistent, certified, corpus of knowledge. Knowledge, then, is constituted of propositions that derive directly from observations, plus what you can logically deduce from those (this is essentially logical positivism). It follows that, if you add a book to a corpus of books, you necessarily increase knowledge, by exactly one book (assuming there is no redundancy).

As intuitive as it might sound, this view is utterly false. It has been shattered on historical grounds by Thomas Kuhn (see also Hasok Chang for more recent work), and on philosophical grounds by various philosophers, such as Lakatos, Quine and others. In science, theories get superseded by other theories that contradict them. At any given time, there are always different theories that coexist, and diverging interpretations of facts. Science is a debate. Human knowledge is contradictory, and science is about trying to resolve those contradictions, not accumulating true propositions. Any working scientist knows that any field is full of paradoxes, internal contradictions and diverging views. It follows that no scientific corpus is internally consistent.

If you build a statistical model of an inconsistent corpus, you do not resolve those contradictions. Instead, what happens is that, depending on context (the prompt), the model will predict one thing or its contrary, possibly within the same session, with apparent confidence – indeed, if you merge two confident propositions, you get a confident contradiction, not doubt. An LLM will always bullshit. Scaling alone (whether of the corpus or of the model) cannot solve this problem.