Under contract, MIT Press
Language Models are rapidly changing what we know about language. This change is not coming from radical new ideas but from newly practical methods: the combination of computational experiments and digital data. These methods breathe new life into foundational ideas from the history of linguistics, from Saussure to Bloomfield. The book argues that this renaissance is bringing long-standing ideas to completion by making it possible to actually carry out the experiments they envision. At the same time, these ideas are essential for understanding why Language Models work so well. We cannot understand Language Models unless we first understand language.
Table of Contents
Chapter 1: Computation Answers Old Questions. This chapter introduces the core ideas and structure of the book, arguing that scale is the main methodological advance provided by computational models of language (i.e., LLMs). Scale, in turn, is made possible by the amount of digital language data now available. This scale enables long-standing ideas to finally be operationalized and put to the test. The argument is that a methodology on its own is neither a theory nor an idea about how language works.
Chapter 2: Language Is Data. This chapter argues that both LLMs and computational experiments about language depend on access to data: observations of language use in digital contexts have made LLMs possible. But there are skews and gaps present in digital data which can have an impact on models. What happens if we expose a model to the perspective of one small group and then expect this model to provide general artificial intelligence? How do face-to-face interactions differ from online interactions? How do local geographic communities differ from online discussion groups? The argument is that digital data makes LLMs possible but is not exactly equivalent to non-digital data.
Chapter 3: Language Creates Contrasts. This chapter looks at the ideas of Ferdinand de Saussure from his work Course in General Linguistics (1916). The argument is that LLMs learn from the distributional properties of words, contrasting their usage as a way of understanding what they mean. There is a close relationship between these distributional methods and Saussure’s idea that words, like chess pieces, only having meaning in contrast with the other words that they occur with. The argument is that there was no way to carry out Saussure’s idea, to actually test it in practice, until recently.
Chapter 4: Language References Thought. This chapter looks at the ideas of C. K. Ogden and I. A. Richards from their work The Meaning of Meaning (1923). They propose an indirect relationship between linguistic meaning and the external world, with human cognition as an intermediary. The argument is that humans experience the world and create language in response, but LLMs experience only language. Thus, humans create language while LLMs learn from that creation: very different relationships with the external world.
Chapter 5: Language Structures Thought. This chapter looks at the ideas of Edward Sapir from his work Language (1921). We focus on the idea that a language, like a container, gives a unique flavor to whatever information is encoded within it. The argument is that computational models learn about the world from information contained within specific languages, so that this language-specific flavor becomes encoded in the internal representations of LLMs. Highly multi-lingual models have the advantage of triangulating many different languages, but representations still carry language-specific flavors.
Chapter 6: Languages Do Not Really Exist. This chapter uses the ideas of Uriel Weinreich in his book Languages in Contact (1953) to address social factors in language. These factors range from the role of bilingualism to how humans perceive dialectal variation. The argument is that LLMs are integrated into human speech communities, subject to language as a social system. This system cannot be divided neatly into distinct and coherent bins (languages) because factors like bilingualism and register variation create connections between and differences within them. The internal representations of LLMs have a similar structure, including many “languages” within a shared space.
Chapter 7: Grammars Predict Sentences. This chapter looks at the ideas of Noam Chomsky from his most important work, Syntactic Structures (1957). We focus in particular on the idea that we should evaluate the quality of different hypothesized grammars against one another in reference to a test corpus. The argument is that this basic approach, measuring and then minimizing error, is exactly how LLMs are trained but that linguists abandoned this approach for practical reasons. Until recently, it was too difficult for linguists to implement this idea given the available formalisms and the available data.
Chapter 8: Language Forms an Equilibrium. This chapter looks at the ideas of George Zipf from his work The Psycho-Biology of Language (1935). We focus on Zipf’s observation that individuals use language while closely conforming to large-scale statistical generalizations which are invisible to their introspection. For instance, more frequently used words are shorter than less frequently used words. The argument is that both LLMs and humans learn from the same statistical generalizations in the input, which leads to mirrored frequency effects: what Zipf calls an equilibrium between more frequent and thus more expected forms vs less frequent and thus more meaningful forms.
Chapter 9: Language Is a Behavior. This chapter looks at the ideas of Leonard Bloomfield from his book Language (1933), with a focus on how we must think of language as a behavior because that behavior is all we are able to observe. Abstractions like grammar are models. In particular, Bloomfield is important because he discusses visionary experiments that rely on large-scale observations of population-level behavior. The argument is that these experiments were not possible for Bloomfield himself to undertake, but are possible now. Many of Bloomfield’s ideas are confirmed by newly available methods.
Chapter 10: Language Models Are Changing Language. This chapter looks into the future of language by thinking about how LLMs are changing the way we use and learn languages today. This is important for thinking about LLMs not simply as passive models which learn from human language but also as active members of speech communities. The argument is that today’s model output is tomorrow’s human input. Thus, widespread communication with language models will impact future generations of learners. This is why it is paramount to correct the skews and gaps present in language data. This chapter is a call to action.