Researchers Release 'Talkie': A 13B Language Model Trained Exclusively on Pre-1931 Text

Nick Levine, David Duvenaud, and Alec Radford have unveiled talkie-1930-13b, a 13-billion-parameter language model trained exclusively on 260 billion tokens of English-language text published before 1931. Hosted on GitHub and Hugging Face, the project aims to create contamination-free models for studying how AI systems generalize beyond their training data—a critical concern in AI safety and capabilities research.

The core innovation lies in the model's temporal boundary. By restricting training data to pre-1931 sources (books, newspapers, patents, scientific journals), the team ensures talkie possesses zero knowledge of events, technologies, or cultural shifts occurring after its knowledge cutoff. This allows clean experiments: researchers can test whether the model independently 'rediscovers' historical innovations like the helicopter patent (Sikorsky, 1935) or Turing's 1936 paper on computability, or whether it can learn modern programming paradigms from in-context examples alone.

Early evaluations reveal nuanced insights. On standard language understanding benchmarks, talkie underperforms a modern architectural twin trained on contemporary web data (FineWeb), even after filtering for anachronistic questions. However, the gap narrows significantly when excluding post-1930 knowledge, suggesting the vintage model grasps core linguistic structures comparably well. More intriguingly, in HumanEval coding tests, talkie successfully generated a correct decoder for a rotation cipher given only the encoder as an example—a single-character edit (swapping addition for subtraction) that implies an emergent understanding of inverse functions, despite training on text devoid of digital computers.

The project also highlights acute data quality challenges in historical AI research. Conventional OCR transcription of 19th- and early 20th-century texts yields only 30% of the learning efficiency achievable with human-transcribed sources, as shown in their analysis of The Wonderful Wizard of Oz. Simple regex cleaning improves this to 70%, but the team is developing a vintage-specific OCR system to close the gap. They've similarly built a post-training pipeline using era-appropriate sources like etiquette manuals and letter-writing guides to avoid injecting modern conversational biases.

Looking ahead, the researchers plan to scale talkie significantly—expanding the corpus beyond English, improving leakage detection with advanced anachronism classifiers, and refining post-training collaboration with historians. A preliminary estimate suggests a trillion-token historical corpus could yield capabilities comparable to GPT-3.5. For now, talkie serves as a powerful tool to interrogate fundamental questions: How much of AI's behavior stems from data versus architecture? Can models trained on isolated historical slices develop generalized reasoning? And how does data provenance shape what we perceive as 'intelligence' in language models? The model and its evaluation framework are openly available, inviting further study into the foundations of AI generalization.