Make a Time‑Machine, Not a Toy: Building a Vintage 13B

When Swyx described wanting “low‑background tokens” and a “chat tuned 13B vintage model,” he was naming a practical engineering gap, not pitching nostalgia. The problem people call “vintage” — models that reliably speak, reason, and imagine from a past era’s worldview — is primarily data and objective mis‑specification, not a hardware or capacity shortage. Small, hand‑built vintage LLMs are charming proof‑of‑concepts, but they’re fragile: tiny models memorize style and hallucinate facts; bigger models, trained with the right data and loss weighting, actually model distributions. That difference is the difference between a toy timepiece and a functioning time machine. The evidence is already here. Projects like talkie trained a 13B model on approximately 260 billion tokens restricted to pre‑1931 sources and produced a model that consistently errs historically rather than anachronistically. That’s not accidental: a 13B model has the representational headroom to internalize lexical shifts, syntactic preferences, and topical priors that distinguish one century’s prose from another. It also has enough capacity to generalize — to, for example, re‑derive basic algorithmic patterns it never explicitly saw — while remaining small enough to be practical for researchers and hobbyists. In short: scale matters, but smart scaling matters. Why does a purpose‑built 13B base beat the tiny specialist approach? Two ideas from modern training practice explain it. First, scaling laws (Kaplan et al.; Chinchilla/Hoffmann et al.) show predictable returns with more parameters and data when you balance them. If you underdo data quality or quantity, bumping parameters alone underperforms; but a compute‑balanced 13B trained on a carefully constructed historical corpus sits in a practical sweet spot — large enough to model nuance, cheap enough to iterate. Second, you don’t need to brute‑force historical fidelity by exclusion alone. Domain and token reweighting lets you sculpt the loss so that the model treats vintage tokens and sources as primary training signal and modern web noise as downweighted background. That’s the knob Swyx implicitly wanted when he said “low‑background tokens.” The academic toolbox for that knob is mature. DoReMi — Domain Reweighting with Minimax Optimization — trains a small proxy model on source domains, solves a group‑DRO style objective to produce domain weights, and then resamples the big training run with those weights. In experiments DoReMi improved few‑shot downstream accuracy by roughly 6.5 percentage points and reached baseline accuracy with about 2.6× fewer steps. That’s concrete: the right mixture matters a lot. PRESENCE and related self‑influence reweighting papers push the idea to sample‑ or token‑level granularity, learning which examples and even which tokens to emphasize to improve pretraining stability and novelty. And in vision‑language work, “semantic token reweighting” papers show you can alter the importance of individual tokens inside a transformer stack to steer outputs in interpretable ways. Together these form a practical recipe: choose the domains, then learn the weights that make the large model into a faithful period speaker. Concretely, a pragmatic engineering recipe looks like this. Start with a robust 13B architectural baseline — an off‑the‑shelf LLM with well‑behaved attention and layernorm choices — and prepare a vintage corpus with strict temporal filters and high‑quality OCR/transcription. The talkie team found OCR quality and contamination are the hard details: automated OCR can cost you 70% of performance versus careful transcription unless cleaned, and stray later‑era prefaces or catalog metadata can leak modern facts into the model. Run a small proxy (a few hundred million parameters) over your domain partitions and use DoReMi to find a domain mixture that preserves perplexity across domains. Use PRESENCE‑style self‑influence scores or microbatch token importance to upweight tokens that express period‑specific signals and downweight the tiny fraction of “spurious” tokens that drive modern‑style hallucination. After pretraining with an optimized mixture and token weights, finish with a light, period‑aware instruction‑tuning step. Don’t overdo reinforcement training that injects modern conversational norms; instead fine‑tune against curated historical transcripts (etiquette manuals, letters, cookbooks, newspapers) and evaluate with “surprise” metrics: measure how the model ranks post‑cutoff events and how often it uses anachronistic vocabulary. The talkie team ran the New York Times “On This Day” archive through their model and watched surprise scores spike after 1930 — exactly the behavior you want: the model should be confidently wrong in a period‑consistent way, not confidently modern. This approach flips a common assumption. People assume that “vintageness” is a stylistic mask you apply to a modern model or that you must shrink a model to force it into a narrow era. Both are half‑right and half‑dangerous. Tiny models can mimic tone but lack the internal structure to hold period semantics; modern models can be forced into style with prompts but will leak contemporary factual priors. A tuned 13B trained on vintage data with learned domain and token weights is both expressive and controllable: it speaks like 1910 and thinks like a 1910 language model, but it can still follow instructions and generalize in useful ways. There are limits and tradeoffs. Data scarcity in old corpora is real; public‑domain cutoffs and copyright mean some periods are easier than others. OCR and metadata contamination are the engineering weeds that will devour naive runs. And yes, you still need to balance compute with data per Chinchilla guidelines — a 13B model trained on only a few billion vintage tokens will underperform. But those are solvable engineering problems, not conceptual barriers. Swyx’s line about “time travel” was rhetorical, but it points to an elegant practice: build the base for the behavior you want, don’t hack it into existence. We have the papers that tell us how to choose the right mixture and reweighting, we have practical examples like talkie that prove a 13B vintage base is possible, and we have the scaling theory that tells us where to spend compute. If you want a believable, robust, and repeatable time‑machine of a language model, make it a purpose‑built 13B and let the data and the loss do the heavy lifting. You’ll get something that reads like the past and explains why the past sounded the way it did — and that’s the sort of engineering win that actually teaches us about models, history, and how language encodes worldview.