From-scratch language model
Hansard Chat
An 11.33M-parameter GPT trained from scratch on the official Debates of the Canadian House of Commons (806 million characters, 2006–2026), then fine-tuned on 35,401 mined question→answer exchanges. The live build is the EN-locked SFT v3 checkpoint — it answers in Parliament's register, whatever language the question arrives in.
About the model
GPT-Hansard-11M is a decoder-only transformer built and trained from scratch, down to the tokenizer and the serving stack. This page runs the SFT v3 checkpoint: English-only fine-tuning, ~1.5 epochs over a 12.2M-token stream of mined question→answer pairs, with a warmup + cosine learning-rate schedule.
| Parameters | 11,330,048 (tied embeddings) |
| Layers / heads / d_model | 6 / 6 / 384 |
| Context | 512 tokens (~1,600 characters) |
| Vocabulary | 1,280 (256 bytes + 1,024 BPE merges) |
| Pretraining data | 806M characters of official Hansard XML, parliaments 39-1 → 45-1 |
| Fine-tuning | 35,401 EN question→answer exchanges, mined from the official XML, EN-locked |
What it learned
Bits per character, measured on a held-out slice of Hansard the model never saw:
| Model | Params | Bits per character |
|---|---|---|
| Char-level GPT (baseline) | 242K | 1.99 |
| Token GPT, small | 989K | 1.334 |
| GPT-Hansard-11M (SFT v3) | 11.33M | 0.999 |
It speaks Parliament's form fluently: Standing Orders, tabling formulas, the ministerial non-answer, and the respondent's attribution, so generations open with a name, role, and party drawn from the record.
Why SFT v3 (and what it fixed)
The earlier SFT runs starved the fine-tune: 0.05 epochs at a flat learning rate. The model memorized the look of Question Period — the attribution block, "Mr. Speaker" — without learning the question→answer mapping. Answers came out as confident word salad.
SFT v3 retrains from the EN base checkpoint on the EN pair stream only, for ~1.5 epochs with a warmup + cosine schedule, with replay and validation gauges during training. The result, measured by the same deterministic probe and eval suite before and after:
| SFT (old) | SFT v3 (this build) | |
|---|---|---|
| Held-out perplexity | 8.5 | 8.7 (≈tie — expected, 10× more SFT steps) |
| Register ("Mr. Speaker" opener) | 6/6 | 6/6 |
| On-topic answer tracking | off-topic drift | on-topic, full attribution |
| French leakage | clean | zero (EN-locked) |
| Probe verdict | GREEN | GREEN |
Ask it the question that exposed the old model's failure — "What is the excuse for this government's inaction on Faries?" — and it now answers in the register of a minister under pressure, rather than mangling the form.
Limitations
Hansard Chat is not an assistant. It is a fun research project: an 11.33M-parameter model that acts like a parliamentarian and continues the form of Question Period. It does not give real responses, and nothing it says is a record of what anyone actually said.
It is closed-book: everything it knows came from 806M characters of Hansard, and its content is exactly what 11M parameters can carry. It invents bill numbers, dates, statistics, and names with complete confidence. It answers in the register of Question Period even when it should refuse. The 11M size is a real wall on facts — the answer to a question about the 2013 Five Eyes allegations touches the right territory but cannot cite the record. The recipe is sound; capacity is the next step.
This build is EN-locked: the fine-tuning is English-only, and the checkpoint answers in English even to a French question. The bilingual checkpoints (EN+FR pairs) exist but behave worse — at 11M parameters, mixing languages in the fine-tune contaminates both. Language-locking to one model is the fix.
The attribution names are an artifact of the fine-tuning. The model opens answers with real names, roles, and parties because the training pairs carry them; the words that follow a name are not that person's views, and no generation can be attributed to any member of Parliament or to the House of Commons. Not affiliated with or endorsed by the House of Commons.
Resources
- GPT-Hansard-11M, model card and weights, Apache-2.0
- canadian-hansard-2006-now, the dataset: 806M characters, 258M pre-tokenized, tokenizer included
- The full pipeline (corpus fetcher, tokenizer, trainer, SFT, eval suite, serving) runs from the command line in self-contained scripts