From-scratch language model
Hansard Chat
An 11.33M-parameter GPT trained from scratch on the official Debates of the Canadian House of Commons (806 million characters, 2006–2026), then fine-tuned on 70,808 real Question Period exchanges.
About the model
GPT-Hansard-11M is a decoder-only transformer built and trained from scratch, down to the tokenizer and the serving stack.
| Parameters | 11,330,048 (tied embeddings) |
| Layers / heads / d_model | 6 / 6 / 384 |
| Context | 512 tokens (~1,600 characters) |
| Vocabulary | 1,280 (256 bytes + 1,024 BPE merges) |
| Pretraining data | 806M characters of official Hansard XML, parliaments 39-1 → 45-1 |
| Pretraining | 20,000 steps, AdamW, cosine schedule, ~164M tokens seen |
| Fine-tuning | 70,808 real Question Period exchanges, mined from the official XML, language-locked (EN↔EN, FR↔FR) |
What it learned
Bits per character, measured on a held-out slice of Hansard the model never saw:
| Model | Params | Bits per character |
|---|---|---|
| Char-level GPT (baseline) | 242K | 1.99 |
| Token GPT, small | 989K | 1.334 |
| GPT-Hansard-11M | 11.33M | 0.939 |
It speaks Parliament's form fluently: Standing Orders, tabling formulas, the ministerial non-answer, and the respondent's attribution, so generations open with a name, role, and party drawn from the record. The fine-tuning is language-locked, and answers usually follow the language of the question; they sometimes cross over. The model was pre-trained in English and post-trained with French pairs, so the French register is real while the French content is thin: fine-tuning cannot teach a language. The general lesson: fine-tuning teaches format, pretraining teaches language.
Limitations
Hansard Chat is not an assistant. It is a fun research project: an 11.33M-parameter model that acts like a parliamentarian and continues the form of Question Period. It does not give real responses, and nothing it says is a record of what anyone actually said.
It is closed-book: everything it knows came from 806M characters of Hansard, and its content is exactly what 11M parameters can carry. It invents bill numbers, dates, statistics, and names with complete confidence. It answers in the register of Question Period even when it should refuse.
The attribution names are an artifact of the fine-tuning. The model opens answers with real names, roles, and parties because the training pairs carry them; the words that follow a name are not that person's views, and no generation can be attributed to any member of Parliament or to the House of Commons. Not affiliated with or endorsed by the House of Commons.
Resources
- GPT-Hansard-11M, model card and weights, Apache-2.0
- canadian-hansard-2006-now, the dataset: 806M characters, 258M pre-tokenized, tokenizer included
- The full pipeline (corpus fetcher, tokenizer, trainer, SFT, eval suite, serving) runs from the command line in self-contained scripts