The problem with the original premise.
The repository began as a Telegram “psychologist” bot. It stored sensitive conversations and framed generated responses as something closer to care than the software could justify.
I kept one useful engineering question: how does a small text classifier learn, and where does it fail? I removed accounts, transcripts, diagnosis and therapeutic claims. The new project makes the model, data split, calibration policy and safety boundary visible.
What a credible experiment needs.
The redesign had to make evaluation leakage and uncertainty harder to hide:
- Related prompts must stay grouped so paraphrases cannot leak across train and test partitions.
- Calibration, policy selection, in-distribution testing and out-of-distribution testing need separate data roles.
- Weak evidence must produce abstention instead of a forced confident label.
- The browser and Rust implementations must verify the same versioned artifact and inference contract.
From demo accuracy to a protocol.
A single train/test score would not answer the important questions. It would not show whether similar prompt families crossed the split, whether thresholds were tuned on final test data or what happens outside the training domain.
The v3 protocol therefore freezes group-aware partitions and separates model fitting, probability calibration, abstention-policy selection and final evaluation. Rust types keep final test sets out of selection APIs.
The open-set pipeline.
Strict TSV validation feeds a deterministic group-aware split. A training-only TF-IDF vocabulary and multinomial logistic regression produce probabilities, temperature scaling calibrates them, and a separately selected policy decides whether the model should abstain.
Why the experiment stays small and explicit.
The implementation favours inspectable behaviour and separated data roles over a more impressive-looking model.
Rust implements the training, evaluation and inference pipeline.
- Why
- The project publishes a repeatable executable workflow rather than a sequence of interactive cells. Rust types also encode the distinct data roles so final test fixtures cannot enter selection APIs by accident.
- What I ruled out
- A Python-only notebook would be convenient for exploration, but mutable cell state and loosely connected scripts would make the frozen protocol and release artifact harder to audit as one system.
- What it cost
- Model experimentation requires more explicit code and a smaller library surface, and changes take longer than editing a notebook.
Training uses TF-IDF features and multinomial logistic regression.
- Why
- The bounded synthetic corpus suits a compact linear model whose vocabulary, probabilities, top-two margin and feature contributions can all be inspected and reproduced locally.
- What I ruled out
- A transformer could model richer language, but this corpus does not justify its data and compute demands, and a larger opaque model would distract from the evaluation protocol the lab is meant to expose.
- What it cost
- The classifier has limited semantic reach and remains sensitive to vocabulary and phrasing; the project publishes those weak cases instead of implying broader understanding.
Group-aware nested selection keeps prompt families together.
- Why
- Paraphrases in one semantic family are not independent examples. Grouped outer and inner folds keep related wording on one side of each boundary while separating candidate selection from out-of-fold measurement.
- What I ruled out
- A random row split would be simpler, but near-related formulations could appear in both training and evaluation and make the resulting score easier to overstate.
- What it cost
- The effective sample size is smaller, the protocol fits many more models and the split ledger is more involved to maintain.
Temperature calibration feeds an explicit abstention policy.
- Why
- Weak in-distribution evidence and out-of-distribution inputs should not be forced into a known intent. Separate calibration and OOD-development roles let confidence and margin thresholds be fixed before final testing.
- What I ruled out
- Returning the highest-probability class for every input would produce a simpler demo, but it would present an answer even when the model has little evidence for one.
- What it cost
- Abstention reduces coverage, requires additional fixtures and threshold governance, and does not turn the remaining predictions into a safety guarantee.
Decisions that keep the result honest.
The project treats the evaluation protocol as part of the software.
Bound the claim to the synthetic fixture
The versioned corpus is synthetic and deliberately bounded. Reported behaviour describes that fixture and frozen protocol, not clinical language, broad intent coverage or production readiness.
The trade-offThis limits the conclusions, but it makes the educational experiment reproducible without borrowing credibility from a domain it has not measured.
Publish the selection record
Fold assignments, candidate ranks and out-of-fold probabilities are frozen in a SHA-256-pinned report linked to the released bundle, so the chosen model and policy can be reconstructed from the artifact.
The trade-offThe release carries more governed files and consistency checks, but its summary is independently auditable instead of resting on a final score alone.
Explain the actual margin
Predictions expose probabilities, confidence, top-two margin and feature contributions that reconstruct the winning logit difference.
The trade-offFeature attribution explains this linear model’s calculation; it does not explain human meaning or intent.
Reproduction and release checks.
The model, policy, metrics and split plan live in a SHA-256-linked bundle. The CLI can rebuild the bundle, verify every contract and run bounded batch inference. A declared reporting precision keeps the v3 bundle byte-identical across supported release targets.
Rust and browser code run parity fixtures against the same model. A separate SHA-256-pinned selection report publishes every out-of-fold probability, fold assignment and candidate rank; the browser reconstructs its metrics and fails closed if the bytes or aggregates change.
What the project demonstrates.
ELIZA Lab demonstrates a complete small-model workflow: nested group-aware selection, calibration, open-set policy selection, frozen testing, artifact verification and local inference.
It is not a therapist, crisis detector or production language model. Its value is that a learner can inspect the experiment and reproduce the result instead of trusting a black-box demo.
Evidence ledger.
The checked artifacts expose the selection result, the frozen test and the weak cases together:
- Selection protocol
- 385 train-and-development rows in 77 families run through 11 outer and 5 inner group folds, for 506 fitted models.
- Selection result
- Out-of-fold accuracy is 62.597% and macro-F1 is 62.640%; the family-clustered 95% accuracy interval is 57.143–68.571%.
- Frozen ID test
- 82.857% accuracy and 82.278% macro-F1 on 70 synthetic English rows.
- Open-set result
- The frozen policy covers 62.857% of ID rows and 11.11% of OOD rows; OOD AUROC is 0.80278 and FPR at 95% TPR is 0.7778.
- Known weakness
- The 28-row contrast fixture reaches 42.86% pair accuracy; the project does not hide that failure.
Verified release v1.6.0 Verified on
This case study describes the immutable v1.6.0 release at commit cacd4448. Its signed tag and 20 attested assets cover Linux x64, Windows x64, macOS Intel and Apple Silicon. The synthetic corpus does not establish clinical validity, broad language coverage or production readiness.