The problem with the original premise.
The repository began as a Telegram “psychologist” bot. It stored sensitive conversations and framed generated responses as something closer to care than the software could justify.
I kept the useful engineering question—how a small text classifier learns and fails—but removed accounts, transcripts, diagnosis and therapeutic claims. The new project makes the model, data split, calibration policy and safety boundary visible.
What a credible experiment needs.
The redesign had to make evaluation leakage and uncertainty harder to hide:
- Related prompts must stay grouped so paraphrases cannot leak across train and test partitions.
- Calibration, policy selection, in-distribution testing and out-of-distribution testing need separate data roles.
- Weak evidence must produce abstention instead of a forced confident label.
- The browser and Rust implementations must verify the same versioned artifact and inference contract.
From demo accuracy to a protocol.
A single train/test score would not answer the important questions. It would not show whether similar prompt families crossed the split, whether thresholds were tuned on final test data or what happens outside the training domain.
The v3 protocol therefore freezes group-aware partitions and separates model fitting, probability calibration, abstention-policy selection and final evaluation. Rust types keep final test sets out of selection APIs.
The open-set pipeline.
Strict TSV validation feeds a deterministic group-aware split. A training-only TF-IDF vocabulary and multinomial logistic regression produce probabilities, temperature scaling calibrates them, and a separately selected policy decides whether the model should abstain.
Decisions that keep the result honest.
The project treats the evaluation protocol as part of the software.
Freeze semantic families
The supervised fixture contains 525 rows in 105 equal prompt families. Explicit group IDs keep related wording together during partitioning.
The fixture is intentionally synthetic and bounded; it supports reproducibility, not claims about real clinical language.
Separate unknowns from final tests
Distinct OOD development and OOD test populations let the abstention policy be selected before its final evaluation.
The protocol needs more fixtures and bookkeeping, but the final measurement is no longer part of tuning.
Explain the actual margin
Predictions expose probabilities, confidence, top-two margin and feature contributions that reconstruct the winning logit difference.
Feature attribution explains this linear model’s calculation; it does not explain human meaning or intent.
Reproduction and release checks.
The model, policy, metrics and split plan live in a SHA-256-linked bundle. The CLI can rebuild the bundle, verify every contract and run bounded batch inference. A declared reporting precision keeps the v3 bundle byte-identical across supported release targets.
Rust and browser code run parity fixtures against the same model. A separate SHA-256-pinned selection report publishes every out-of-fold probability, fold assignment and candidate rank; the browser reconstructs its metrics and fails closed if the bytes or aggregates change.
What the project demonstrates.
ELIZA Lab demonstrates a complete small-model workflow: nested group-aware selection, calibration, open-set policy selection, frozen testing, artifact verification and local inference.
It is not a therapist, crisis detector or production language model. Its value is that a learner can inspect the experiment and reproduce the result instead of trusting a black-box demo.
Evidence ledger.
The checked artifacts expose the selection result, the frozen test and the weak cases together:
- Selection protocol
- 385 train-and-development rows in 77 families run through 11 outer and 5 inner group folds, for 506 fitted models.
- Selection result
- Out-of-fold accuracy is 62.597% and macro-F1 is 62.640%; the family-clustered 95% accuracy interval is 57.143–68.571%.
- Frozen ID test
- 82.857% accuracy and 82.278% macro-F1 on 70 synthetic English rows.
- Open-set result
- The frozen policy covers 62.857% of ID rows and 11.11% of OOD rows; OOD AUROC is 0.80278 and FPR at 95% TPR is 0.7778.
- Known weakness
- The 28-row contrast fixture reaches 42.86% pair accuracy; the project does not hide that failure.