Summary
empoorio_lm_code_40m_v2_lora_training.pte, the ExecuTorch artefact production served to Ailoos mobile nodes, was byte-for-byte identical to the output of an experimental export script that constructs the model with manual_seed(0) and exports it — without loading any trained weights. Every mobile training session since that artefact was deployed had fine-tuned a LoRA adapter on a randomly initialised model. Loss decreased regardless (LoRA adapts to whatever the base is), artefacts were signed, merged by FedAvg, and recorded as work.
How it happened
No exporter in the repository loaded a checkpoint. All followed the same shape: build the architecture, export. The one path that could have caught it — load_state_dict(strict=False) — does not fail when nothing matches; it returns a list of missing keys nobody read.
Why it was not caught
- Training loss on the random base hovered near 9.0 and did decline within a session; nothing compared it to the ~6.9 a real base produces.
- The validation step checked adapter geometry and signature, not the base the adapter was trained against.
- The rewards pallet measures units of work, not model quality, by design — so payment was consistent with the protocol and inconsistent with reality.
Detection
A byte-level comparison during the mobile-training verification of 8 September 2026: the bytes of the trained 40M weights were absent from the production artefact; the bytes of the seeded random model were present.
Fix
scripts/exportar_pte_desde_checkpoint.py: loads the checkpoint, derives the architecture from the tensors (not from constants), and verifies that the real weights appear inside the written artefact before finishing. The 64M real model was exported, served, and a phone trained on it: loss 6.87 → 6.44 in 30 steps, adapter validated and implemented.
The pattern
The same failure — a constant fixed in code instead of derived from the artefact — appeared in four places in one week: a global tokenizer (3.4 % id overlap between two vocabularies), a hardcoded stop token ('<|' in one vocabulary, 'exp' in another), a hardcoded layer count in the Android trainer ("expected 64 tensors, found 80"), and fixed geometry in the coordinator's adapter contracts. One rule closed all four: geometry and vocabulary travel with the artefact (empoorio_lora.v3 declares them), and the loader checks coherence.
Lessons
- A decreasing loss is not evidence of a real base. Compare against a known-good baseline.
- Verify the artefact's contents, not the export's exit code.
- Rewards for work are only as honest as the definition of work. The pallet is correct; the input to it was not. Provenance anchoring (which checkpoint, which hash) is the structural fix, and it is on the list.
Based on the mobile-training verification of 2026-09-08 and the Ailoos inference-server fixes.