EmpoorioLM: Models & Specs.

Architecture modules

• 1

  Paged Attention

  KVBlockTable with Copy-on-Write. 16-token blocks. Page reuse in parallel generation. Compatible with beam search and speculative decoding. Memory efficiency comparable to vLLM.

• 2

  Selective SSM — Mamba S6

  MambaBlocks with SelectiveSSM interleaved every N layers (ssm_every_n_layers). D=4 expansion with discrete kernel. Captures long dependencies in O(L) instead of O(L²). Lower quadratic cost than pure attention.

• 3

  Multi-Token Prediction

  MTPModule with MTPHead. Predicts K future tokens in parallel (configurable mtp_num_heads). 1.8–2.4× decoding speedup vs pure autoregression with the same output quality.

• 4

  FP8 Training

  FP8Linear with DelayedScaling and amax history. apply_fp8_to_model replaces Linear layers. Reduces VRAM usage ~50% vs bfloat16. Compatible with FSDP2 + Tensor Parallelism + Pipeline Parallelism.

• 5

  SimPO Alignment

  Simple Preference Optimization without external reward model. Trains from SFT checkpoint with 34+ annotated pairs in simpo_pairs.jsonl. Converges in minutes. Efficient alternative to classic RLHF.

• 6

  zkML — Verifiable Inference

  EmpoorioLM inferences can be cryptographically proven on-chain via EmpoorioChain pallet-zk-verifier. Full model transparency without a central custodian. Auditable AI by default.