Mandip Adhikari
← Index
ProjectAug 2025 — Sep 2025

NEPL-LM

A scaffold for continual preference learning. An inference server samples k candidates per prompt and logs them, objective verifiers score them, and the scores become DPO / RRHF pairs that train LoRA adapters — which are hot-loaded back into the same server.

Python · FastAPI · Uvicorn · PyTorch · Transformers · PEFT / LoRA · Pydantic · SymPy · Pint

Research scaffold · each stage is a CLI step over JSONL, no scheduler

5
verifiers: math, code, facts, safety, LLM judge
3
preference trainers: DPO, RRHF, ORPO
0
restarts to swap an adapter

The loop

/v1/generate returns k candidates with their average token log-prob and appends each turn to a JSONL log. /v1/adapters/load, select and unload hot-swap PEFT adapters without restarting the server.

An offline evaluator matches each logged turn to a task spec and runs its verifier. A pair builder fuses the judgements, confidence-weighted, and writes preference rows using top-vs-all, adjacent or sampled pairing.

┌─────────────────┐  logged turn ┌──────────────────┐
│ inference server│─────────────▶│ turns.jsonl      │
│ /v1/generate k  │              └────────┬─────────┘
└────────▲────────┘                       ▼
         │                       ┌──────────────────┐
         │                       │ offline eval     │
         │                       │ math code facts  │
         │                       │ safety llm judge │
         │                       └────────┬─────────┘
         │ /v1/adapters/load              ▼ judgements
┌────────┴────────┐              ┌──────────────────┐
│ trainers        │◀─────────────│ pair builder     │
│ dpo rrhf orpo   │  pairs.jsonl │ combine → reward │
└─────────────────┘              └──────────────────┘

Verifiers

The math verifier does symbolic and numeric checks with SymPy, plus unit consistency with Pint. The code verifier runs snippets in a subprocess under CPU, memory and wall-clock limits against unit tests. Facts and safety are deliberately light heuristics behind clean interfaces, so they can be swapped for real ones.

Training and rollout

The DPO loss is implemented directly against a frozen reference model — no TRL — with LoRA on the attention projections. Canary routing uses epsilon-greedy or Thompson-sampling bandits, and promotion gates require +3% win rate, no safety regression, and KL ≤ 0.5.

A nightly cadence with monthly consolidation is sketched in the design notes; that part is a plan, not code.