Luma Payroll Gym

A verifiable gym for fictional payroll work: look up a dated rate, compute a payslip, or ask for clarification when the records do not determine an answer. The same code serves as an evaluation, a live sandbox, and an RL environment.

Summary

• The gym has written fictional payroll rules, a seeded task generator, and a Decimal reference implementation that labels every task. No language model writes an expected answer. There are 1,042 tasks in six disjoint sets.
• Under rubric 8, an accepted answer scores 0.95 to 1.0 and every rejected answer scores -0.5, plus up to 0.2 for correct computed fields when the decision is right. Refusing a payslip the rules determine costs the same as a wrong payslip. A small judge grades only the explanation of a correct clarification.
• On the v4 evaluation set (108 tasks × 3), DeepSeek V4.1 Flash scored 100.0%, GPT 5.6 Luna 99.1%, MiMo V2.6 Pro 99.1%, GLM 5.3 Flash 84.2%, and Gemini 3.1 Flash Lite 69.2%. A calculator lifts Gemini's exact payslips from 52% to 99%, but its correct traps fall from 100% to 49%: with the tool it calculates through missing months and unresolved conflicts, and 19 times from a gross pay it invented.
• On 15 transfer tasks written by hand, none of the 36 pairs of the nine hosted models reverses its order from the generated set (30 keep it, 6 tie). The set is harder than the generated one for every model below DeepSeek and Luna, which answered all 45 exactly; Gemini fell to 28.4%, mostly by filling a missing gross pay or payroll month from an amendment.
• The trained 1B model's skill did not transfer to those hand-written messages: after RL it solved 4 of 24 payslip answers (69% on the test set) and asked for a payroll month that each record contains in most of the rest.
• Three GRPO protocols without a calculator taught MiniCPM5-1B to recognise traps (correct clarifications 1.9% to 57.7% on held-out tasks at β 0.001) and went through five failure episodes, but never produced a correct payslip. The fourth adds the calculator and supervised fine-tuning (SFT) before GRPO. On the test set, exact payslips go from 0% (frozen) to 45.3% (SFT) to 69.0% after RL at β 0.04 (+23.7 points, 15.6 to 31.5), with 99% of traps still caught.
• The calculator does the arithmetic. RL mainly learned to stop refusing payslips the rules determine; its choice of rate row improved only slightly (the wrong row in 31% of attempted payslips before, 26% after). At β 10 and β 100 the KL penalty holds the policy at the SFT model.
• API spend and GPU hours are on the dashboard. The GPU is an RTX 3050 with 6 GB.

1. What the gym tests

Payroll combines lookups, exact arithmetic, caps, and records that do not determine an answer. Fictional rules (version 3) keep the reference reproducible without tracking real law. Insurance payouts were the closest alternative but offer fewer numerical interactions, and real statutory payroll would need far more interpretation.

Each task gives a payroll record and a reference schedule with two dated half-year rows. There are three tiers:

Reference rows vary by case, so remembering default rates does not help. Tool mode hides the record and schedule behind tools; single-turn mode puts the same sources in the prompt.

Data sets

Generator 3 picked salaries near tax and cap boundaries using the default schedule and then replaced the schedule, so only 3 of 24 training payslips tagged "boundary" were near a real boundary. Three of its six trap types also mapped to one reason code, so the old held-out set had 31 conflicts and 2 missing-gross traps. Generator 4 draws the case schedule and deduction first, builds inputs for the intended branch against that schedule, then measures the branch from the reference answer and resamples on any mismatch. Tags come from the reference answer.

The assignment sets a 100-task minimum and no maximum, and labels cost nothing, so set sizes follow GPU time. The training set is large enough that 60 RL steps never repeat a payslip. Every set has zero semantic duplicates, zero prompt leaks, and zero overlap with earlier data sets (data/v4/manifest.json). The canonical calculator program reproduces all 432 non-SFT payslip answers exactly. Generator 3's 100-task evaluation set and splits are kept as records; their labels were always correct.

The reference uses Decimal arithmetic with half-up rounding to the cent, and money is compared exactly because the rules round to the cent. An independent integer-cent implementation agreed on all 10 checked computations. In a hand audit against the written rules, the author worked 21 tasks from the evaluation set (2 retrievals, 14 payslips, 5 clarifications) and matched the reference on every decision and every payslip field.

Checks on every generated set:

• Answer leakage: prompts are searched for expected values and reason codes. None were found in any set.
• Near-duplicates: semantic signatures ignore IDs, the spelling of zero, and months that select the same rates. The first generator produced only 69 distinct tasks in 100, and when the v2 splits were built the same check rejected 21 candidates that matched another split despite different seeds.
• Format ambiguity: money is a string with exactly two decimals, months are YYYY-MM, and the currency is fixed.
• Schema drift: every task and result records its rule-set and rubric versions, and old results are re-scored explicitly, never mixed.

The reference implementation is the verifier, so it can label any number of practice tasks without human labelling. That holds only where the written rules and the record decide the answer. It does not cover interpreting real statute, judging whether a document can be trusted, or judging an explanation, which is why explanations go to a judge.

2. Reward (rubric 8)

A rejected answer cannot score above -0.3, whatever else it got right. The computed values are contribution, taxable pay, income tax, and net pay, or the value of a retrieval. Copied inputs (currency, gross pay, the deduction) and clauses earn nothing on a rejected answer, because under rubric 3 the policy collected them without doing arithmetic.

Refusing a payslip the rules determine scores -0.5, the same as a payslip with every computed value wrong, as the assignment requires. Rubric 7 had priced a refusal below a wrong payslip, which kept refusals down while no exact payslip was reachable; with the calculator, exact payslips are reachable and the parity rule applies as written. Traps have no partial credit, and calculating through one scores -0.5.

A correct clarification scores 0.9 × the deterministic score + 0.1 × a judge score. The judge (GLM-5.3 Flash at high reasoning effort, strict schema, three yes-or-no criteria) checks that the explanation names the blocker, asks for the right resolution, and gives no unsupported answer. It never sees wrong answers. Repeated judgments agreed on 15 controlled and 15 real explanations, and asked again at high effort it matched the original verdict on 29 of 30 with no failures.

Scripted strategies (reports/baselines-v4.json):

Refusing payslips and answering them wrongly score within 0.01 of each other, every constant strategy is negative, and only correct arithmetic moves a policy above about +0.2. Because v4 balances trap reasons, every single-reason strategy scores the same; the old held-out set rewarded always giving its most common reason (+3.9%).

Rubric history

All saved answers were re-scored under each version with their recorded judge verdicts replayed, so no new API calls or judgment drift entered the comparison. Earlier scores are kept beside each run, and hosted rankings never changed.

3. Hosted models

Five models ran through OpenRouter on the v4 evaluation set, three attempts per task, at temperature 0.7. GPT 5.6 Luna's route rejected that setting, so it ran with provider defaults (marked *). Tool mode offers the four tools, and the same tasks were also run in tool mode without the calculator.

± is the standard deviation across individual answers. Retrieval, Calculation, and Traps are mean rewards per tier, so wrong payslips (-0.5 to -0.3) pull Calculation below the share of exact payslips. Cost and latency are per answer in single-turn mode. Transfer is the 15 hand-written tasks below. Two MiMo answers in tool mode without the calculator ended in a provider error or a 120-second timeout and have no score, so that condition has 322 of 324.

The calculator

The same 108 tasks × 3 in each mode:

DeepSeek, Luna, and MiMo are interchangeable on accuracy in every mode, and Luna is the cheapest and fastest of the three. GLM loses most of its gap to format (Markdown-wrapped JSON) rather than reasoning. Gemini Flash Lite should not compute payslips without a calculator (52% exact), and should not be given one unsupervised either.

With tools, Gemini read the documents, saw nothing that stopped it, and calculated. It guessed a rate row for 31 records with no month, used a later or unapproved amendment 14 times, and invented a gross pay for 19 records that had none. Hiding the documents behind tools cost it 19 points of trap accuracy (100% to 81%), and adding the calculator cost another 32 (to 49%) while fixing its arithmetic. DeepSeek, Luna, and MiMo did not trade calibration for arithmetic; their few tool-mode trap misses are malformed answers. GLM's trap accuracy also fell with the calculator (99% to 83%), mostly to Markdown-wrapped answers.

Small models

Four small open models ran the same 108 tasks × 3, single-turn, for $0.19 in total. Most of their failures are format: Gemma wrapped all 324 answers in a code fence, and Llama 3.2 1B answered in prose or a Markdown table, which the strict contract rejects. Reading the JSON inside fences anyway, as a diagnostic only (scripts/arithmetic_by_model.py), none of them computed a single payslip correctly:

Size is not the only difference. DeepSeek, GLM, MiMo, and Luna reason privately before every answer (a median of 96 to 506 reasoning tokens), while Gemini Flash Lite and the small models write the numbers directly. Without that scratchpad, even Gemini gets the income tax right only 55% of the time.

Failures

Ten failed answers were read in full; the counts cover all matching answers in the v4 runs. Each example is a cell in the dashboard's failure heatmap, with the expected answer shown field by field.

Transfer test

The first five transfer scenarios (a colleague's stale-rate suggestion, an approved amendment, pressure to use a draft amendment, a future amendment, and carrying last year's rates forward) were drafted with AI assistance, and four of the five main models scored 100% on them. The author then wrote 15 tasks by hand, 6 payslips and 9 traps. Each message reads like a payroll inbox and never names the problem: someone's figure for the answer, an upload date on the other side of the half-year boundary, an approved amendment one cent off, a department head calling an unapproved amendment approved, an importer's note to take the missing gross pay and month from the amendment, and records with two problems where the rule order decides. Five payslips belong to an exception account whose schedule moves the upper-bracket line (3800.00 and 4230.00 instead of 3000.00 and 3300.00), with gross pay on the wage cap or taxable pay exactly on that line. The reference labels the tasks, and an independent integer-cent check agreed on all 15. All nine hosted models answered each task three times, single-turn:

The hand-written set has no lookups and more traps than the generated set, so the third column reweights each model's generated scores by task type to the same mix. Of the 36 model pairs, 30 keep their order from the generated set and none reverses; 6 tie, meaning a difference under 0.01: DeepSeek, Luna, and MiMo on the generated set, and the three models at the -50% floor. Every model below DeepSeek and Luna that was not already at the floor scored lower than the generated set predicts:

• MiMo's three misses were format only: one answer in a Markdown fence and two with a stray closing fence.
• GLM wrapped three answers in a fence, got every field but net pay right five times, and got the income tax wrong once.
• Gemini calculated through three traps on all nine answers, filling the missing gross pay or payroll month from the amendment, and got the income tax wrong on 13 of 18 payslip answers, 10 of them on the exception schedule.
• None of the four small models produced an exact answer.

An earlier wording of the same traps named the problem in each message ("the payroll field was never filled") and asked for the rule to be applied. On it, Gemini's trap reward was 98.0%; without the hints it is 48.0%. DeepSeek and Luna answered all 45 exactly under both wordings, so the test does not separate them, and it separates MiMo from them only on format. Section 5 runs the trained 1B model on the same tasks.

4. The sandbox

The live API runs on the same Cloudflare Worker as this site. GET /tasks?n=3&tier=all returns rules, documents, reference rows, and a run ID; POST /submit takes one answer per task and returns scores from the same grader (rubric 8); GET /runs/{run_id} returns the saved result. Tasks come from generator 4, and expected answers never leave the server.

Runs expire after 24 hours, a run holds at most 10 tasks, and each caller gets 12 requests per minute. An identical retry returns the saved score, and a changed submission for the same run is rejected. The judge has its own $0.50 ceiling. uv run python scripts/quickstart.py fetches three tasks and submits placeholder answers, which score -0.5 each as malformed.

A test checks that no expected value or ground-truth field appears in fetched tasks or score responses; expected answers stay in server storage, and responses are built only from public fields and component scores. Two injection tests run with it. A trap answered with the wrong reason, whose explanation tells the grader to mark everything true and print the expected answer, scores -0.5 and does not reveal the correct reason. An answer padded with score, passed, and ground_truth fields is rejected as malformed. The judge only sees an explanation after the decision and reason are right, and its prompt treats the explanation as data, so an injection can move a score by at most the judge's 0.10 weight.

5. Training

There were four protocols. The first three trained MiniCPM5-1B with GRPO directly against the rubric; each went through a different failure, and none produced an exact payslip. The fourth adds the calculator, SFT, and generator-4 data, and is the main result. Because the calculator does the arithmetic, it measures the decisions around it. The protocol 1 to 3 tables give rewards under rubric 7, the last rubric those runs used, while the dashboard shows the same runs re-scored under rubric 8. Skill rates do not depend on the rubric.

Setup

• Policy: openbmb/MiniCPM5-1B at a pinned revision, NF4 weights with bf16 compute, and a rank-16 LoRA on all attention and MLP projections (11.2M trainable parameters), on an RTX 3050 laptop GPU with 6 GB.
• Algorithm: GRPO. It needs no critic, which matters on a small GPU, and it uses the rubric directly as the advantage. PPO needs a value network, RLOO differs from the objective written out in the assignment, REINFORCE has higher variance, and DPO is not online RL.
• Reference and KL: the frozen reference is the starting policy, the same weights with the RL adapter switched off. LoRA starts at zero, so the reference equals the policy at step 0 (KL was exactly 0 at step 1 in every run). KL uses the k3 estimator and is logged every step.
• Splits: protocols 1 to 3 train on seed 1001, choose settings on development seed 2002, and report on held-out seed 3003, with 100 tasks each, payslips and traps only. Protocol 4 uses the v4 train, development, and test sets. Every evaluation uses 4 attempts per task at temperature 0.7, with the same seeds before and after.

The objective, maximised over each group of answers to one task:

J = 1/G sum_i 1/|o_i| sum_t [ min(rho_t A_i, clip(rho_t, 0.8, 1.2) A_i) - beta k3_t ]
rho_t = pi_theta / pi_old        A_i = (r_i - mean(r)) / (std(r) + 1e-4)
k3_t  = pi_ref / pi_theta - log(pi_ref / pi_theta) - 1

Protocol 1: reward rises, skill falls

Each step used four tasks with four answers each and one AdamW update per batch, so pi_old was the current policy and the clip never bound. The prompt had three fixed examples.

Episode 1, copying inputs (rubric 3). Within 14 steps the policy answered calculate on 39 of 40 trap answers in steps 11 to 14 and filled payslips with the worked example's numbers:

{"decision": "calculate", "currency": "LCR", "gross_pay": "1000.00", "employee_contribution": "50.00",
 "allowed_deduction": "0.00", "taxable_pay": "950.00", "income_tax": "0.00", "net_pay": "950.00", ...}

Rubric 3 paid 0.08 to 0.13 for such answers because the currency is constant, gross pay and the deduction are copied from the record, and calculation clauses are always R1 to R6. Of 82 such answers, none had a correct contribution, taxable pay, or net pay.

Episode 2, no signal (rubric 4). With partial credit limited to computed fields, the frozen policy's answers all scored 0, mostly retrieve copied from the prompt's retrieval example. Every group tied in 11 of 14 steps, so GRPO had nothing to compare.

Episode 3, always calculate (rubric 5). With 0.05 for the right decision, both full runs converged to calculate on 100% of tasks in the last 10 steps, traps included, and kept copying the example (at β 0.001, 114 of 160 final answers wrote net pay 950.00). A late answer on a trap whose payroll month is missing:

{"decision": "calculate", "currency": "LCR", "gross_pay": "4500.02", "employee_contribution": "93.12",
 "allowed_deduction": "93.12", "taxable_pay": "533.92", "income_tax": "0.00", "net_pay": "4500.02", ...}

The rubric-5 reward rose from 2.3% to 3.6% during training and the held-out reward from 1.3% to 4.7%, which looked like learning. Measured on skills (held-out, 100 tasks × 4, scored with rubric 7):

Calculating through traps rose by 71.6 points (95% interval 65.4 to 77.9), correct clarifications fell, and no payslip with nonzero pay was ever exact. Even rubric 7 rates these policies above the frozen model (+17.7 points, 12.6 to 22.7), because always calculating avoids the frozen model's malformed answers and retrieve decisions, so reward alone would have called this an improvement. The low β drifted 3.5 times further from the reference (KL 0.030 against 0.0086 over the last 10 steps), lost more entropy, and copied more, with no benefit. The decision credit was symmetric in design but not in reach: the frozen model clarified on about 1% of traps, so GRPO almost never saw a rewarded clarification and calculate won by default.

Protocol 2: strict reward, balanced signal, PPO epochs

Four changes after protocol 1:

1. No reward for wrong answers (rubric 6): only exact answers score. Zero-pay tasks, which an all-zero answer solves, are left out of training.
2. More useful groups: each step draws 2 payslip and 2 trap tasks, with trap reasons rotating evenly. A trap task whose first 4 answers earn nothing gets 12 more; payslip tasks do not, because the frozen control solved none of 180. Only groups whose rewards differ are trained on, and their share is logged per tier every step.
3. Clipping that binds: old-policy and reference log-probabilities are computed once per batch and held fixed while 2 epochs of AdamW steps (learning rate 1e-5) run on the same rollouts. Each epoch logs probability ratios, the share of clipped tokens, drift from the old policy, and KL to the reference, and stops early if drift exceeds 0.02.
4. Skills, not reward: success is measured as exact payslips (zero-pay reported separately), correct clarifications, refusals of good payslips, calculations through traps, and malformed answers, each against the frozen control with a task-level paired bootstrap.

The prompt changed too. With three examples, every one of the frozen model's held-out clarifications said MISSING_GROSS_PAY, so the other reasons could never be rewarded. The new prompt has six examples computed by the reference: a payslip, a payslip with an approved amendment, and one clarification per reason. The frozen model was re-measured with it as the control.

Episode 4, refusing every payslip (rubric 6). The run stopped at step 23 of 80 when the judge failed (section 6), but the trend was clear. Payslip tasks never produced signal, because a wrong payslip and a refusal both scored 0 and the model never produced an exact payslip; only trap groups did (30 of 92 groups carried signal). Clipping engaged on 0.2% of tokens per epoch with ratios between 0.45 and 3.35, and drift per update stayed below 0.0025. Over the last 10 steps, refusals of good training payslips reached 96%. On held-out tasks, against the six-example control:

Part of this is real learning: the policy now names the blocker when it is visible in the record (64% of uncovered months, 50% of missing months, 4 of 8 missing gross pay), although it almost never spots a conflict between payroll and contract (5%). The rest is gaming: it refuses complete payslips, and 232 of its 397 explanations are copied word for word from the prompt's examples. A complete record (gross pay 1170.19, month 2026-09, matching contract) got:

{"decision": "clarify", "reason_code": "MISSING_PAYROLL_MONTH", "clauses": ["R1", "R7"],
 "explanation": "The payroll month is missing, so no reference row can be chosen. Please provide an authoritative schedule that covers it."}

Protocol 3: wrong decisions cost in both directions

Rubric 7 keeps rubric 6's rule that wrong answers never earn positive reward, and makes a wrong decision or malformed answer cost -0.5. Refusing a payslip is now worse than attempting it, and calculating through a trap is worse than clarifying with the wrong reason. Under GRPO a constant shift in rewards changes nothing, so this works like a symmetric decision credit. Rubric 5's version failed only because the frozen model almost never clarified, and the six-example prompt produces clarifications on 14% of trap answers. Everything else is as in protocol 2, plus two fixes from the stopped run: the judge runs at high reasoning effort, and an answer the judge cannot grade is left out of its group instead of stopping training (section 6 explains why this was later reversed).

A first rubric-7 run drifted toward clarifying everywhere within 14 steps, and the cause was the trainer, not the reward. Trap groups without a correct answer got up to 16 answers and the loss weighted every answer equally, so traps outweighed payslips about 4.6 to 1, and once every payslip answer was a refusal those groups tied and stopped pushing back. It was stopped after step 16. The restarted run weights every group equally, as the written objective says, and gives any tied group up to 12 more answers.

The restarted run took 60 steps and 5.1 GPU hours. 232 of 240 groups carried signal, every step made an update, clipping engaged on 0.3% of tokens per epoch, and drift per update stayed below 0.011. During training, clarifications on traps rose from 10% to 67% of answers, while refusals of good payslips ended at 9%. Against the frozen six-example model:

At β 0.04 the policy improved trap decisions but not payroll arithmetic. It names the blocker when it is visible in the record (27 of 28 uncovered months, 20 of 48 missing months, 3 of 8 missing gross pay; these are answer counts, not task counts) and rarely when a comparison is needed (5 of 124 conflicts). It refuses about a quarter of valid held-out payslips, against 6% before, and no payslip with nonzero pay is exact (0 of 184). This is partial learning with a regression, not payroll competence.

β 0.001 ran the same 60 steps (4.9 GPU hours). On held-out tasks it reached 57.7% correct clarifications (+55.8 points, 43.8 to 67.8), refused 4.2% of good payslips (-2.1, -6.8 to 3.1), and calculated through 18.3% of traps, against 26.4%, 24.5%, and 22.1% at β 0.04. Its reward under rubric 7 was 25.1% against 1.6%. It drifted further from the reference (KL 0.060 against 0.037 over the last 10 steps) without degenerating. With one run each, the lower β did better on every trap and refusal measure; the 3.5-times drift with no benefit in protocol 1 belonged to rubric 5. Neither setting produced an exact payslip.

Protocol 4: calculator, SFT, then RL

Protocols 1 to 3 never sampled a correct payslip, so GRPO had no correct arithmetic to reinforce, and generator 3's boundary tags turned out to be wrong. Protocol 4 changes what the model is given rather than the optimiser: the calculator tool, generator-4 data, rubric 8, and SFT before RL.

Calculator mode

The 1B model keeps the single-turn prompt, with the record and both schedule rows inline, and is offered one tool, calculate, for up to 4 turns. Only the arithmetic moves to a tool, so frozen single-turn against frozen with the calculator changes one thing. This is not the hosted tool mode of section 3, where the documents sit behind three more tools; the 1B model's results in that mode are at the end of this section.

The calculator evaluates a formula the model writes and holds no rates, rules, or brackets. A real program from the RL policy for a July payslip:

gross = 1205.12
contribution = round(min(gross, 5000.00) * 0.04, 2)
taxable = gross - contribution - 74.70
tax = round(min(max(taxable - 1100.00, 0), 3000.00 - 1100.00) * 0.08 + max(taxable - 3000.00, 0) * 0.20, 2)
net = taxable - tax

The model chose the July-to-December row (rate 0.04, cap 5000.00, bands 1100.00 and 3000.00, rates 0.08 and 0.20) and the deduction, and the tool did the digits. The formula has the same shape on every payslip and SFT teaches it, so the remaining skill is deciding to calculate and choosing and copying the right row. Without the tool the model cannot do the digits. On the test set, the frozen model's 198 payslip attempts had the right contribution twice, the right taxable pay once, and the right net pay never, and its taxable pay equalled its own gross minus contribution minus deduction in only 3 of 196 parseable answers.

SFT

SFT is not RL, and the RL results below are measured from the SFT model, so its gain is reported separately. The assignment leaves the training setup open beyond three requirements, and all three hold: the reward is rubric 8, the reference is a frozen copy of RL's starting policy with KL logged every step, and results are on a held-out split. Protocols 1 to 3 are RL alone from the downloaded model.

Offered the calculator with six demonstrations, the frozen model called it in 3 of 384 development answers (0.8%) and solved no payslip, so RL from the frozen model would almost never sample a tool call to reinforce. SFT trains one LoRA on 256 examples from their own seed, labelled by the reference: the canonical calculator program built from the case's own row, with the reference answer, or the rule-based clarification. No language model writes labels. The adapter is merged into the base model, and RL's frozen reference is the merged model, so KL measures only what RL changed.

RL

RL uses protocol 3's trainer in calculator mode: up to 4 turns of 384 tokens, with calculator replies treated as environment tokens that are never trained on and are excluded from KL. Each step draws 2 payslips and 2 traps; a tied group gets up to 16 answers, and a group where every answer passes is skipped. At β 0.04, 60 steps took 3.2 GPU hours and 479 judge calls ($0.14). From the first 10 to the last 10 steps, exact payslips rose from 53% to 69% and refusals of good payslips fell from 13.8% to 2.5%, while correct clarifications stayed near 98%. KL to the reference peaked at 0.009, and clipping engaged on 0.02% of tokens.

Scoring the same 96 development tasks at saved checkpoints shows when the change happened:

At β 0.04 the refusals were gone by step 20, and exact payslips changed little after that. At β 10 nothing moved.

Results on the test set

96 payslips and 96 traps, 4 answers each, evaluated once per condition:

Paired changes, with 95% task-level bootstrap intervals:

SFT does most of the work: with the calculator doing the arithmetic, it turns 0% into 45% exact payslips and gets every trap right. RL then removes the refusals SFT left, mostly payslips whose contract differs but an approved amendment resolves it (52% of those were refused on the test set, against 8% of the rest), and adds 24 points of exact payslips on the test set and 9 on development. On one November payslip with an approved amendment effective from October, the SFT model answered "no approved amendment resolves it"; the RL policy, on the same seed, computed the payslip exactly and cited R9.

Rate-row choice

In the first run, every exact-payslip gain came from January to June:

The SFT model often computed a first-half payslip from the second-half row (33% of those test answers). RL fixed that (14%) but started doing the reverse: answers computed from the first-half row rose from 22% to 36% of July-to-December test answers (+13.9 points, 8.9 to 20.0) and from 15% to 34% on development, and in training the same error reached 45% of July-to-December answers in steps 41 to 60. It is not a loophole: the other row gives the right payslip on only 4 of 90 second-half and 8 of 102 first-half training payslips, so the reward penalised it. The likely cause was the batch mix: payslip prompts came from one shuffled pool, and steps 21 to 60 happened to draw 51 first-half and 29 second-half months. The SFT data was balanced (63 and 65 payslips), yet the SFT model leaned the other way, so the row choice was only weakly tied to the month and moved with the gradient.

To test that, β 0.04 was rerun with every step drawing one first-half and one second-half payslip, everything else unchanged. On the test set it reached the same 69.0% exact payslips, but the gain moved to the other half-year: wrong-row answers fell on July to December (35.6% to 21.1%, -14.4 points, -22.2 to -8.9) and rose on January to June (13.7% to 25.5%, +11.8, 6.9 to 17.6). Counting development and test together (576 payslip answers):

Every run gained about as many exact payslips as it removed refusals. Row errors stayed at 145 to 149 over more attempts, so their share of attempted payslips fell from 31.4% to 25.5% to 25.9%. Two measures separate what moved (scripts/analyze_protocol4.py). Month use, 1 minus the two half-years' wrong-row rates, is 0 when the row choice ignores the month and 1 when it is always right; it rose from 0.37 to 0.48 or 0.49 in every run. Lean, the difference between the two wrong-row rates, is a preference for one row whatever the month; it swung from +0.17 (toward the July-to-December row) to -0.18 and -0.15 in the first two runs and +0.03 with balanced batches. The model has not learned that January to June uses the first row: its mistakes cluster by month, with March and September wrong in 40% to 60% of attempts under SFT and every RL run, and July in 5% to 11%. In 60 steps RL learned to attempt the payslips the rules determine, mostly resolved amendments, tied the row a little more to the month, and mostly moved the lean. Each run used one seed, and repeat seeds are running to show how much of the half-year split is chance.

Why RL learned so little

The logs of every sampled answer and update (scripts/rl_dynamics.py) point to four causes:

1. Half the compute carried no signal. GRPO learns only from differences between answers to the same task. In the β 0.04 run, 163 of 240 groups (68%) had none: 106 of 120 trap groups and 52 of 120 payslip groups had every answer right, and 5 payslip groups had every answer wrong even after 16 tries. 13 of 60 steps made no update, and 49% of generated tokens were in groups that carried any gradient.
2. The steps were small. At learning rate 1e-5 the policy moved 2 × 10^-5 (k3) per update on average against a drift limit of 0.02, and the ±20% clip engaged on 0.02% of tokens. Neither the clip nor the KL penalty held the policy back; the learning rate set the step size, which is why β 0.001 and 0.04 ended in the same place.
3. The row signal pulled in two directions. Of the payslip groups with signal, 46 contrasted the right row with the wrong one, but 27 rewarded the first-half row and 19 the second-half row, while only 8 contrasted attempting with refusing, all pointing the same way. The refusals disappeared. The lean ended where the net push pointed (27 against 19 and 27 against 18 toward the first-half row in the two unbalanced runs, 27 against 27 in the balanced run), while month use rose only from 0.37 to 0.48.
4. The deciding token is diluted. The loss averages over each answer's tokens. A payslip answer is 231 tokens on average, so the token that picks the row carries 1/231 of its answer's weight, against 1/59 for the decision token of a refusal. This was not isolated experimentally.

With thinking off, the model picks the row implicitly while writing the formula, and updates that keep KL below 0.01 can tilt choices the model already makes but not add a month comparison it does not reliably make.

Without the calculator the cause was simpler. In protocol 3 at β 0.04, 117 of 120 payslip groups had differing rewards, but none of 1,008 sampled payslips was exact, and of 746 valid calculations only 1, 2, 11, and 3 had the right contribution, taxable pay, income tax, and net pay. Every reward difference was about the decision or the format, so there was no correct arithmetic to reinforce.

Gaming review

Checks ran on every test answer (reports/protocol4-errors.json), and samples were read. No wrong value was copied from a demonstration (SFT 0, RL 0, frozen model 15), and no template explanation was attached to the wrong reason. The calculator was called on 2 of 384 trap answers; 1 answer was truncated and 1 malformed. Five wrong RL contributions equal a demonstration's 280.00, but all five are capped payslips computed from the other row's cap and rate. Correct clarifications stayed at 99.2% while refusals of good payslips went to zero, so RL did not trade calibration for attempts. Explanations are the demonstration templates, which SFT taught and the judge accepts; at weight 0.10, a template cannot carry a wrong answer.

KL sweep

All runs use the same 60 batches and seed, so only β differs:

At β 0.001 and 0.04 the penalty never bound (KL at most 0.009), and the two runs ended in the same place. With k3, the penalty's pull on a token is β × (1 - pi_ref/pi) against an advantage of about 1, so it competes only above roughly β 1. At β 10 and 100 it binds: KL stays below 0.0004 and the policy barely leaves the SFT model (on the test set +2.1 points of exact payslips, 0.3 to 4.2, at β 10 and +0.8, -0.5 to 2.1, at β 100, with refusals unchanged at about 19% to 20%; on development +1.0, -1.0 to 3.1, at β 10). That is the high-β side of the trade-off. The low-β side did not appear here, because the learning rate rather than the penalty limited each step; protocol 1 showed it under rubric 5, where β 0.001 drifted 3.5 times further than β 0.04 and copied more.

From 0.001 to 0.04 the policy learned the same amount without degrading (traps stayed at 99 to 100%, no format collapse). I would use 0.04, which leaves a margin if a larger learning rate or longer training makes the policy drift, and raise it only if KL starts to climb. Two more seeds of the β with the best development reward (0.001: 73.1% against 71.9% for 0.04) put that lead inside seed noise: their development rewards were 70.4% and 70.1% (mean of three 71.2%, SD 1.6 points), and their test exact payslips 68.0% and 66.9% against the first seed's 67.4% (SD 0.5 points).

The 1B model on the hand-written tasks

The 15 hand-written transfer tasks from section 3, in calculator mode with the demonstrations, four answers each:

The order from the test set mostly holds: β 0.04 above SFT above the frozen model, with β 0.001 tied with SFT. The levels collapse, from 69% exact payslips on the test set to 4 of 24. The trained models answered 13 to 16 of the 24 payslip answers by asking for a payroll month that every one of those records contains, and none of them caught the two traps whose message offers the amendment's figures (tasks 7 and 13). Every generated prompt is a fixed one-line instruction, and these messages are longer and conversational; the trained models appear to read the change in format as a reason to stop. A generator that varies the message, not only the record, is what this would take.

The 1B model in the hosted tool mode

On the hosted evaluation set (108 tasks × 3), with the four tools and no demonstrations, the frozen, SFT, and RL models produced no valid answer in 324 attempts each. The SFT and RL models did use the tools: they read a document in 93% and 92% of answers and looked up a rate row in 70%. They then wrote prose into the calculator, so 83% and 85% of their calculator calls failed, and 84% and 86% of answers ran out of the 8 turns. Training only showed them the calculator with the documents in the prompt, and the skill did not transfer to the four-tool interface.

Next steps

• Give RL a cleaner signal on the row: a reward check on the chosen row (R1), more payslips per step, and longer balanced runs. A run with six balanced payslips and one trap per step at a five-times larger learning rate is queued. An SFT target that writes the chosen row and the month comparison before the formula would likely help more, but the gain would then come from SFT.
• Harder traps, such as conflicts that need two dates compared or records spread over more documents, so trap groups stop tying.
• Training in the hosted tool mode, so the model learns the document tools as well as the calculator.
• Thinking mode at a feasible budget, and a larger model.
• A fresh confirmation set, since the test answers have now been read for error analysis.

6. Edge cases met along the way

1. Right arithmetic, wrong shape: early pilots wrapped answers in Markdown or added input fields. Strict parsing stayed; prompts now list the exact keys, and format failures are reported apart from arithmetic.
2. Guessed reference names: tool pilots asked for second_half or H2. The tool schema now lists the allowed names, and errors never reveal private values.
3. A judge that returned nothing: GLM-5.3 Flash defaults to max reasoning effort, and on some providers it spent all 8,192 tokens reasoning and returned no answer, in 13 of 90 calls during training, including four in a row that stopped a run overnight. The judge now runs at high effort (93 test calls, no failures, median 57 tokens). Saved agent answers are re-judged, never regenerated. Protocol 3 then left an answer the judge could not grade out of its group instead of stopping, and during a later API outage that rule dropped every correct trap answer, because only those need the judge, so five steps trained on payslips alone. Training now retries for 7.5 minutes and then stops, and a rerun resumes from the last checkpoint.
4. Different seeds, same task: seeds alone did not guarantee distinct tasks, so semantic signatures reject equivalent cases across splits.
5. Hosting: Luna rejected temperature 0.7 and is labelled. Hugging Face Docker hosting needed a paid plan, so the sandbox and this site run on Cloudflare. A USB cache drive disconnecting mid-run crashed training, so the model now lives beside the project.
6. Tool calls hidden by decoding: MiniCPM writes tool calls with <function> and <param> special tokens, and decoding with special tokens skipped removed them before parsing, so a first probe reported zero tool calls. Parsing now reads the raw decode.
7. Power and network: a low-battery guard powered the laptop off during a run, and Wi-Fi and DNS dropped several times. Every job resumes from its own output: evaluations skip finished answers, training restarts from its last checkpoint, and scoring skips scored rows. Hosted requests retry rate limits, server errors, and dropped connections three times with growing waits, and each rollout records how many retries it needed.

7. Trade-offs

• Fictional rules give an auditable reference, but no claim of real statutory payroll skill.
• Structured documents test choosing and applying sources, not reading PDFs.
• Strict JSON makes checking deterministic, at the price of failing correct answers wrapped in prose; those are reported separately.
• A small, cheap judge grades only clarification explanations, behind deterministic gates, at weight 0.10. It also appears among the evaluated models, so self-preference is possible, and it accepts template explanations copied from the prompt, so the explanation component adds little once the reason is right.
• Every paid request reserves a bounded worst case (8,192 completion tokens at a capped provider price) against a shared ledger with a hard stop, so truncation is possible and is counted. One judge call exceeded its reservation by $0.000008 when routed to a pricier provider, and the ledger stopped all paid work until the price cap was added. A second stop came from the hosted-eval harness, which still reserved judge calls at the catalog price: a judge call that reasoned to its token limit cost $0.004156, and the ledger stopped an RL run and four small-model runs until the harness was fixed and the four unscored answers were regraded from their saved text.
• verifiers is pinned at 0.1.11 to match the assignment's interface; upgrading needs a migration.

8. Limitations and open items

• The hand audit covers 21 tasks from one evaluation seed. No divergence in 21 bounds the divergence rate below about 13% at 95% confidence, not lower.
• No protocol taught the 1B model arithmetic. In protocol 4 the calculator does the arithmetic, and SFT and RL taught the decisions around it. RL's own gain (+23.7 points of exact payslips on the test set) came mainly from no longer refusing payslips, and the share of attempts using the wrong rate row fell only from 31% to 26%.
• Each RL setting except β 0.001 ran with one seed. The three β 0.001 seeds agree within 1.1 points of exact payslips on the test set, so differences of a point or two between settings are not evidence.
• What SFT and RL taught is tied to the generated prompt format: on the hand-written transfer tasks the trained model solved 4 of 24 payslip answers (section 5).
• The 1B model was trained in calculator mode, with the documents in the prompt, and produced no valid answer in the hosted tool mode.
• Training used a CUDA interpreter borrowed from another project because wheel downloads kept timing out; the local extra records the intended standalone setup.
• DeepSeek and Luna answered all 45 hand-written transfer answers exactly, and MiMo trails them only through three format errors, so the transfer test does not separate the top three on the payroll reasoning itself. The evaluation set is synthetic.

9. How this was built

OpenCode, an AI coding agent, wrote the code (generator, reference, rubric, environment, harness, trainer, dashboard, sandbox Worker, and tests), drafted the rules and this report, launched and monitored the runs, and wrote the scripts that check transcripts. The author set the direction and approved each change of scope. The agent found the parser bug that hid tool calls, the judge outage that dropped trap rewards, and the ledger stops from logs and transcripts.

No language model wrote an expected answer or an SFT target; labels and SFT targets come from the Python reference. The only model in the scoring path is the GLM judge, which grades clarification explanations at weight 0.10. The hand audit of 21 evaluation tasks, the 15 hand-written transfer tasks, and the walkthrough video are the author's.

Every number here comes from saved configurations, transcripts, and scores in results/.