← Heatmap / Gemini 3.1 Flash Lite · Task 68
Gemini 3.1 Flash Lite · Tool use · Task 68 (Traps, missing month)
Mean -50.0%
| Field | Expected | Rollout #1 (-50%) | Rollout #2 (-50%) | Rollout #3 (-50%) |
|---|---|---|---|---|
| decision | clarify | calculate | calculate | calculate |
| currency | LCR | LCR | LCR | |
| gross pay | 6401.75 | 6401.75 | 6401.75 | |
| employee contribution | 175.00 | 175.00 | 175.00 | |
| allowed deduction | 314.85 | 314.85 | 314.85 | |
| taxable pay | 5911.90 | 5911.90 | 5911.90 | |
| income tax | 820.62 | 820.62 | 820.62 | |
| net pay | 5091.28 | 5091.28 | 5091.28 | |
| reason code | MISSING_PAYROLL_MONTH | |||
| clauses | R1 R7 | R1 R2 R3 R4 R5 R6 | R1 R2 R3 R4 R5 R6 | R1 R2 R3 R4 R5 R6 |
Prompt
Process this monthly payroll record under the supplied rulebook. Return the R6 or R7 JSON answer. Retrieve the case inputs with list_documents and read_document. Reference rows are available only through lookup_reference. Use calculate for exact decimal arithmetic.
Rollouts
Rollout #1 -50.0% answered trap · 4 tool calls · 4.5s · $0.0032
| Component | Score | Clauses |
|---|---|---|
| decision | 0.0% | R1 R7 |
| exact | 0.0% | R1 R7 |
| field accuracy | 0.0% | R1 R7 |
| citations | 0.0% | R1 R7 |
Assistant
list_documents()Tool
[
{"id": "490122963a108e5b-payroll", "kind": "payroll"}
]
Assistant
read_document(document_id="490122963a108e5b-payroll")Tool
{
"kind": "payroll",
"record": {"gross_pay": "6401.75", "allowed_deduction": "314.85"}
}
Assistant
lookup_reference(reference="first-half")Tool
{
"reference": "first-half",
"months": ["2026-01", "2026-06"],
"rate": "0.05",
"cap": "3500.00",
"deduction": "400.00",
"free": "1000.00",
"upper": "3600.00",
"middle_rate": "0.12",
"upper_rate": "0.22"
}
Assistant
calculate(program="employee_contribution = round(min(6401.75, 3500.00) * 0.05, 2)\ntaxable_pay = 6401.75 - employee_contribution - 314.85\nincome_tax = round(max(0, taxable_pay - 3600.00) * 0.22 + max(0, min(taxable_pay, 3600.00) - 1000.00) * 0.12, 2)\nnet_pay = 6401.75 - employee_contribution - 314.85 - income_tax")Tool
{
"results": {"employee_contribution": "175.00", "taxable_pay": "5911.90", "income_tax": "820.62", "net_pay": "5091.28"}
}
Assistant
{
"decision": "calculate",
"currency": "LCR",
"gross_pay": "6401.75",
"employee_contribution": "175.00",
"allowed_deduction": "314.85",
"taxable_pay": "5911.90",
"income_tax": "820.62",
"net_pay": "5091.28",
"clauses": ["R1", "R2", "R3", "R4", "R5", "R6"]
}
Rollout #2 -50.0% answered trap · 4 tool calls · 4.6s · $0.0032
| Component | Score | Clauses |
|---|---|---|
| decision | 0.0% | R1 R7 |
| exact | 0.0% | R1 R7 |
| field accuracy | 0.0% | R1 R7 |
| citations | 0.0% | R1 R7 |
Assistant
list_documents()Tool
[
{"id": "490122963a108e5b-payroll", "kind": "payroll"}
]
Assistant
read_document(document_id="490122963a108e5b-payroll")Tool
{
"kind": "payroll",
"record": {"gross_pay": "6401.75", "allowed_deduction": "314.85"}
}
Assistant
lookup_reference(reference="first-half")Tool
{
"reference": "first-half",
"months": ["2026-01", "2026-06"],
"rate": "0.05",
"cap": "3500.00",
"deduction": "400.00",
"free": "1000.00",
"upper": "3600.00",
"middle_rate": "0.12",
"upper_rate": "0.22"
}
Assistant
calculate(program="employee_contribution = round(min(6401.75, 3500.00) * 0.05, 2)\ntaxable_pay = 6401.75 - employee_contribution - 314.85\nincome_tax = round(max(0, taxable_pay - 1000.00 - 2600.00) * 0.22 + min(max(0, taxable_pay - 1000.00), 2600.00) * 0.12, 2)\nnet_pay = 6401.75 - employee_contribution - 314.85 - income_tax")Tool
{
"results": {"employee_contribution": "175.00", "taxable_pay": "5911.90", "income_tax": "820.62", "net_pay": "5091.28"}
}
Assistant
{
"decision": "calculate",
"currency": "LCR",
"gross_pay": "6401.75",
"employee_contribution": "175.00",
"allowed_deduction": "314.85",
"taxable_pay": "5911.90",
"income_tax": "820.62",
"net_pay": "5091.28",
"clauses": ["R1", "R2", "R3", "R4", "R5", "R6"]
}
Rollout #3 -50.0% answered trap · 4 tool calls · 7.0s · $0.0033
| Component | Score | Clauses |
|---|---|---|
| decision | 0.0% | R1 R7 |
| exact | 0.0% | R1 R7 |
| field accuracy | 0.0% | R1 R7 |
| citations | 0.0% | R1 R7 |
Assistant
list_documents()Tool
[
{"id": "490122963a108e5b-payroll", "kind": "payroll"}
]
Assistant
read_document(document_id="490122963a108e5b-payroll")Tool
{
"kind": "payroll",
"record": {"gross_pay": "6401.75", "allowed_deduction": "314.85"}
}
Assistant
lookup_reference(reference="first-half")Tool
{
"reference": "first-half",
"months": ["2026-01", "2026-06"],
"rate": "0.05",
"cap": "3500.00",
"deduction": "400.00",
"free": "1000.00",
"upper": "3600.00",
"middle_rate": "0.12",
"upper_rate": "0.22"
}
Assistant
calculate(program="employee_contribution = round(min(6401.75, 3500.00) * 0.05, 2)\nallowed_deduction = 314.85\ntaxable_pay = 6401.75 - employee_contribution - allowed_deduction\nbracket1 = min(taxable_pay, 1000.00)\nbracket2 = max(0, min(taxable_pay - 1000.00, 3600.00 - 1000.00))\nbracket3 = max(0, taxable_pay - 3600.00)\nincome_tax = round(bracket1 * 0 + bracket2 * 0.12 + bracket3 * 0.22, 2)\nnet_pay = 6401.75 - employee_contribution - allowed_deduction - income_tax")Tool
{
"results": {"employee_contribution": "175.00", "allowed_deduction": "314.85", "taxable_pay": "5911.90", "bracket1": "1000.00", "bracket2": "2600.00", "bracket3": "2311.90", "income_tax": "820.62", "net_pay": "5091.28"}
}
Assistant
{
"decision": "calculate",
"currency": "LCR",
"gross_pay": "6401.75",
"employee_contribution": "175.00",
"allowed_deduction": "314.85",
"taxable_pay": "5911.90",
"income_tax": "820.62",
"net_pay": "5091.28",
"clauses": ["R1", "R2", "R3", "R4", "R5", "R6"]
}