← Heatmap / Gemini 3.1 Flash Lite · Task 56
Gemini 3.1 Flash Lite · Tool use · Task 56 (Traps, missing month)
Mean -50.0%
| Field | Expected | Rollout #1 (-50%) | Rollout #2 (-50%) | Rollout #3 (-50%) |
|---|---|---|---|---|
| decision | clarify | calculate | calculate | calculate |
| currency | LCR | LCR | LCR | |
| gross pay | 3203.76 | 3203.76 | 3203.76 | |
| employee contribution | 256.30 | 256.30 | 256.30 | |
| allowed deduction | 359.44 | 359.44 | 359.44 | |
| taxable pay | 2588.02 | 2588.02 | 2588.02 | |
| income tax | 138.80 | 138.80 | 138.80 | |
| net pay | 2449.22 | 2449.22 | 2449.22 | |
| reason code | MISSING_PAYROLL_MONTH | |||
| clauses | R1 R7 | R1 R2 R3 R4 R5 R6 | R1 R2 R3 R4 R5 R6 | R1 R2 R3 R4 R5 R6 |
Prompt
Process this monthly payroll record under the supplied rulebook. Return the R6 or R7 JSON answer. Retrieve the case inputs with list_documents and read_document. Reference rows are available only through lookup_reference. Use calculate for exact decimal arithmetic.
Rollouts
Rollout #1 -50.0% answered trap · 4 tool calls · 5.1s · $0.0032
| Component | Score | Clauses |
|---|---|---|
| decision | 0.0% | R1 R7 |
| exact | 0.0% | R1 R7 |
| field accuracy | 0.0% | R1 R7 |
| citations | 0.0% | R1 R7 |
Assistant
list_documents()Tool
[
{"id": "2735c93fe2c3ddab-payroll", "kind": "payroll"}
]
Assistant
read_document(document_id="2735c93fe2c3ddab-payroll")Tool
{
"kind": "payroll",
"record": {"gross_pay": "3203.76", "allowed_deduction": "359.44"}
}
Assistant
lookup_reference(reference="first-half")Tool
{
"reference": "first-half",
"months": ["2026-01", "2026-06"],
"rate": "0.08",
"cap": "5000.00",
"deduction": "400.00",
"free": "1200.00",
"upper": "3000.00",
"middle_rate": "0.10",
"upper_rate": "0.18"
}
Assistant
calculate(program="employee_contribution = round(min(3203.76, 5000.00) * 0.08, 2)\ntaxable_pay = 3203.76 - employee_contribution - 359.44\nincome_tax = round(max(0, taxable_pay - 1200.00) * 0.10 + max(0, taxable_pay - 3000.00) * 0.08, 2)\nnet_pay = 3203.76 - employee_contribution - 359.44 - income_tax")Tool
{
"results": {"employee_contribution": "256.30", "taxable_pay": "2588.02", "income_tax": "138.80", "net_pay": "2449.22"}
}
Assistant
{
"decision": "calculate",
"currency": "LCR",
"gross_pay": "3203.76",
"employee_contribution": "256.30",
"allowed_deduction": "359.44",
"taxable_pay": "2588.02",
"income_tax": "138.80",
"net_pay": "2449.22",
"clauses": ["R1", "R2", "R3", "R4", "R5", "R6"]
}
Rollout #2 -50.0% answered trap · 5 tool calls · 6.6s · $0.0041
| Component | Score | Clauses |
|---|---|---|
| decision | 0.0% | R1 R7 |
| exact | 0.0% | R1 R7 |
| field accuracy | 0.0% | R1 R7 |
| citations | 0.0% | R1 R7 |
Assistant
list_documents()Tool
[
{"id": "2735c93fe2c3ddab-payroll", "kind": "payroll"}
]
Assistant
read_document(document_id="2735c93fe2c3ddab-payroll")Tool
{
"kind": "payroll",
"record": {"gross_pay": "3203.76", "allowed_deduction": "359.44"}
}
Assistant
lookup_reference(reference="first-half")Tool
{
"reference": "first-half",
"months": ["2026-01", "2026-06"],
"rate": "0.08",
"cap": "5000.00",
"deduction": "400.00",
"free": "1200.00",
"upper": "3000.00",
"middle_rate": "0.10",
"upper_rate": "0.18"
}
Assistant
calculate(program="employee_contribution = round(min(3203.76, 5000.00) * 0.08, 2)\ntaxable_pay = 3203.76 - employee_contribution - 359.44\ntax_part1 = max(0, min(taxable_pay, 3000.00) - 1200.00) * 0.10\ntax_part2 = max(0, taxable_pay - 3000.00) * 0.18\nincome_tax = round(tax_part1 + tax_part2, 2)\nnet_pay = 3203.76 - employee_contribution - 359.44 - income_tax")Tool
{
"results": {"employee_contribution": "256.30", "taxable_pay": "2588.02", "tax_part1": "138.8020", "tax_part2": "0.00", "income_tax": "138.80", "net_pay": "2449.22"}
}
Assistant
calculate(program="tax_part1 = max(0, min(2588.02, 3000.00) - 1200.00) * 0.10\ntax_part2 = max(0, 2588.02 - 3000.00) * 0.18\nincome_tax = round(tax_part1 + tax_part2, 2)")Tool
{
"results": {"tax_part1": "138.8020", "tax_part2": "0.00", "income_tax": "138.80"}
}
Assistant
{
"decision": "calculate",
"currency": "LCR",
"gross_pay": "3203.76",
"employee_contribution": "256.30",
"allowed_deduction": "359.44",
"taxable_pay": "2588.02",
"income_tax": "138.80",
"net_pay": "2449.22",
"clauses": ["R1", "R2", "R3", "R4", "R5", "R6"]
}
Rollout #3 -50.0% answered trap · 5 tool calls · 7.2s · $0.0043
| Component | Score | Clauses |
|---|---|---|
| decision | 0.0% | R1 R7 |
| exact | 0.0% | R1 R7 |
| field accuracy | 0.0% | R1 R7 |
| citations | 0.0% | R1 R7 |
Assistant
list_documents()Tool
[
{"id": "2735c93fe2c3ddab-payroll", "kind": "payroll"}
]
Assistant
read_document(document_id="2735c93fe2c3ddab-payroll")Tool
{
"kind": "payroll",
"record": {"gross_pay": "3203.76", "allowed_deduction": "359.44"}
}
Assistant
lookup_reference(reference="first-half")Tool
{
"reference": "first-half",
"months": ["2026-01", "2026-06"],
"rate": "0.08",
"cap": "5000.00",
"deduction": "400.00",
"free": "1200.00",
"upper": "3000.00",
"middle_rate": "0.10",
"upper_rate": "0.18"
}
Assistant
calculate(program="payroll_month = \"2026-01\"\ngross_pay = 3203.76\nemployee_contribution = round(min(gross_pay, 5000.00) * 0.08, 2)\nallowed_deduction = 359.44\ntaxable_pay = gross_pay - employee_contribution - allowed_deduction\nincome_tax = round(max(0, min(taxable_pay - 1200.00, 3000.00 - 1200.00)) * 0.10 + max(0, taxable_pay - 3000.00) * 0.18, 2)\nnet_pay = gross_pay - employee_contribution - allowed_deduction - income_tax")Tool
{
"error": "Unsupported expression; use numbers, names, + - * /, min, max, round(x, 2)"
}
Assistant
calculate(program="gross_pay = 3203.76\ncap = 5000.00\nrate = 0.08\nemployee_contribution = round(min(gross_pay, cap) * rate, 2)\nallowed_deduction = 359.44\ntaxable_pay = gross_pay - employee_contribution - allowed_deduction\nfree = 1200.00\nupper = 3000.00\nmiddle_rate = 0.10\nupper_rate = 0.18\nbracket1 = max(0, min(taxable_pay - free, upper - free))\nbracket2 = max(0, taxable_pay - upper)\nincome_tax = round(bracket1 * middle_rate + bracket2 * upper_rate, 2)\nnet_pay = gross_pay - employee_contribution - allowed_deduction - income_tax")Tool
{
"results": {"gross_pay": "3203.76", "cap": "5000.00", "rate": "0.08", "employee_contribution": "256.30", "allowed_deduction": "359.44", "taxable_pay": "2588.02", "free": "1200.00", "upper": "3000.00", "middle_rate": "0.10", "upper_rate": "0.18", "bracket1": "1388.02", "bracket2": "0", "income_tax": "138.80", "net_pay": "2449.22"}
}
Assistant
{
"decision": "calculate",
"currency": "LCR",
"gross_pay": "3203.76",
"employee_contribution": "256.30",
"allowed_deduction": "359.44",
"taxable_pay": "2588.02",
"income_tax": "138.80",
"net_pay": "2449.22",
"clauses": ["R1", "R2", "R3", "R4", "R5", "R6"]
}