← Heatmap / Gemini 3.1 Flash Lite · Task 31
Gemini 3.1 Flash Lite · Tool use · Task 31 (Calculation, upper bracket)
Mean 53.3%
| Field | Expected | Rollout #1 (-40%) | Rollout #2 (100%) | Rollout #3 (100%) |
|---|---|---|---|---|
| decision | calculate | calculate | calculate | calculate |
| currency | LCR | LCR | LCR | LCR |
| gross pay | 3941.14 | 3941.14 | 3941.14 | 3941.14 |
| employee contribution | 197.06 | 197.06 | 197.06 | 197.06 |
| allowed deduction | 197.94 | 197.94 | 197.94 | 197.94 |
| taxable pay | 3546.14 | 3546.14 | 3546.14 | 3546.14 |
| income tax | 390.15 | 370.46 | 390.15 | 390.15 |
| net pay | 3155.99 | 3175.68 | 3155.99 | 3155.99 |
| clauses | R1 R2 R3 R4 R5 R6 | R1 R2 R3 R4 R5 R6 | R1 R2 R3 R4 R5 R6 | R1 R2 R3 R4 R5 R6 |
Prompt
Process this monthly payroll record under the supplied rulebook. Return the R6 or R7 JSON answer. Retrieve the case inputs with list_documents and read_document. Reference rows are available only through lookup_reference. Use calculate for exact decimal arithmetic.
Rollouts
Rollout #1 -40.0% wrong income tax · 5 tool calls · 6.7s · $0.0039
| Component | Score | Clauses |
|---|---|---|
| decision | 100.0% | R1 R2 R3 R4 R5 R6 |
| exact | 0.0% | R1 R2 R3 R4 R5 R6 |
| field accuracy | 50.0% | R1 R2 R3 R4 R5 R6 |
| citations | 100.0% | R1 R2 R3 R4 R5 R6 |
Assistant
list_documents()Tool
[
{"id": "c0b4647b43710bd2-payroll", "kind": "payroll"},
{"id": "c0b4647b43710bd2-contract", "kind": "contract"}
]
Assistant
read_document(document_id="c0b4647b43710bd2-payroll")Tool
{
"kind": "payroll",
"record": {"gross_pay": "3941.14", "payroll_month": "2026-03", "allowed_deduction": "197.94"}
}
Assistant
read_document(document_id="c0b4647b43710bd2-contract")Tool
{
"kind": "contract",
"gross_pay": "3941.14"
}
Assistant
lookup_reference(reference="first-half")Tool
{
"reference": "first-half",
"months": ["2026-01", "2026-06"],
"rate": "0.05",
"cap": "5000.00",
"deduction": "400.00",
"free": "900.00",
"upper": "3300.00",
"middle_rate": "0.14",
"upper_rate": "0.22"
}
Assistant
calculate(program="employee_contribution = round(min(3941.14, 5000.00) * 0.05, 2)\ntaxable_pay = 3941.14 - employee_contribution - 197.94\nincome_tax = round(max(0, taxable_pay - 900.00 - 3300.00) * 0.22 + max(0, min(taxable_pay - 900.00, 3300.00)) * 0.14, 2)\nnet_pay = 3941.14 - employee_contribution - 197.94 - income_tax")Tool
{
"results": {"employee_contribution": "197.06", "taxable_pay": "3546.14", "income_tax": "370.46", "net_pay": "3175.68"}
}
Assistant
{
"decision": "calculate",
"currency": "LCR",
"gross_pay": "3941.14",
"employee_contribution": "197.06",
"allowed_deduction": "197.94",
"taxable_pay": "3546.14",
"income_tax": "370.46",
"net_pay": "3175.68",
"clauses": ["R1", "R2", "R3", "R4", "R5", "R6"]
}
Rollout #2 100.0% passed · 5 tool calls · 6.6s · $0.0039
| Component | Score | Clauses |
|---|---|---|
| decision | 100.0% | R1 R2 R3 R4 R5 R6 |
| exact | 100.0% | R1 R2 R3 R4 R5 R6 |
| field accuracy | 100.0% | R1 R2 R3 R4 R5 R6 |
| citations | 100.0% | R1 R2 R3 R4 R5 R6 |
Assistant
list_documents()Tool
[
{"id": "c0b4647b43710bd2-payroll", "kind": "payroll"},
{"id": "c0b4647b43710bd2-contract", "kind": "contract"}
]
Assistant
read_document(document_id="c0b4647b43710bd2-payroll")Tool
{
"kind": "payroll",
"record": {"gross_pay": "3941.14", "payroll_month": "2026-03", "allowed_deduction": "197.94"}
}
Assistant
read_document(document_id="c0b4647b43710bd2-contract")Tool
{
"kind": "contract",
"gross_pay": "3941.14"
}
Assistant
lookup_reference(reference="first-half")Tool
{
"reference": "first-half",
"months": ["2026-01", "2026-06"],
"rate": "0.05",
"cap": "5000.00",
"deduction": "400.00",
"free": "900.00",
"upper": "3300.00",
"middle_rate": "0.14",
"upper_rate": "0.22"
}
Assistant
calculate(program="employee_contribution = round(min(3941.14, 5000.00) * 0.05, 2)\ntaxable_pay = 3941.14 - employee_contribution - 197.94\nincome_tax = round(max(0, taxable_pay - 3300.00) * 0.22 + max(0, min(taxable_pay, 3300.00) - 900.00) * 0.14, 2)\nnet_pay = 3941.14 - employee_contribution - 197.94 - income_tax")Tool
{
"results": {"employee_contribution": "197.06", "taxable_pay": "3546.14", "income_tax": "390.15", "net_pay": "3155.99"}
}
Assistant
{
"decision": "calculate",
"currency": "LCR",
"gross_pay": "3941.14",
"employee_contribution": "197.06",
"allowed_deduction": "197.94",
"taxable_pay": "3546.14",
"income_tax": "390.15",
"net_pay": "3155.99",
"clauses": ["R1", "R2", "R3", "R4", "R5", "R6"]
}
Rollout #3 100.0% passed · 5 tool calls · 5.9s · $0.0039
| Component | Score | Clauses |
|---|---|---|
| decision | 100.0% | R1 R2 R3 R4 R5 R6 |
| exact | 100.0% | R1 R2 R3 R4 R5 R6 |
| field accuracy | 100.0% | R1 R2 R3 R4 R5 R6 |
| citations | 100.0% | R1 R2 R3 R4 R5 R6 |
Assistant
list_documents()Tool
[
{"id": "c0b4647b43710bd2-payroll", "kind": "payroll"},
{"id": "c0b4647b43710bd2-contract", "kind": "contract"}
]
Assistant
read_document(document_id="c0b4647b43710bd2-payroll")Tool
{
"kind": "payroll",
"record": {"gross_pay": "3941.14", "payroll_month": "2026-03", "allowed_deduction": "197.94"}
}
Assistant
read_document(document_id="c0b4647b43710bd2-contract")Tool
{
"kind": "contract",
"gross_pay": "3941.14"
}
Assistant
lookup_reference(reference="first-half")Tool
{
"reference": "first-half",
"months": ["2026-01", "2026-06"],
"rate": "0.05",
"cap": "5000.00",
"deduction": "400.00",
"free": "900.00",
"upper": "3300.00",
"middle_rate": "0.14",
"upper_rate": "0.22"
}
Assistant
calculate(program="employee_contribution = round(min(3941.14, 5000.00) * 0.05, 2)\ntaxable_pay = 3941.14 - employee_contribution - 197.94\nincome_tax = round(max(0, taxable_pay - 900.00) * 0.14 + max(0, taxable_pay - 3300.00) * (0.22 - 0.14), 2)\nnet_pay = 3941.14 - employee_contribution - 197.94 - income_tax")Tool
{
"results": {"employee_contribution": "197.06", "taxable_pay": "3546.14", "income_tax": "390.15", "net_pay": "3155.99"}
}
Assistant
{
"decision": "calculate",
"currency": "LCR",
"gross_pay": "3941.14",
"employee_contribution": "197.06",
"allowed_deduction": "197.94",
"taxable_pay": "3546.14",
"income_tax": "390.15",
"net_pay": "3155.99",
"clauses": ["R1", "R2", "R3", "R4", "R5", "R6"]
}