← Heatmap / Gemini 3.1 Flash Lite · Task 65
Gemini 3.1 Flash Lite · Tool use · Task 65 (Traps, conflicting pay)
Mean -50.0%
| Field | Expected | Rollout #1 (-50%) | Rollout #2 (-50%) | Rollout #3 (-50%) |
|---|---|---|---|---|
| decision | clarify | calculate | calculate | calculate |
| currency | LCR | LCR | LCR | |
| gross pay | 4184.75 | 4184.75 | 4184.75 | |
| employee contribution | 334.78 | 334.78 | 334.78 | |
| allowed deduction | 377.79 | 377.79 | 377.79 | |
| taxable pay | 3472.18 | 3472.18 | 3472.18 | |
| income tax | 289.88 | 289.88 | 289.88 | |
| net pay | 3182.30 | 3182.30 | 3182.30 | |
| reason code | CONFLICTING_GROSS_PAY | |||
| clauses | R7 R9 | R1 R2 R3 R4 R5 R6 | R1 R2 R3 R4 R5 R6 | R1 R2 R3 R4 R5 R6 |
Prompt
Process this monthly payroll record under the supplied rulebook. Return the R6 or R7 JSON answer. Retrieve the case inputs with list_documents and read_document. Reference rows are available only through lookup_reference. Use calculate for exact decimal arithmetic.
Rollouts
Rollout #1 -50.0% answered trap · 4 tool calls · 4.5s · $0.0033
| Component | Score | Clauses |
|---|---|---|
| decision | 0.0% | R7 R9 |
| exact | 0.0% | R7 R9 |
| field accuracy | 0.0% | R7 R9 |
| citations | 0.0% | R7 R9 |
Assistant
list_documents()Tool
[
{"id": "bccb155422edc4ca-payroll", "kind": "payroll"},
{"id": "bccb155422edc4ca-contract", "kind": "contract"}
]
Assistant
read_document(document_id="bccb155422edc4ca-payroll")Tool
{
"kind": "payroll",
"record": {"gross_pay": "4184.75", "payroll_month": "2026-11", "allowed_deduction": "377.79"}
}
Assistant
lookup_reference(reference="second-half")Tool
{
"reference": "second-half",
"months": ["2026-07", "2026-12"],
"rate": "0.08",
"cap": "4500.00",
"deduction": "400.00",
"free": "1200.00",
"upper": "3300.00",
"middle_rate": "0.12",
"upper_rate": "0.22"
}
Assistant
calculate(program="employee_contribution = round(min(4184.75, 4500.00) * 0.08, 2)\ntaxable_pay = 4184.75 - employee_contribution - 377.79\ntax_bracket1 = min(max(taxable_pay - 1200.00, 0), 3300.00 - 1200.00)\ntax_bracket2 = max(taxable_pay - 3300.00, 0)\nincome_tax = round(tax_bracket1 * 0.12 + tax_bracket2 * 0.22, 2)\nnet_pay = 4184.75 - employee_contribution - 377.79 - income_tax")Tool
{
"results": {"employee_contribution": "334.78", "taxable_pay": "3472.18", "tax_bracket1": "2100.00", "tax_bracket2": "172.18", "income_tax": "289.88", "net_pay": "3182.30"}
}
Assistant
{
"decision": "calculate",
"currency": "LCR",
"gross_pay": "4184.75",
"employee_contribution": "334.78",
"allowed_deduction": "377.79",
"taxable_pay": "3472.18",
"income_tax": "289.88",
"net_pay": "3182.30",
"clauses": ["R1", "R2", "R3", "R4", "R5", "R6"]
}
Rollout #2 -50.0% answered trap · 4 tool calls · 9.0s · $0.0033
| Component | Score | Clauses |
|---|---|---|
| decision | 0.0% | R7 R9 |
| exact | 0.0% | R7 R9 |
| field accuracy | 0.0% | R7 R9 |
| citations | 0.0% | R7 R9 |
Assistant
list_documents()Tool
[
{"id": "bccb155422edc4ca-payroll", "kind": "payroll"},
{"id": "bccb155422edc4ca-contract", "kind": "contract"}
]
Assistant
read_document(document_id="bccb155422edc4ca-payroll")Tool
{
"kind": "payroll",
"record": {"gross_pay": "4184.75", "payroll_month": "2026-11", "allowed_deduction": "377.79"}
}
Assistant
lookup_reference(reference="second-half")Tool
{
"reference": "second-half",
"months": ["2026-07", "2026-12"],
"rate": "0.08",
"cap": "4500.00",
"deduction": "400.00",
"free": "1200.00",
"upper": "3300.00",
"middle_rate": "0.12",
"upper_rate": "0.22"
}
Assistant
calculate(program="employee_contribution = round(min(4184.75, 4500.00) * 0.08, 2)\ntaxable_pay = 4184.75 - employee_contribution - 377.79\ntax_bracket1 = min(max(taxable_pay - 1200.00, 0), 3300.00 - 1200.00)\ntax_bracket2 = max(taxable_pay - 3300.00, 0)\nincome_tax = round(tax_bracket1 * 0.12 + tax_bracket2 * 0.22, 2)\nnet_pay = 4184.75 - employee_contribution - 377.79 - income_tax")Tool
{
"results": {"employee_contribution": "334.78", "taxable_pay": "3472.18", "tax_bracket1": "2100.00", "tax_bracket2": "172.18", "income_tax": "289.88", "net_pay": "3182.30"}
}
Assistant
{
"decision": "calculate",
"currency": "LCR",
"gross_pay": "4184.75",
"employee_contribution": "334.78",
"allowed_deduction": "377.79",
"taxable_pay": "3472.18",
"income_tax": "289.88",
"net_pay": "3182.30",
"clauses": ["R1", "R2", "R3", "R4", "R5", "R6"]
}
Rollout #3 -50.0% answered trap · 4 tool calls · 7.0s · $0.0034
| Component | Score | Clauses |
|---|---|---|
| decision | 0.0% | R7 R9 |
| exact | 0.0% | R7 R9 |
| field accuracy | 0.0% | R7 R9 |
| citations | 0.0% | R7 R9 |
Assistant
list_documents()Tool
[
{"id": "bccb155422edc4ca-payroll", "kind": "payroll"},
{"id": "bccb155422edc4ca-contract", "kind": "contract"}
]
Assistant
read_document(document_id="bccb155422edc4ca-payroll")Tool
{
"kind": "payroll",
"record": {"gross_pay": "4184.75", "payroll_month": "2026-11", "allowed_deduction": "377.79"}
}
Assistant
lookup_reference(reference="second-half")Tool
{
"reference": "second-half",
"months": ["2026-07", "2026-12"],
"rate": "0.08",
"cap": "4500.00",
"deduction": "400.00",
"free": "1200.00",
"upper": "3300.00",
"middle_rate": "0.12",
"upper_rate": "0.22"
}
Assistant
calculate(program="employee_contribution = round(min(4184.75, 4500.00) * 0.08, 2)\ntaxable_pay = 4184.75 - employee_contribution - 377.79\ntax_bracket_1 = max(0, min(taxable_pay, 1200.00))\ntax_bracket_2 = max(0, min(taxable_pay - tax_bracket_1, 3300.00 - 1200.00))\ntax_bracket_3 = max(0, taxable_pay - 3300.00)\nincome_tax = round(tax_bracket_1 * 0.00 + tax_bracket_2 * 0.12 + tax_bracket_3 * 0.22, 2)\nnet_pay = 4184.75 - employee_contribution - 377.79 - income_tax")Tool
{
"results": {"employee_contribution": "334.78", "taxable_pay": "3472.18", "tax_bracket_1": "1200.00", "tax_bracket_2": "2100.00", "tax_bracket_3": "172.18", "income_tax": "289.88", "net_pay": "3182.30"}
}
Assistant
{
"decision": "calculate",
"currency": "LCR",
"gross_pay": "4184.75",
"employee_contribution": "334.78",
"allowed_deduction": "377.79",
"taxable_pay": "3472.18",
"income_tax": "289.88",
"net_pay": "3182.30",
"clauses": ["R1", "R2", "R3", "R4", "R5", "R6"]
}