AI thoughts for this week: Our teams sat in on a live demo of an AI tool designed to help caseworkers conduct SNAP interviews.

The tool is being pitched to states across the country, and it’s framed as innovation.

Here’s what happened: close-ended questions that were worse than what experienced caseworkers already do. QC error codes displayed to interviewers who have no use for them. Less context surfaced, not more.

I’m not sharing this to attack a prototype with all the best intentions. I’m sharing it because it’s the most important question in government AI right now — and almost nobody is asking it out loud:

Who decides what “working” means?

This week, I am excited to hear that $10M in new philanthropic funding entered the AI-for-benefits space. Federal agencies now report 3,600+ AI use cases — up 105% in a single year. States are making eight-figure AI bets under fiscal pressure, often without independent evaluation. The pace is real. The urgency is real.

But a tool that moves faster through an interview isn’t better if it misses the detail that keeps a family enrolled. Efficiency alone is not progress.

Since COVID, we’ve spent years inside the systems where these tools get deployed — sitting with caseworkers, mapping error patterns, watching what actually happens when technology meets human need. This week we also published our findings on the most common missteps on the road to lower SNAP payment error rates. Not principles. Findings. From real programs.

That’s the standard AI for benefits should be held to.

Not - does this tool use AI?

But - does this tool help the person on the other side of the desk — and the person across from them?

The frame for AI in government is being set right now. The organizations that need to shape it aren’t the ones with the boldest vision statements. They’re the ones willing to sit in the room, watch the demo, and say what they actually saw.

That’s what will get us to a better system.