Where AI Actually Works in Government: Citizen Chatbots, Fraud Detection, and the Due-Process Problem

In 2013, Michigan's Unemployment Insurance Agency turned on a system called MiDAS to automatically flag fraudulent unemployment claims. Over the next two years it accused roughly 40,000 people of fraud, levied penalties of up to four times the disputed amount, and garnished wages and tax refunds without a human ever reviewing most of the flagged cases. Later review found the overwhelming majority of those fraud determinations were wrong. The state eventually settled multiple lawsuits, and the Michigan Court of Claims found the process unconstitutional.

MiDAS wasn't a large language model — it was a rules-based automated decision system, the kind of thing that would today get called "legacy algorithmic government." But it's the right starting point for any honest account of AI in government, because it set the template that current systems are still negotiating: government has a structural reason to reach for automation (huge caseloads, understaffed agencies, backlogs that make headlines), and a structural reason to be terrified of it (automated decisions about benefits, custody, and liberty carry due-process weight that a bad e-commerce recommendation doesn't).

Ten years later, the underlying tension hasn't gone away — it's just running on better models. Government AI today sits on two largely separate tracks that get lumped together in press coverage but behave completely differently in practice: citizen-facing service delivery, and back-office risk scoring. They deserve separate treatment.

Track one: citizen-facing chatbots, where the bar is "boring and correct"

The most visible government AI deployments are conversational interfaces sitting on top of policy documents: GSA's internal generative AI tools for federal employees, the UK's piloted GOV.UK Chat for navigating government services content, Estonia's long-running Bürokratt virtual-assistant initiative, and dozens of state DMV and 311 chatbots that answer questions like "do I need to renew my registration in person" without a hold-music wait.

The technical shape of these systems is unglamorous by design: retrieval-augmented generation over a fixed, agency-approved corpus (statutes, benefit-eligibility rules, form instructions), with the model instructed to answer only from retrieved passages and to cite the source. The interesting engineering problem isn't generation quality — modern models are more than capable of writing a clear paragraph about SNAP eligibility. It's grounding and refusal: making the system say "I don't know, contact your local office" rather than confidently inventing an eligibility threshold, because a wrong answer about a filing deadline has real consequences and, unlike a wrong restaurant recommendation, can't be shrugged off as "just a chatbot."

That's also why procurement matters more here than the model choice does. In the U.S., the General Services Administration's USAi platform — launched to give federal agencies access to commercial models like those from OpenAI, Anthropic, and Google through a shared, FedRAMP-adjacent procurement path — exists precisely because individual agencies can't each run their own model-vetting and security-review process for every LLM vendor. The bottleneck for citizen-facing AI in government isn't capability, it's the compliance pipeline: data residency, records-retention law (a chatbot transcript can be a public record), and Section 508 accessibility requirements that most consumer chat products were never built against.

Track two: eligibility and fraud scoring, where the bar is due process

The second track is less visible and much higher stakes: systems that score people — for benefits fraud, for underwriting, for risk — and route the ones that cross a threshold toward denial, investigation, or enhanced scrutiny.

This is where MiDAS's ghost is still present. State unemployment insurance agencies still run automated fraud-detection layers, because UI fraud is real and material (COVID-era pandemic unemployment programs alone lost tens of billions of dollars to fraud, by multiple federal inspector general estimates), and modern systems are meaningfully better than 2013-era MiDAS at cross-referencing wage records, employer reports, and identity-verification signals. But the due-process lesson from MiDAS shows up directly in current guidance: OMB's Memorandum M-24-10 (2024) classifies benefits-eligibility and fraud-determination systems as "rights-impacting" AI, which triggers specific requirements for federal agencies — a human in the loop before an adverse action, a documented AI impact assessment, an opportunity to contest the determination, and a public entry in the agency's AI use-case inventory. State agencies aren't bound by federal OMB memos, but several states (Illinois, Colorado, and others) have passed their own algorithmic-accountability statutes requiring similar disclosures for consequential automated decisions, largely written in direct response to fraud-detection scandals like Michigan's.

The practical effect is that the most mature fraud-detection deployments today look less like "the algorithm decides" and more like "the algorithm triages, a person decides" — a queue-prioritization tool rather than an adjudicator. That's a real architectural constraint, not just a legal fig leaf: it caps how much labor the system can actually save, because someone still has to review every flagged case before consequences attach.

Predictive policing: the application area that's actually shrinking

If there's one area of government AI that's gone in reverse over the past several years, it's predictive policing. PredPol (later rebranded Geolitica), which generated geographic "hot spot" predictions for patrol deployment, was adopted by dozens of U.S. police departments through the 2010s. The LAPD ended its PredPol contract in 2020 after an internal audit found inconsistent data logging and no clear evidence the program reduced crime. Chicago shut down its "Strategic Subject List" — a person-based risk-scoring tool, not a location-based one — in 2019 after research found it didn't reliably predict who would be involved in violence and disproportionately flagged Black residents. Chicago also ended its ShotSpotter (acoustic gunshot-detection) contract in 2024 following city audits questioning its cost-effectiveness, though ShotSpotter's parent company, now branded SoundThinking, remains under contract in many other U.S. cities.

The underlying accuracy critique traces back to ProPublica's 2016 "Machine Bias" investigation into COMPAS, a recidivism risk-assessment tool used in pretrial and sentencing decisions in several states. ProPublica found the tool's false-positive rate for Black defendants was roughly double that for white defendants, at similar overall accuracy — a textbook illustration that "equally accurate on average" and "equally fair across groups" are different, sometimes incompatible, properties (a tension the algorithmic-fairness literature describes as the impossibility of simultaneously satisfying multiple common fairness metrics). Northpointe (COMPAS's vendor, later Equivant) disputed ProPublica's framing, and the academic debate over which fairness definition should govern pretrial risk tools is still unresolved. But the political result has been consistent: several jurisdictions have scaled back or eliminated location- and person-based predictive policing tools rather than resolve the underlying dispute.

Where the maturity actually sits

The takeaway

The pattern across all four areas is the same: government AI succeeds where it's scoped as an assistant to a human decision-maker and fails — sometimes catastrophically — where it's allowed to make or effectively force a consequential decision alone. That's not a temporary limitation of current models; it's a structural response to what due process actually requires. The near-term roadmap for public-sector AI isn't "make the model more accurate until it can decide alone" — it's building the audit trail, the contestability, and the human review step into the system from day one, because the legal and political environment (OMB M-24-10, state algorithmic-accountability laws, and the next MiDAS-style lawsuit) is going to demand it regardless of how good the underlying model gets.