Humans Must Control Human-Serving AI

Trooper Sanders is CEO of Benefits Data Trust, a data and technology-powered nonprofit solving America’s $80 billion benefits access challenge.

For leaders of human services agencies delivering food, health care, and other critical assistance to people and families in need, the fizzy talk about artificial intelligence (AI) may seem enticing. Dedicated public servants on the front lines of fighting poverty, illness, and exclusion made it through the unprecedented demands of the COVID-19 pandemic only to find themselves adjusting to a new normal-- serving their communities while severely understaffed, overstretched, and reliant upon systems and practices from a bygone era.

The purveyors of AI say their wares will bring an unprecedented boost to productivity, knowledge, and wealth creation. The truth is that AI brings many benefits, but it is wise for governments and non-profits to proceed with caution. The promise of AI is tempered by well-known risks such as exacerbating bias and discrimination, contributing to a toxic workplace, and menacing privacy. These failings should be of particular concern to the leaders of human services agencies.

Indeed, rogue code can upend an agency’s mission to improve and save lives. An ongoing scandal in Australia tells a cautionary tale.

In 2015, the Australian government used AI to turbocharge efforts to track and recover suspected fraudulent social security payments. The cabinet minister in charge of social services at the time said he would be a “strong welfare cop on the beat bringing cheats to heel. The government’s “Robodebt” effort tapped an automated decision-making system, but the system was not fit for this purpose. In an exhaustive report, the Royal Commission that investigated the Robodebt scheme found gross inaccuracies in the system’s automated flags, unreasonable demands placed on the accused and, most tragically, lives lost. “Robodebt was a crude and cruel mechanism, neither fair nor legal, and it made many people feel like they were criminals,” the Commission concluded. “In essence, people were traumatized on the off chance they might owe money. It was a costly failure of public administration, in both human and economic terms.”

There is a growing body of work, including the U.S. federal government’s Blueprint for an AI Bill of Rights and AI Risk Management Framework, as well as a range of tools on trustworthy AI produced by civil society groups that can help human services agencies chart a responsible AI course. But the key lesson from Australia is that leadership matters. Leaders can start by ensuring AI decisions are part of agency-wide strategy and policy instead of being left to data and technology experts or the back office. Then they must set the right tone. Political posturing about being the high sheriff of waste, fraud, and abuse signals to technical and procurement officials to shop for AI products that are little more than algorithmic night sticks. To be sure, program integrity should be a top priority, but true program integrity must both keep the ineligible from drawing down assistance while also ensuring those who are eligible can get help efficiently, with dignity, and free from unwarranted suspicion.

Human services leaders can look to the military for a key principle on managing AI: keeping the human-in-the-loop. Put simply, advances in AI and military capabilities are giving weapons the ability to select and engage targets - i.e., kill – without human intervention. This technical reality can be checked by policies that require humans stay at the controls of lethal force. Whether human control can keep pace with autonomous capabilities is an open question, but establishing the human-in-the-loop principle is a critical, and potentially life-saving, first step.

A human-in-the-loop principle for human services would cover two affected parties: front line workers and the people served by programs. What’s key to remember is that AI does not think. That’s an exclusively human power. Where low-risk, rote tasks consume too much human bandwidth, AI can be an important efficiency tool, and should cautiously be given more room to operate. But when AI affects the rights, safety and well-being of people and society, such as determining eligibility for public benefits or adjudicating disputes, AI should be kept on a short, tightly controlled leash. It can augment human expertise but not replace it. In Australia’s case, before Robodebt, payments flagged as fraudulent were reviewed manually, with compliance officers contacting recipients to see if discrepancies could be resolved using the rules, human judgment, and common sense. Robodebt shoved skilled workers to the sidelines with tragic consequences

When designing AI strategies, human services leaders should bring experts and frontline workers into the mix early, tailoring requirements for AI products to the lived experience of workers, and they should ensure workers remain in the loop throughout the lifecycle of its procurement and use. For example, as part of our work closing the nation’s more than $80 billion benefits access gap, Benefits Data Trust worked closely with financial aid experts, college advisors, and students to guide our use of natural language processing, a branch of AI, to power the chatbot Wyatt that advises rising college students through the federal financial aid application.

The people seeking and receiving services should have a seat at the table as well. This is best done by bringing human-centered design and the voice and interests of the client into the heart of AI planning. Centering the best interests of people is a useful screen for AI use cases and purchasing requirements, as well as evaluating the technology’s impact on an agency’s work.

The Australian review found no evidence that the creators of Robodebt attempted to consider how the vulnerable and disadvantaged would experience a wholly automated process. Human-centered design could have avoided the catastrophe by not only engaging in public consultation but changing the way the agency worked as well. As Civilla, a nonprofit with expertise in human-centered design, advises: “successful human-centered design in public institutions requires us to manage change across complex organizations – planning for ongoing evaluations, developing internal capacity, and guiding culture change at scale.”

Tapping the wisdom of both workers and residents alike can dramatically increase the chances of getting the best out of AI and mitigating the worst. Trustworthy AI can not only introduce much-needed technical muscle to important government programs, thoughtful strategies guiding its use can also drive change that brings more efficiency and dignity to human services and delivers more value to the taxpayer. Making it happen is all a question of leadership.