The Hidden Value of Prototyping for Policy Making in Tech and Beyond

Michal Luria is a Research Fellow at the Center for Democracy & Technology.

In 2016, Superflux, a “futures” design studio, collaborated with the United Arab Emirates government to speculate on possible energy strategies through role playing, modeling and prototyping. In what they named the “business as usual” scenario, in which no steps were taken to improve air quality, Superflux translated data about projected air quality in the next couple of decades into lab-based “prototypes.” Politicians, policymakers and energy organizations were then invited to experience and interact withwhat could be future air compositions. As Superflux put it: “One sniff of them made evident what data projections speculated: this was not the future you would want your children to inherit.” And indeed, soon after, the government announced that it would be investing billions in a sustainable energy plan.

Prototyping is a critical component of design practice that we don’t see very often in policymaking. A prototype, in this context, can be a physical expression of an idea, or a rough draft of a possible problem space or solution. It can be a sketch, a physical demonstration, or a digital interface. And prototyping has particular benefits when considering problems at the intersection of technology and policy.

Don’t Skip the Prototype

Human-centered research and “design thinking” methods are gaining traction in the public sector, with government organizations, nonprofits and other groups conducting interviews, workshops and focus groups that aim to understand people and their needs as part of policy-making processes. Human-centered research is democratic by definition. It prioritizes people's needs and values, promotes dialogue as part of decision-making, and brings new and underrepresented voices into the conversation. It also has a common goal with policymaking to shift the current state of being into a preferred state.

While it is quite encouraging to see this kind of human-centered work in the public sector, I believe prototyping is a key aspect of these design methods that should be utilized more often as a research tool in policymaking due to its unique strengths. Prototypes help transform hypothetical thinking into concrete thinking. They allow designers and researchers to work through a problem, “think-by-doing,” and consider questions and solutions that are otherwise difficult to conceive.

Recently, I authored a research report at the Center for Democracy & Technology, “This is Transparency to Me”: User Insights into Recommendation Algorithm Reporting, that exemplifies some of the potential benefits of using prototypes in policymaking. We collaborated with people to identify possible recommendations for algorithm transparency reports on social media platforms, and to learn about what information and interfaces would best inform users about how recommendation algorithms work. In collaboration with Caroline Sinders and Schessa Garbutt, we created design prototypes. These prototypes were not suggested solutions, but rather provocations for conversation, an initial step towards understanding what algorithmic transparency reports should include to best support people’s needs.

As the report illustrates, prototypes can benefit policymaking in (at least) three ways. They can:

• Explore a range of problems and solutions, outside current assumptions;
• Help pinpoint the most important problem(s) to tackle; and
• Identify potential flaws in problem solutions.

Prototypes Explore Solutions Beyond Assumptions

Prototypes allow us to go beyond the expected and obvious, and even to break assumptions through low-cost, low-risk experimentation. In the example of recommendation algorithm transparency, when asked, participants in our research study expected to see a large body of text, with general or vague terms, as a so-called “transparency report.” Instead, we experimented with what a personalized view of information could include, which caught most participants by surprise. Our prototypes suggested that a transparency report could be personalized to the reader and share information that is specifically collected about them.

Prototyping personalization and collecting responses allowed us to test this “high-risk” concept with minimal resources. It is an example of the exploration that can be done through prototyping before committing to a single, perhaps more predictable, direction. We learned that personalization of transparency reports ties information about how an algorithm works to the individual, and thus will likely encourage people to be more interested in reading the report.

In a similar experiment, we created an interactive version of algorithmic transparency. We tested a “plug-and-play” concept, in which users would be able to see the relationship between input and output. In this feature, participants were able to view how specific information about them changed the content that is recommended to them.

In our findings it was evident that the interactive and visual aspects of this concept strongly resonated with users. As a result, we recommended that more resources should be allocated to visualizing selected information, ideally in an interactive way.

Prototypes Frame the Problem and Solution Space

Prototypes are experiential. And to get to them, designers must iteratively work through ideas towards a single framing of the problem or possible solution. They can test whether a particular problem is the right problem to tackle by making it concrete in a prototype. The “designing the right thing” approach can avoid a huge pitfall of “designing the thing right,” a common failure in which an organization puts out resources to design a particular policy, product or service in the best way possible, only to find out that’s not even something people want or that there are other issues that reduce its efficacy.

One of the prototypes in our study focused on communicating how data is transferred between platforms, and included both incoming data from external sources and outgoing data that is collected on social media. In that prototype, people were able to move different data types from one privacy setting to another and to control how their data would be shared (hypothetically).

When starting off, we hadn’t considered data transfers as a critical component of algorithmic transparency. Yet, through conversations with people and iterative prototyping, we’ve identified this as a key issue to explore. Testing the concept provided further insight, such as that being able to see a detailed representation of data types is critical, but that there also need to be higher-level controls in place. In other words, participants indicated that they would not want to move each data type, one by one. In our report we recommend to platforms that there should be a combination of detailed information and centralized control levers. Our findings also make room for additional cycles of prototyping and testing, moving towards the best framing of this problem space to serve people’s needs.

Prototypes Identify Solution Flaws Early On

Prototypes are also specific. They require the designer to turn hypothetical ideas into a physical form, and thus reveal unanswered questions, gaps and other problems a policy or solution may hold. By putting ideas into concrete form and sharing them with people and stakeholders, prototypes can identify flaws early on.

We created a prototype with a section that laid out all the inferences that a social media platform made about people in a “story-like” format. For example, a paragraph about location inferences stated: “you have accessed our website from Italy, the UK and Mexico, but we think you live in the UK, and visit Mexico and Italy frequently for business, based on your job location, events that you attended, friends you engaged with, and posts you read.”

We found that people did not appreciate the way that inferences about them were presented. While the intention and the information itself was notable, the implementation was off-putting. This finding contributed to our recommendation that, when sharing people’s personal information, platforms should strive to be data-oriented and neutral, while letting people themselves interpret it in “human” forms. The prototype, in this case, allowed us to identify some critical issues with what seemed to be a valid solution, to surface some of the pushback early on and to adjust recommendations accordingly. Prototyping and iteration can allow practitioners to improve their solutions towards a better result and more positive engagement.

These were three ways in which prototypes can inform policy-making and projects in the public sector. While more design approaches are applied in the public sector, prototyping is a key aspect of design that is still largely absent, and that should be considered as another tool to strengthen and understand citizen’s needs and perspectives. The process of prototyping and “forcing” ideas into tangible forms necessarily surfaces unanswered questions and potential problems in both the high level framing and in specific implementation. As a result, it can provide a way to test ideas and solutions with low-risk involved, and at an early stage, in tech and beyond.