{"id":2712,"date":"2019-10-13T18:16:25","date_gmt":"2019-10-13T22:16:25","guid":{"rendered":"https:\/\/itp.nyu.edu\/adjacent\/issue-6\/?p=62"},"modified":"2024-10-08T23:06:40","modified_gmt":"2024-10-08T23:06:40","slug":"a-technocratic-delusion-is-eating-our-future","status":"publish","type":"post","link":"https:\/\/itp.nyu.edu\/adjacent\/issue-6\/a-technocratic-delusion-is-eating-our-future\/","title":{"rendered":"A Technocratic Delusion is Eating Our Future"},"content":{"rendered":"\n

In his 2005 book<\/a>, What Do Pictures Want?, <\/em>WJT Mitchell wrote that our time is \u201cbest described as a limbo of continually deferred expectations and anxieties. Everything is about to happen, or perhaps it has already happened without our noticing it.\u201d 
<\/p>\n\n\n\n

Nearly 15 years later, we\u2019re in the midst of assembling<\/a> a surveillance architecture designed to ensure that even the minutiae of our lives won\u2019t be beyond notice. Satellites that can detect<\/a> the books on your coffee table. Doorbells that double <\/a>as neighborhood nanny-cams for the police. Face-recognizing CCTV with an authoritarian back-door<\/a>. Wearables counting our steps and modulating our insurance<\/a> premiums. Real-time location from our mobile phones sold<\/a> to bounty hunters. Speakers recording our most intimate and banal moments for machine \u2014 er, human<\/a> translation.
<\/p>\n\n\n\n

This militaristic vision of safety, in the form of \u201ctotal information awareness\u201d, commits to policing a symptom (human behavior) while underwriting the deeper disease: a narrative<\/a> of dominance, consumption, and growth that directs our institutions, shapes our conceptions of progress, and threatens our future on this planet. 
<\/p>\n\n\n\n

This is not an essay about technology, so much as it is about entropy.
<\/span><\/p><\/blockquote>

\n\n\n\n

The Future is a Business<\/strong><\/h3>\n\n\n\n

For decades we have had an industry, though relatively few realize it exists. It\u2019s a sort of discipline of practice, devoted to tracking and unpacking the present, all under the guise of predicting what happens next. Deployed in the service of operational goals, the project here is not just to anticipate possible futures, but to actively create the <\/strong>future. 
<\/p>\n\n\n\n

This is the futurology<\/em> industry (whose practitioners are often called \u201cfuturists<\/a>,\u201d not to be confused with practitioners of the 1920s Italian art movement<\/a> of the same name). Futurology is a deeply flawed, sometimes ridiculous<\/a>, and for many years, closely guarded set of tools for collaborative sensemaking (\u201ctrend sensing,\u201d \u201cforecasting,\u201d \u201chorizon scanning\u201d) and strategic planning (\u201copportunity discovery,\u201d \u201cscenario planning,\u201d \u201cstrategic roadmapping\u201d). <\/p>\n\n\n\n

Since the industry\u2019s rise in the mid-20th Century, these world-building tools have typically<\/a> been operated by institutions of power as closed processes in contexts dangerously supportive of \u2014even reliant on\u2014the status quo. 
<\/p>\n\n\n\n

As a distributed project largely only accessible to those in positions of power and privilege<\/a>, the central, if not explicit, goal here is to enact one of many possible futures. And to construct, measure, and police conceptual and structural borders and boundaries that ensure the general continuity of business-as-usual. 
<\/p>\n\n\n\n

For the multinational corporations and government agencies with the budget to ingest its $50,000 syndicated research subscriptions, bespoke opportunity discovery workshops, and customized consulting contracts, the futures industry promises early warning on the future. But signposts on the road are not enough. Futurists also sell the strategic foresight to land at the top of any number of possible societal configurations, and the lead time to devise machinations to make them more or less likely to occur. This is full-stack futuring: it is much easier to read the tea leaves if you have been genetically engineering and then trimming them yourself. 
<\/p>\n\n\n\n

Much of this business of tracking and prognosticating appears to center on technology. This focus on the shiny, new, and networked, is not only due to a close industrial allegiance to Silicon Valley (although it is<\/em> futurology\u2019s intellectual and historic home in the United States), but also because as a window into shifting power relations, technology change provides a way to explore the strength and resilience of deep structures: laws, regulations, narratives, and planning processes.
<\/p>\n\n\n\n

The Bureaucratic Imaginary<\/strong>
<\/h3>\n\n\n
\n
\"a<\/figure><\/div>\n\n\n

For decades, dedicated futurists have tracked progress on distributed sensors, batteries, computing speed, processing power, spectrum availability, data volume and veracity, machine learning, and other enabling technologies that promise to embed computational forecasting as a layer of infrastructure across private and public systems, often under the heading of \u201ccyber-physical systems\u201d\u2014long before the \u201crevolution\u201d was heralded and called \u201cbig data,\u201d \u201cAI,\u201d or \u201cautomation.\u201d 
<\/p>\n\n\n\n

And as predicted, this software revolution first \u201cate<\/a>\u201d the private sector and is now rapidly<\/a> colonizing<\/a> the public sector, being implemented at every level of government around the world. 
<\/p>\n\n\n\n

Planning for the future is much simpler if we assume linearity and knowability through quantitative data. However, through the logic of machine learning, we\u2019re conflating pattern-identification and sensemaking.
<\/p>\n\n\n\n

The automation-via-analytics fantasy has given up on human judgment. It implicitly prefers discrimination-at-scale and by-design over the unpredictable dice throw of decentralized human prejudice (and institutional programming). But this preference also extends over human capacity for nuanced reasoning and contextualized<\/a> decision-making. It tinkers, optimizing at the margins, incrementally nudging toward \u201cimproved outcomes\u201d (only ever the ones that are measurable). It bars even the possibility of fundamental reorientation based on revised first principles.
<\/p>\n\n\n\n

What\u2019s more, we need to acknowledge how those marketing fantasies and policy scenarios have limited our imaginations\u2026Billions of dollars later, we\u2019re still pursuing the same old dreams.<\/em><\/span><\/p>– Shannon Mattern<\/a><\/cite><\/blockquote>

\n\n\n\n

As we predict, we plan. And we plan to measure the success of our plans according to the outcomes we plan to achieve. 
<\/p>\n\n\n\n

Most of our metrics are designed to track the trajectory of individuals under conditions of systemic machinations. A process change here, an updated standard there, a technology stack, a new policy, a \u201cdisruptive innovation<\/a>!\u201d To know the effects of any implemented plan, we measure the behavior of individuals, the essential unit of control. 
<\/p>\n\n\n\n

Alas, facts don\u2019t change minds. But data can tell stories<\/em>.
<\/span><\/p> <\/cite><\/blockquote>

\n\n\n\n

Forecasting, planning, measurement, and evaluation operate as an ouroboros: a feedback loop that relies on and produces mounting amounts of individual data, as a sort of toxic<\/a> exhaust. Importantly, this data, whose very design and collection have been shaped by a particular set of values (often: neoliberal efficiency), is framed as objective \u201cevidence\u201d and the best (or only) source of informed decision making. Manipulated through computational recipes called algorithms, data can train systems to both predict and automate the outlines of the future. We must now recognize, in Ruha Benjamin\u2019s words<\/a>, \u201cthe weathermen who make it rain.\u201d
<\/p>\n\n\n\n

This mathematical decoupling of data from the context of the structures, systems, stories, theories, and values that drove its collection and shaped its borders\u2014under the guise of predicting the future\u2014 is \u201cstatistical malpractice<\/a>\u201d, smoke and mirrors. We\u2019re being captured, blindfolded, and taken for a ride by our own deafening meta-narratives<\/a> of who we are and what we think we deserve. 
<\/p>\n\n\n\n

Our future is eating itself. It dreams<\/a> of transcending its body. It dreams of first nuking, and then rocketing to and colonizing Mars<\/a>, leaving its home, the planet, a dried husk or shed cocoon. Our future sees itself as separate from that home. It not only accepts these dreams, it treats them as axioms. It cannot imagine anything else. It plans <\/strong>on them. 
<\/p>\n\n\n\n

A Cyborg Dream of Electric Inequality<\/strong>
<\/h3>\n\n\n
\n
\"\"<\/figure><\/div>\n\n\n

There are many elephants in the room of futures thinking, but none as pressing as the future of forecasting itself. Futurists have long relied on the business of public sector planners. But the logic of targeted advertising has arrived: everything is measurable, knowable, predictable, and plannable, through data. 
<\/p>\n\n\n\n

Legal restrictions intended<\/a> to limit surveillance architectures, by preventing large scale data sharing, are largely surmountable and come with their own externalities. Incentivizing the inclusion of law enforcement data to make use of \u201cpublic safety\u201d exceptions, is one example. Many of the worst<\/a> current examples of automation-via-data-integration sit at the intersection of educating children and calling the police<\/a>.   
<\/p>\n\n\n\n

In this technocratic dreamscape, civil servants have been fr<\/a>e<\/a>ed<\/a> from the drudgery and ambiguous work of making important determinations about individuals and their interactions with public systems. Should she be denied bail? Might that child someday be expelled or arrested? Does he really need a home health aide? Should we take away their daughter?<\/em>
<\/p>\n\n\n\n

This project is fundamentally about telling machines about individuals in order to train those machines to manage people. Over time, so the fantasy goes, requiring less and less human oversight. 
<\/p>\n\n\n\n

The trick<\/a>, of course, is that the line between human and machine is one of those boundaries that blur as you look closer. The training data collected through human logics about humans, must be reviewed, cleaned and labeled by humans (the same reason people are listening to our incantations to Alexa), its relevance for inclusion in integration efforts determined by humans, the algorithms that combine and sort it designed by humans, targets of optimization chosen by humans, informed by human values, defined through human institutions, structured by human power relationships\u2026
<\/p>\n\n\n\n

Machining Morality<\/strong>
<\/h3>\n\n\n\n
\"\"<\/figure>\n\n\n\n

In marked contrast to the marketing hype around the \u201cpower of AI to change the world,\u201d data scientists themselves have long been alarmed<\/a> by what data is being asked to do. Disciplines from computer science to anthropology, STS, and design, are generating useful critiques and interventions.<\/p>\n\n\n\n

But some of these may reinforce<\/a> a problematic<\/a> narrative of inevitable technological progress. Investing in the fairness, accuracy, and transparency of the algorithms, while important, also suggests<\/a> that eventually, the machine may make itself readable to, redressable by, and responsible for humanity.
<\/p>\n\n\n\n

While critically important to generate, critiques of “bias<\/a>” in AI may imply that a technical fix is possible; that the fault exists as something to be measured and corrected for as a sort of mathematical adjustment. Algorithms can be \u201caudited\u201d for fairness. Code can be tweaked to be \u201crace-blind.\u201d <\/p>\n\n\n\n

Likewise, \u201csolving\u201d for privacy via technology ignores (and may propel) personal and systemic harms that are only<\/a> hinted at by the concept of privacy. Exposure. Exploitation. Vulnerability. Loss of control of our own narrative. Being haunted by the worst thing that\u2019s ever happened to us. A buggy misunderstanding that lingers like a bad smell. 
<\/p>\n\n\n\n

Even entirely reasonable arguments positing that technologies such as face recognition, are not \u201cready for primetime\u201d because they have not been trained inclusively on diverse populations and therefore fail to recognize non-white faces, likewise fail to miss the point<\/a>.
<\/p>\n\n\n\n

Inequality is a feature of capitalist institutions, not a bug.
<\/span><\/p><\/blockquote>

\n\n\n\n

And the machine is us. The useful distinction here is not in some fundamental separation between man and machine conferring objectivity and neutrality upon the latter, but in the appearance of difference that permits a recapitulation and reification of capitalist institutional power, under the guise of technological progress.
<\/p>\n\n\n\n

Automated social service delivery systems are fundamentally about, as Virginia Eubanks puts<\/a> it, \u201cmanaging the individual poor in order to escape our collective responsibility to eradicate poverty.\u201d 
<\/p>\n\n\n\n

Before we outsource planning for the future to computational analytics,<\/p>\n\n\n\n

Can they ever be a tool of systemic transformation?<\/em><\/p>\n\n\n\n

Can they see systems, or are they blinded by our myopic measurement of individual outcomes?<\/em><\/p>\n\n\n\n

Can they be directed by, and open to continual feedback from, impacted communities?<\/em>
<\/p>\n\n\n\n

These are questions to contend with before<\/em> embedding predictive analytics as a layer in planning and service delivery, not after the infrastructure is too big to fail. 
<\/p>\n\n\n\n

Can they do moral work?<\/em>
<\/p>\n\n\n\n

The work of improving communities is moral work. Not moral in any religious sense of the word, but in the sense that we must prioritize a different set of logics. This kind of work may require foregoing the default optimization targets of efficiency and cost savings, and instead take a deeper and harder look at the systems and structures that continue to generate inequitable outcomes, replicating the worst injustices of our past. 
<\/p>\n\n\n\n

Can they imagine different stories?<\/em>
<\/p>\n\n\n\n

Stories, Not Solutions <\/strong>
<\/h3>\n\n\n\n
\"a<\/figure>\n\n\n\n

What would it look like to embed hope<\/a> or aspiration into our vision of technological progress?
<\/p>\n\n\n\n

As long as we are stuck in myths about dominating \u201cnature\u201d, about othering our environment and each other, neither human nor machine logics will be able to transcend our ecocidal operating system. 
<\/p>\n\n\n\n

As my colleague Salome Asega has<\/a> pointed out, \u201cthe fear of superintelligent machines is really channeling something much deeper \u2014 a guilt and fear of machines doing unto humans what humans have done unto each other.\u201d 
<\/p>\n\n\n\n

We are something that the Earth is doing<\/em><\/span><\/p>– unknown<\/cite><\/blockquote>

\n\n\n\n

Can you start a real revolution without chaos? Maybe not. But if you could, it would start with a new story<\/a>. Creating a new story starts with recognizing how fundamental storytelling is to the production of \u201cevidence,\u201d which our current planning and innovation infrastructure only absorbs through institutional touchpoints that confer \u201cauthoritative\u201d analysis (read: elite institutions, large-scale data-sets, randomized (if not reproducible<\/a>) controlled trials). As we have seen with MIT\u2019s \u201cfuture factory,\u201d such institutions undermine the lived experience of the communities most impacted by public planning at a structural<\/a> level.  
<\/p>\n\n\n\n

It also means recognizing that our civic infrastructure was built for the normative perpetuation of a status quo that is morally wrong and fundamentally unsustainable. As a society, we don\u2019t need to agree on policy choice, problem framing, or ideology. But can we agree to build the infrastructure needed to have those conversations? Can we move toward centering different stories<\/a>?
<\/p>\n\n\n\n

This infrastructure won\u2019t be delivered algorithmically. In lieu of infrastructure and apparatus for societal civic sensemaking, we\u2019ve seen fit \u2014 tautologically absent any capacity to debate the trade-offs in clear terms \u2014 to automate decision-making via pattern identification and replication. We currently have a massive swath of our energy and resources invested in programming our future based on the flawed patterns of our past. 
<\/p>\n\n\n\n

Algorithms, when commonly applied toward the optimization targets of efficiency or cost savings, have the effect of not only casting historical biases inherent in the data they are trained on into our plans for the future, but also function as a mechanical stifling of the space for imagining what true systems reform might look like. If we wish to use innovative tools in our pursuit of making communities better, that innovation must be in how we engage those most impacted by the injustice of our current systems.
<\/p>\n\n\n\n

Make no mistake: this isn\u2019t a technology issue, but a civil rights issue. Unfortunately, there\u2019s a lot of jargon (technical terminology, marketing hype, legalese), keeping communities from engaging in the wave of data collection and technology procurement that will shape how social services like education, welfare, and child protection are delivered for decades to come. There\u2019s little space for the meaningful and collaborative radical reimagining of alternative futures when the public is repeatedly told that the revolution is already here and they just don\u2019t understand it.
<\/p>\n\n\n\n

A Machine Readable Reality<\/strong>
<\/h3>\n\n\n\n
\"four<\/figure>\n\n\n\n

Under the heading of solving problems, we are in the midst of a gold-rush to fit reality to machine readability<\/a>. 
<\/p>\n\n\n\n

Whether it\u2019s \u201cbuilding a city from the internet up,\u201d as Google sibling Sidewalk Labs has proposed<\/a> to do in Toronto, or a retrofitting of sidewalks<\/a> themselves for an autonomous car future, into \u201cbasically cages with defined doors that open only when the traffic lights are green so the world becomes simple enough for cars to \u2018understand.\u2019\u201d This is the default imaginary data-driven future, made manifest. 
<\/p>\n\n\n\n

Alongside the now ubiquitous \u201cdark patterns<\/a>\u201d designed to extract ever more data from our physical and digital transactions and interactions, are the more subtle, long-term effects of algorithmic intermediation.
<\/p>\n\n\n\n

Ali Alkhatib puts<\/a> it like this, \u201cit\u2019s not necessarily calamitous when we build systems that help us interpret the world in simplified ways, but it becomes a problem when those systems begin to reshape the world according to the simple, digestible, but hollowed out metrics that we developed initially to interpret the world.\u201d
<\/p>\n\n\n\n

Just as we scarcely understand the bodily effects of a lifetime of exposure to chemicals in consumer products, the cross-sectoral ideological drive toward data collection will have long term impacts, many of them unintended. In particular, the resultant pervasive distribution of surveillance architecture throughout our built environment, just as we are pushing the limits of natural resource consumption and feeling the impacts of climate change, will affect<\/a> culture and social norms in ways we can scarcely begin to imagine<\/a>. 
<\/p>\n\n\n\n

Today, the line between the corporate capture of human experience, which gobbles up individual-data as \u201cbehavioral surplus<\/a>\u201d for prediction products that shape the space for consumer behavior, is beginning to blur to indecipherability, seeking to capture citizenship as well. It pursues the civic landscape through the same business models, marketing<\/a> events, data evangelism, indeed through the same enterprise tech companies \u2014 leaving out, as always, the communities most impacted, to be acted upon by design.  
<\/p>\n\n\n\n

\u201c[This is] the <\/p>\n\n\n\n

Datafication of Injustice<\/strong>…<\/h3>\n\n\n\n

in which the hunt for more and more data is a barrier to acting on what we already know\u201d <\/p>\n\n\n\n

\u2013 Ruha Benjamin, Race After Technology<\/a>, <\/em>p. 116
<\/p>\n\n\n\n

\"a<\/figure>\n\n\n\n

With our public systems of planning and civic management increasingly trained to frame complex systemic problems as individual in nature, we are effectively planning to blind our power structures to structural inequality. Focused on efficiency, we are doomed to deepen the grooves of inequity.
<\/p>\n\n\n\n

These are already systems designed in a paradigm of scientific management. Problems are viewed as linear and solvable through analytical, bureaucratic, and technocratic means. But the complex problems we face as a society (white supremacy, social and financial inequality, climate change, floating garbage islands, and many more) are interrelated, and exist, at least partially, due to the very systems designed to perform public problem-solving. The transformation necessary to better equip these systems to lift Americans up won\u2019t happen<\/a> through leveraging technology that is fundamentally about identifying and replicating \u2014 more efficiently and at scale \u2014 patterns of past service delivery.
<\/p>\n\n\n\n

The transformation needed<\/a> is to equip locally administered systems with tools of collective action (the availability of descriptive data is only a small piece of this tool-kit) to work in concert with their community members and community-based organizations. 
<\/p>\n\n\n\n

In most cases, communities already know what they need. The slow work of coalition-building and organizing, consensus building, and resourcing human capital just doesn\u2019t often fit into the typically solutionist, scalability-obsessed, ROI focused, optimizing-for-efficiency, \u201cevidence-based\u201d funding hole carved out for \u201ccommunity innovation.\u201d
<\/p>\n\n\n\n

And what is left for communities, unengaged on surveillance and algorithmic management \u2014 even where there has been<\/a> a specific community engagement process \u2014 than to resist<\/a> and refuse<\/a>? 
<\/p>\n\n\n\n

There is nothing inevitable about the application of predictive analytics to our public sector and social services. This is an elaborate delusion. 
<\/p>\n\n\n\n

We desperately need civic infrastructure to reclaim planning for the future from its current<\/a> technocratic neoliberal capitalist<\/a> trajectory. We must extend new norms, laws, and protections for a world in which we must suddenly codify a right to sanctuary and autonomy in our own lives. This may even require a new language in order to move past what already feels<\/a> like an outdated reliance on binaries like human|nonhuman and man|machine which in practice, only provide an intellectual cover for hierarchical oppression. 
<\/p>\n\n\n\n

We need to recognize the colonial<\/a> flavor of the data extraction suggested by technocratic approaches to community improvement. Institutions must accept responsibility for building the infrastructure to enable truly informed<\/a> consent, without unduly burdening individuals. But it\u2019s up to ordinary citizens to reclaim planning for the future, by telling different stories.<\/p>\n\n\n\n

These stories might not have us at the center, or at least, not the “us” we think we understand. We might be more of a supporting character in a story that centers on the interconnectedness of all things as a central driver of purpose and experience. By failing to conceive of ourselves as the multivariate and interdependent ecosystems<\/a> we are, we plan on a future that won’t include us if we can’t include the multitudes we contain and are contained by. <\/p>\n","protected":false},"excerpt":{"rendered":"

Is technology good or bad for the future? Should we be excited or scared of new technological developments? Perhaps we’re asking the wrong questions. This is not an essay about technology, but an essay about systemic inequality.<\/p>\n","protected":false},"author":23,"featured_media":2724,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[25],"tags":[],"class_list":["post-2712","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-issue-6"],"_links":{"self":[{"href":"https:\/\/itp.nyu.edu\/adjacent\/wp-json\/wp\/v2\/posts\/2712"}],"collection":[{"href":"https:\/\/itp.nyu.edu\/adjacent\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/itp.nyu.edu\/adjacent\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/itp.nyu.edu\/adjacent\/wp-json\/wp\/v2\/users\/23"}],"replies":[{"embeddable":true,"href":"https:\/\/itp.nyu.edu\/adjacent\/wp-json\/wp\/v2\/comments?post=2712"}],"version-history":[{"count":5,"href":"https:\/\/itp.nyu.edu\/adjacent\/wp-json\/wp\/v2\/posts\/2712\/revisions"}],"predecessor-version":[{"id":3192,"href":"https:\/\/itp.nyu.edu\/adjacent\/wp-json\/wp\/v2\/posts\/2712\/revisions\/3192"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/itp.nyu.edu\/adjacent\/wp-json\/wp\/v2\/media\/2724"}],"wp:attachment":[{"href":"https:\/\/itp.nyu.edu\/adjacent\/wp-json\/wp\/v2\/media?parent=2712"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/itp.nyu.edu\/adjacent\/wp-json\/wp\/v2\/categories?post=2712"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/itp.nyu.edu\/adjacent\/wp-json\/wp\/v2\/tags?post=2712"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}