‘Exploit Machina’ With Andrea Matwyshyn

ANDREA MATWYSHYN: Thank you for such a lovely introduction, Elizabeth, and thank you to Danielle and Elizabeth for inviting me here to share this work with you. It's really a thrill to be here. There's even a connection to Virginia in multiple places in the slide deck. So there's your bonus prize if you can find that little Easter egg of sorts.

And I would tremendously appreciate your comments and feedback on this. This paper is heading out to law reviews this weekend, assuming that my amazing RA gets the notes done, if not, by next weekend. So plenty of time for feedback.

All right, so without further ado, this is a paper called Exploit Machina. And for those of you who are of the tech world, you know that exploit means a piece of code that takes advantage of a vulnerability. And, in this case, it's a little bit of a play on words. So who am I? You heard about this? I've done some things. I've been appointed to some government agencies. I hang out with hackers. They're cooler than I am, et cetera.

All right. In case a nap overtakes you during this talk, here is the whole talk in one slide. So what I've started to notice-- it pains me to say-- is that there is a dynamic in our economy and in our society where broken technologies, and I mean that in the technological sense, are combining with broken governance processes, both in industry and in government, to cause irreparable harms, sometimes at scale. I think this dynamic needs a name, and I think we need to talk about it head on. We have not been doing that to this point.

These situations that I'm calling Exploit Machina are experienced by the humans that are impacted by them as insider attacks. So even though technologically we may not label them, strictly speaking, as a matter of security, as an insider attack, when you look at the way that they disrupt the principles of confidentiality, integrity, and availability from the perspective of the human beings that are impacted by them, they feel like insider attacks. In other words, a party that is trusted proves untrustworthy from the perspective of the humans that are impacted.

These are not new problems, but they're a little different. So for centuries now, we've been engaged in a conversation between science and what Hayek called scientism. And sometimes the line between these two can be a little blurry, and sometimes we don't know what it is at Time 1, but at Time 2, it becomes apparent. So this is the hard comment of nuance being very important when we talk about these topics.

Increasingly today, these topics involve forms of AI, what we call AI, which, of course, is a marketing term that means multiple different technologies, in particular, predictive and prescriptive analytics. So we're having industries building technologies in some cases that are transformational and helpful to humans, and in some cases, purely technologies of flawed judgments about humans that are being resold and leveraged to the detriment of those humans.

So here's where an unexpected source of insight, for me, arose as I was reading during the pandemic for fun-- Hannah Arendt. Turns out Hannah Arendt had a line of thinking about technology theory. I did not know that. And this is one of her most underexplored, under-written-about theories in both the world of Arendt scholars, which I am very humbled to say, they let me come to their party, their conference. So I got to meet some as a total interloper. And it's not present in the law reviews. It's just not. There's, I think, one article that does a drive-by mention.

So taking Arendt's theory of cybernation, this article pulls apart three dynamics-- questions of investment, which come straight out of Arendt, imagination in a Kantian sense, which also comes from Arendt's interpretation of Kant, and concepts of identity that come out of the writing of developmental psychology scholars and, perhaps unexpectedly, Benjamin Franklin, who literary scholars view as the first autobiographer.

So what does this mean in practice? I have two sets of proposals. The first involves focusing our attention on three first amendment sensitive variables-- context and control, the nature of the harm, and the intent or knowledge of the parties involved-- CHI. CHI, or if you're from Chicago, CHI.

This, in turn, sets up the ability to think through how existing threat modeling techniques in computer security can be expanded to engage more robustly with questions of insider attacks and also with public harms, which are currently not represented, neither are governance harms represented in those models.

And finally, I'm advocating for-- counterintuitively, in this moment we're in-- the creation of a new tech safety regulator of last resort. So this is a regulatory proposal focused on the humans that are harmed. It's not about the technologies. So it's a technology neutral approach. And I'm calling it the Bureau of Technology Safety, which abbreviates to BOTS, which is cut off on the bottom there. All right. So that's the talk. Away we go.

All right, so as some of you may have noticed, things have been getting a little close to dystopia in some cases. We've had bionic eyes turned off for reasons of chosen obsolescence. We've had Theranos result in criminal fraud convictions despite a stellar, on-paper board of directors who, in theory, should have noticed that there were significant problems with the vaporware. But they didn't.

We've had Volkswagen executives go to prison in Germany and face DOJ prosecution in the States for building purpose-built software to cheat on emissions exams by EPA. We have an illegal robodebt, as they call it in Australia, regime that led to enough suicides, unfortunately, that there was a national inquiry.

We have an ongoing tremendous post office scandal in the UK, where Fujitsu and the post office required independent postal office operators, postmasters, to use a particular software product that altered their books, and then they were personally held criminally responsible. Suicides resulted from that as well. Thousands of wrongful convictions. We're still unwinding that 10 years later plus, so keep an eye on that.

And in Michigan, there was an algorithm that wrongly accused 20,000 people of falsely committing unemployment fraud. This has now resulted in a settlement. Recently, there was also a problem with a child welfare algorithm in Pennsylvania that was allegedly discriminating on the basis of ableism among the people who were parents being judged in this context. This was an open inquiry when I last checked. I'm unsure as to its status at the moment.

OK. Those are examples of what I am calling Exploit Machina, where an untrustworthy-- and I'm using that as a term of art from computer science-- technology and context combines with a legally problematic set of governance choices with intent or knowledge that together results potentially in irreparable harms. So we're talking about things that feel like insider attacks, potentially at scale, and they are inherently public private problems in many cases. So we have manipulable devices, manipulable data back ends, and manipulable business models.

These problems can be broken down into a number of categories, including, in particular, various forms of fraud-- or worse, depending on what the consequences are. But perhaps most troublingly because these choices are frequently imposed on us by providers that we don't necessarily have a choice to not use. We are being put into these involuntary infrastructures that potentially house Exploit Machina situations.

There's a robust literature about various forms of technology-related data curation problems, data quality problems, and about the lack of transparency that leads to so many of these issues. In particular, I'll highlight Professor Eichinger's work on public private cyber security, Professor Rowe's work on facial recognition, and Professor Citron's work on privacy. All very related to these issues.

My own work previously has looked also at the public private nature of vulnerability, what it means to have fake content on the internet, and, in particular, the article that, I guess, more than one person read, is called Internet of Bodies and relates to the progressing platformization of the human body into an Internet-of-Things-like construct, for better or worse, and the legal implications of that.

So this work stems out of that tradition of legal scholarship, both others and my own modest contribution to it. So is this dynamic of Exploit Machina really new? No, it isn't. Past insider attacks on various kinds of body-sensing devices have a history. And I think that is one of the key takeaways that I'm trying to channel here is that it's important to engage with that history. This stuff isn't actually new, but by not engaging with the history, we don't learn from the repeating problems that arise.

So why is the Exploit Machina framing useful? It highlights emergent feedback effects and the fact that the vulnerability and the exploitation ends up being public and private, with impacts on public safety and democracy. And it's a combined technology and human effect simultaneously, which means that analyzing only half of the puzzle will not get you to an accurate approach for addressing the totality of the problem.

In particular, in the current moment where AI approaches, whichever kind of AI we're talking about, are being increasingly incorporated, we have diminished dynamics of transparency, diminished dynamics with respect to replicability of the results if you think about the generative artificial intelligence tools that you may have played with. I know my biography changes every time I type my own name into one of them, which is a little concerning. And we have the potential for fast harm at scale.

So let's examine-- oh, sorry about the slide alignment there-- let's examine some examples of confidentiality, integrity, and availability in the context of Exploit Machina. Let's start with President Nixon. President Nixon chose to have a recording device in the Oval Office that he wore on his body. And my examples will focus on body-sensing devices to create a theme.

So he consented to the device. He wore the device on him. And yet the recordings from his own on-body device were some of the primary evidence that were used ultimately in the threat to impeach him. They were repurposed. So this is what we might call a self [INAUDIBLE] in security.

Today, we have situations where people who rely on medical devices, such as CPAP machines, are being told by their insurers to use a particular product, but then they end up realizing that they have chosen to participate in various kinds of data gathering that they did not realize at the time. And so they end up feeling like the device is working against them, and they're left with the choice-- be able to breathe or lose access to the machine.

And frequently, the rationale behind this-- just to flush it out momentarily, and I need to speed up or I'm never going to get through these slides-- is that, for example, if people travel, the machines don't always easily get on hospital networks. So because they phone home every night, if the insurer thinks you're not using it every night, that automatically triggers a review. But it might be an availability problem with internet access that's causing this, not a disuse on the part of the patient.

OK. Integrity. So let's talk about witchcraft trials, but you didn't see that one coming. So we can reframe witchcraft trials as ultimately a use of a body-sensing technology. It just wasn't a reliable one. And when you look at the dynamics, in particular, some of the trials in the colonies, but perhaps most obviously in the trial at Bury St Edmunds that Lord Hale presided over, you have a situation with adversarial attacks on the integrity of data-sensing processes by insiders. You have reliance of human insiders on corruption, problem detection, and you have a known history of prior problems that doesn't get addressed. So there were for profit witchfinders.

There were committees of wealthy locals who were setting up the metrics that were used for the sensing experiments. You had a process irregularity in the way Lord Hale handled the proceeding if you read the court reporters retelling of the proceeding. We don't quite have trial transcripts in the same way. But he disregarded expert evidence of a law enforcement officer and his jury instructions would definitely be described as leading in today's terms.

But, in particular, there were also economic drivers. So many of the defendants who were brought before the court had participated in failed economic transactions with the people who were accusing them of witchcraft. In particular, historians argue that there may have been a dynamic around beer brewing, and that there were brewsters who were viewed as troublemakers, that were making beer that was a little too tasty, and so this was a method for removing them from the picture.

And, of course, Lord Hale's writing then came to the colonies and formed the basis, about 10 years later, for our own witchcraft trials, including trials in Virginia despite the known issues of the UK. Well, England's trials at the time, though there were Scotland trials as well.

All right. We have all these same categories of problems today in machine learning contexts. So when you reframe these conversations through the eye of Exploit Machina, you start to see these parallels. Sensors get corroded. We know that to happen. So if corroded sensors are involved in a particular technology, they're not going to get accurate readings. What are you basing off of the inaccurate readings as a consequence of the data input? What judgments are you making?

Stanislav Petrov, for those of you who aren't familiar with this story, may have single handedly prevented us from having a World War because of his disregard for a sensors readings because the sensor was malfunctioning. But the world of Stanislav Petrov is more of a last generation design and architecture when we're talking about these sensor systems.

And we've already had-- long ago, in the '80s-- people die from defaults that were not properly set in a medical device. So software has killed people. This is not a speculative question. It's merely a question of whether we have been able to limit the scale of the impact up till now.

This one may hit close to your hearts. Some of you may remember the Barmageddon phenomenon, where facial recognition technology was malfunctioning, and the company that was administering the exam alleged there was a-- I quote-- "sophisticated DDoS attack," which I have to tell you, I chuckled because those two words, they do not align for me.

And on the internet, people alleged that the company told them over the phone that the website isn't loading because too many people were trying to log in. So that raises the question whether capacity planning was appropriate when how many bar takers you're going to have.

OK. So in the point of unpatched sensor issues, in medical contexts, again, we know pulse oximeters do not work well on pigmented skin. And so what happens is that you have different protected groups having different medical experiences based on the accuracy of these sensors. And this became an important dynamic, in particular, during COVID, where blood oxygenation was an early indicator and the entry for additional treatment. So the sensors may have been inhibiting standard of care in this case because of this unpatched problem that's been known for a long time.

So in summary, Exploit Machina shifts us in an unsustainable direction for safety at scale. There are economic and innovation policy questions, national security questions, and democracy questions. Medical ethics and the beneficence norm are being threatened with an approach driven by moving fast and no longer breaking things, but breaking people. The ethic of doing no harm that permeates both legal and business ethics approaches is being replaced with one of breaking trust, shipping fast, patching later, maybe, and hurting competitors if you can.

The dignity of all humans is default in advancing science in a rigorous, replicable way is frequently being replaced with an approach driven by fast exits, company flips, maximizing ROI-- sorry, ROI-- excluding low-value users, and leveraging low-regulation markets.

All right. Now, let's jump to a different historical era-- 1933. The United States is struggling with the remnants of the depression, and a World's Fair was mounted, funding was raised through an interesting bond offering that I've been trying to review, but U of C hasn't shared the documents with me yet. And what happened here was this grand fair that was entitled, intended to, and attempting to get people excited about the next generation of technology innovation and getting them familiar with many of the technologies that were being created at the time.

But what's interesting is that the conflict between science and scientistic approaches was present even then. There was a pavilion that had incubators with babies, and people would pay a few cents to walk in and watch the babies kept alive by mechanical devices. So this strikes us today as disrespectful, a sideshow, dignity-reducing.

Meanwhile, on the other side of the fairway, there was a best babies contest where babies were judged. People voted on which was the best air quotes baby. They were measured for various different metrics about their body parts, potentially.

So these two stories I juxtapose because the truth is a little gray. This man with the incubators was absolutely scrupulous about keeping private all the records of the babies. Nobody ever knew who those babies were, to the point that the babies could not even identify other babies that were in the incubators with him.

His results at saving babies were superior to what the hospitals had at the time because, at the time, hospitals viewed some types of babies as not worth saving. In particular, he was saving babies at a much younger age, and he was saving babies that were deemed socially not as desirable by the standards of the time.

Meanwhile, the best baby contests come out of the trend toward fitter families and the next generation of phrenological analysis. And was a mainstay of state fairs, in particular, in the neighboring state of Indiana. So the truth here is great, too.

There's a highlight on infant health. However, the idea of measuring things about bodies and making assumptions based on them, we now know, is not a rigorous way of doing health assessment in key ways. So the truth is, of course, again, gray. In particular, here in Charlottesville, you're all familiar with the case of Buck v Bell.

So here's a situation where that quantification led to a certain set of judgments that were driven by, it turns out, in retrospect, inadequate grounding. And we now recognize that Carrie Buck was not of intelligence that would have marked her as being unable to take care of herself. And so this choice to have a Supreme Court case attempt to pave the way for forced sterilizations took hold, including in economically motivated contexts where in New York there is at least one case of an heiress being forcibly sterilized in order to give her fortune to someone else, and this was then scaled into many more forced sterilizations, a problem that lingered for decades.

So this is a historical ramp-up to say that these issues aren't new, but the thinking behind them has echoes in history that we should connect with. One of those echoes is found in the writing of Hannah Arendt. And as most of you undoubtedly know, Arendt is a political theorist who was prominent in the 20th century, best known for her work on totalitarianism. This is not about that work. This is a discussion of other work.

So I would frame this piece of her writing as a discussion of science versus scientism, the ethics of innovation, and the connection to democracy, though I'm not sure that she would agree with me. So here's what I've dug up that I thought was just fascinating. Arendt was a fan of Norbert Wiener, the author who's credited with developing cybernetics.

And so the point of cybernetics is a feedback mechanism that looks at automation processes and Arendt participated in a conference in 1964 on issues of cybernetics and what she called cybernation. The impact of technologies on humans in context for their labor, for their daily lives, for their dignity, and engagement with other humans, and the evolution of society.

So when we look at what she said, Arendt raised concerns with respect to the evolution toward the state of what she called cybernation as an undesirable one. She juxtaposed computer forms of memory with human memory and pointed out that in her mind, they were not the same thing. And so that it is an impoverished framework that equates to and does not leave space for the uniquely human function of remembrance, as she termed it.

And remembrance is a human psychologically-driven curation process. So it is contextual. It is developmental. It is not replicable by a machine out of the context of your life. So by framing it this way, she presents a very interesting and formidable challenge to the dominant paradigm in most of AI scholarship, that presumes that there is a more mechanistic truth about the brain, and that it can eventually be fully replaced with a general artificial intelligence.

Arendt would disagree with that. She would say instead that the attempt to do that is likely to end in what she called a sterile passivity. So she highlighted two dynamics in particular-- a hyper-mathematization of humanity. So focus on extreme quantifiability, even where she would view quantifiability as not necessarily appropriate or a fit.

And the dynamic that she flagged as alienation and isolation, where the people who are involved in the process of the hyper-quantification and computerization remove themselves from society in some ways and choose to not be of the society that they are engaging with. So it is a politics unmoored from common sense for her, as she would say.

So this idea of investment in society, she frames as being countermanded by a conscious choice to think what we are doing. And that's an Arendtian shibboleth. You'll hear that regularly. And, in particular, she highlights that the political integration of technical power is one of the core considerations that we should have on the table as we think about how we're building the next generation of technology and science.

So we commonly hear of the word hallucination being used with computerized systems. Arendt would find that abhorrent. She would say that is something humans experience, not machines. It's also not how the law really talks about things that are not correct.

We talk about mistakes. We talk about lack of fitness for purpose. We talk about unsuitability. We talk about breach. We talk about defects. So this anthropomorphization in the way we even talk about these things is something that would give Arendt pause.

Here are some of the technologies that we might also consider carefully and think about what we're doing. So there is a trend in some classrooms, at least, to have children use brain-sensing headbands. The earliest trials appeared to be in China, but they're in Oregon now, so we're moving along. So what these brain sensing headbands ostensibly do is that they track certain responses in the brain, which that creators assert map to engagement or focus, and that this, in theory, correlates with increased classroom performance on the part of the children.

Now, let's do a little bit of an Exploit Machina analysis. Let's look at the technology. How do they work? They use EEG headbands except the judgments. And you'll notice they're using judgment words in the way they're describing the technology properly. Says who? Efficiency. Says who? OK.

EEGs-- you can use them to assign a score, but when you're looking at them, you have to realize that there's a long history of their being manipulable. There's a long history of their having limiting functions. So if the child's hair is greasy, if the child had a caffeinated beverage, if the child has oily hair, if the child looks around a lot, if the child has low-blood sugar, these are all things that could impact an EEG reading.

If the child has coarser hair, that's a problem too, potentially. So before we trust the sensor readings from that kind of a device and trust the judgments that are derived from the sensor reading of that kind of a device, the question of whether we really have a sense of what the metrics are that being measured, much like in the witchcraft trials, how they were constructed, and what they're actually telling us.

So brain sensing-- engagement does not mean learning. Attention doesn't mean thinking. Focus doesn't mean creativity. It doesn't mean innovation. I say this to you as someone with a PhD in developmental psychology. Those metrics are not the same. In fact, it's daydreaming that lets you process what you've learned to come up with the next step of your thinking.

So when we reframe these questions and ask whether there are placebo effects that are controlled for such as enhanced parent engagement, which is what some critics have questioned, we start to have a slightly different perspective on whether we should fully trust these technologies. On the point of isolation, there are an unfortunately large number of billionaire bunkers being built right now, and that could signal some isolation.

So let's turn to imagination and identity. For Kant, and this is channeling Arendt's interpretation of Kant. Some people like this approach, some people don't. For Arendt, there are two kinds of Kantian imagination-- productive and reproductive. So reproductive is the use of external baselines to use what you already know to just do it again, to summon it, to do it again.

But productive imagination takes your baseline, but builds on it. It gives you the space to think ahead, to form opinions and judgment. And this notion of imagination dovetails for Kant with notions of heptonomy and autonomy, which I wrote about in a different paper, which is basically the judgments you make inside and then the judgments you act on with the rest of the world. This imagination happens first. This is about listening to yourself and then taking the next steps.

So maximizing space for productive imagination creates the opportunity for those issues of innovative thought that we would ostensibly like to see. But again, we get into measurement questions. So this is a technology that ostensibly is intended to help advertisers, but it's also being used in cars. So the next time you're riding in a car, ask whether you're being facially recognized and sensed and what that means for merger with other databases, particularly, because some of these companies are expanding into mental health questions and potential diagnostics that could be resold as judgments, potentially.

OK. So we have multiple areas of law that are relevant to these issues, of course, and I will-- in the interest of time because I have 4 minutes left-- I'll just get to the end. Identity-- we tell ourselves, and developmental psychologists firmly believe this, that we are the authors of our own story, and that we get second chances. We give ourselves second chances.

So in a world where you are prescriptively given an identity based on longitudinal tracking, like a bad mango on a blockchain, what does that mean for that self narrative? And the king of autobiography-- king may not be the right word-- the innovator of autobiography was Benjamin Franklin. And so when we look at another example, I'll skip this example. But this is a speech pedometer that judges children's brain function, reading skill and is being used by-- well, they may or may not be autism screening-- not sure. But it's used by many school districts now.

Oh, the metric turns out includes adult word count. So whether it's measuring context versus the actual child and how to think about that balance in the way that you engage with these judgments. These are all things to think about. And, in fact, they are putting children's data on the blockchain. So my mangoes, that was not as flippant as I would have liked it to be.

I will skip the story of Tableau, but, basically, his was a cautionary tale of how you can start to question your own beliefs about yourself with the push from external forces. And we've had the first criminal conviction for nudging someone to commit suicide. And we have cases pending that will test that line of nascent case law in the context of AI.

OK. We also have the first brain privacy statute. Colorado had concerns about this kind of imposition of excessive metrics, perhaps much like the Nixon cell phone concern. And for me, this is a legal flashback to the reason we banned polygraphs in employment contests, so things to think about.

All right. In my last one minute, here are all my solutions for whatever it's worth. So focusing on context and control on harm, on intent. Engaging in meta modeling that considers explicitly insider threats and public impact. And this can be buttressed with a bureau of technology safety as a coordination, gap filling, harmonization, and source of expertise.

There's a disconnect between the language of technology, the language of law, and the language of values. It's a mess. Here's the slide that I will not walk through, but would be very pleased to discuss all of in Q&A. So I'm just going to scroll through it and point out that the only clear winners here are the attackers, who will be able to compromise all of these technologies.

There is a lot of threat modeling out there, but even though it looks at context and control, it looks at harm, it looks at intent. It does not engage with questions of human internal controls and does not robustly engage with insider attacks. And I have given this talk at security conferences, and they agree with me.

So here's my little meta model. One way to think about adding in some of those considerations, dealing with technology, regularity, organization, and law. But none of this will work as well as we need it to without more. So if I looked at a case study of computer security liability cases on the district court and circuit court level, courts are simply unwilling to grant any kind of injunctive relief, which means that the only way that we can have meaningful prevention of safety harms at scale is through a regulatory approach. And it's understandable. Standing doctrine is a real thing.

So when we look at these three factors of context, harm, and intent, and come back to laws baselines of not knowingly hurting people is good, is better than knowingly hurting people. We start to be able to layer in nuance across the various situations. We start to realize how comparatively underregulated the technology landscape is, especially as we dig into the history behind why each of these relevant agencies was created. And you may have heard the saying that safety rules are written in blood. That is meant not to be a reference to Laundry, it's a reference to the fact that people have died.

So last two slides. Safety alignment, actually I lied. There are a few more, but I'll stop after two. So my bureau, such as it is, would have three branches that is intended as a gap-filling agency to act as a neutral coordinator across other agencies and to assist with addressing the things that are currently falling through the cracks and providing a source of expertise.

In particular, we're talking about the biggest entities. I'm mapping this to the Hart-Scott-Rodino size of person test. So we're not talking about entrepreneurs. We're talking about multi-million dollar entities, the biggest players in our system. We are likely to see less enforcement, not more. So this is a self-consciously novel agency. Those of you who have taken admin recently know why I'm using that language, and ALJs, I'm agnostic on it, but if they exist, they should be in a separate structure because I think that's the only way they'll survive.

The director structure is upheld by most recent case law on point, so I like that. And despite an initial appropriation, there's no reason this can't be self funded so that it would be not a burden on taxpayers. So I'm happy to discuss more and engage. And I'll just leave you with that thought.