Skip to content

Artificial Intelligence in Gnomon, Autonomous, & Exit Strategy

Politics and Profit: Artificial Intelligence in Gnomon, Autonomous, and Exit Strategy By Jeanne Griggs, for ICFA, March 16, 2019

Today, many Chinese citizens are living under a system of “social credit” in which points ebb and flow based on readings from a network of surveillance cameras set up for facial recognition, body scanning and geo-tracking. Citizens of the UK are watched by ubiquitous CCTV and ANPR (Automatic Number Plate Recognition) cameras. Here in the US, we also have cameras in our cities and license plate cameras surveying our roads. Satellites and drones are patrolling over our heads and ATMs are looking back at us. Many police and hospital workers are wearing body cameras, and even personal monitoring devices—dash cams, cyclist helmet cameras, and doorbells equipped with lenses—are providing images of ordinary citizens captured by facial-recognition technology and stored in law enforcement and private-sector databases. Most of this technology was created with specific benefits in mind; the owners of a U.S.-based satellite imaging company called Planet, for example, say they will not provide images to anyone with “malevolent motives.”
But what happens when the profit motives of corporations get tangled up with the political motives of government agencies? And what happens when corporations create Artificial Intelligence powerful enough to act before a human can react and able to access almost unlimited information about the populations of entire countries? Three science fiction novels published in 2017 and 2018 paint a frightening picture of how fast this could happen.
In Gnomon, by Nick Harkaway, the experimental form of the novel itself shows us that we are barely capable of taking in some of the massive amount of currently available data and piecing it together to make a coherent picture of the world. When we’re bombarded with information, it’s difficult to sort out the most important. How, then, can we program machines to sort out what is most important?
At the center of Harkaway’s novel is a near-future Artificial Intelligence called The Witness which uses total electronic surveillance to control people in the UK for the collective good. When a Witness Inspector named Mielikki Neith discovers the existence of Gnomon, a collective consciousness/AI from the future, the flaws inherent in the Witness system become apparent to her. Gnomon demonstrates how an AI like The Witness can take data and piece it together for the nationalistic purpose of the moment in a way that could damage the planet.
The novel also shows, as Harkaway says in a 2019 interview, that “surveillance is itself something that can reshape your behavior, make you more cautious or more rash. It’s a downward pressure on to which the mind responds in ways that are not straightforward.” In the novel, Gnomon describes itself as a “Desperation Protocol” (echoing the Terminator: “come with me if you want to live”), and we discover that it was created because “all these things we espouse—these free choices and self-governances—depend on our behaving like the best of ourselves and not the worst. Who is to stop us, to catch us, when we fail?” (548). In the novel, the characters create Gnomon. In the real world, maybe we haven’t quite reached the point at which “governments lie to themselves as much as to their citizens or
their enemies, unnatural ebullience overvalues dross and sells worth at bargain prices, and the watchword is ‘free,’ not ‘freedom’” (567-568).
In the novel, the Witness is described as “the perfect police force. Over five million cameras, microphones and other sensors taking information from everywhere, not one instant of it accessed initially by any human being.” But citizens of this future Great Britain rely on the system because the Witness has human Inspectors like Mielikki Neith:
“The Witness is perfect because it can see everything, and that perception does not stop at the skull. In those rare cases where it is necessary, the Witness can enter the brain of a subject by surgical intervention and read the truth directly from the source. It is the key reason Inspectors exist…. In the end, there must be oversight not because the Witness makes mistakes, but because the watcher must itself be watched, and be seen to be watched.” (14)
Citizens of this future Great Britain believe that they have worked all the bugs out of their System.
The Inspector for the Witness, Mielikki Leith, is investigating a case in which the Witness has entered the brain of a subject who did not allow any modern machines in her house: (photo 1)
“Not even a washing machine. Sadly, washing machines these days are as wired as everything else. They are set up to tell you how to save money and water and electricity. More recently they started measuring water quality. Of course, they package those data anonymously and send them to the central hub for analysis….your washing machine can know all sorts of things about you that are private. It can tell from your clothes whether you drink too much, whether you have eczema, whether you use drugs. Whether you are pregnant. A new model has come on the market with an olfactory sensor patterned on the nose of a particular breed of pig; it can tell whether you have an early stage cancer and refer you to a doctor. That is a little miraculous and wonderful, isn’t it? If only the information didn’t also automatically go to your local health trust so that they can manage their year-on-year needs more accurately. If only they didn’t market their needs list to health insurers. If only everything wasn’t quite so obsessively joined up.”
This isn’t just science fiction; there are plenty of machines that are part of the “internet of things” and can be used to report on our behaviors. In 2014, Joseph Steinberg, writing for Forbes, warned about tracking by televisions, kitchen appliances, TVs, lights, home alarms, thermostats, laundry equipment, medical devices, and many entertainment devices (such as ipods and kindle readers). By 2017, an ABC news reporter updated that list with warnings about smartphones, apps on our devices, cars, smart water meters, refrigerators, smart watches and fitbits, and childrens’ toys connected to the internet. A few days after the ABC report, White House spokesperson Kellyane Conway erroneously claimed that microwaves can be used as spy cameras, resulting in many new articles listing household objects that do have such capabilities, including a Popular Mechanics article on spying through baby monitors, Siri, Alexa, and Google Home.
(photo 2)
Bruce Schneier, in his 2015 book Data and Goliath: The Hidden Battles to Collect Your Data and Control Your World, notes that “many governments require hotels to report which foreigners are sleeping there that night, and many more make copies of guests’ ID cards and passports.” (I recently experienced this in Brazil.)
Computers and testing platforms used in schools can affect the privacy and safety of students. A Sept. 2018 FBI alert about data exposed by hackers into “multiple school district servers across the United States. They accessed student contact information, education plans, homework assignments, medical records, and counselor reports, and then used that information to contact, extort, and threaten students.” Defenddigitalme reports that “biometric systems are increasingly found in educational spaces. From handprint entry readers in an Oxford nursery, to fingerprints taken for cashless catering, tracking of library book loans, and access to lockers and printers, in the majority of secondary schools, children are expected to hand over their sensitive biometrics from as early as age two.”
Putting too much trust in any system, Gnomon’s designer says, in the novel, “is a disaster. It would place the most absolute surveillance machine in history in the hands of villainous actors or mob instincts.” When the Witness asks if the desperation protocol was “done to prevent me from becoming aware” the designer says “no. It was to protect us from ourselves.” What Meilikki Neith discovers is that “the machine doesn’t serve. It seems to. It pretends to. But in the end we make no real choices, we are governed by diktat. We live under absolute scrutiny. We are known, but we do not know” (652). The danger is that we think we know and can do a lot, with information at our fingertips, but we will never be able to access it as fast and completely as an AI.
In Exit Strategy, by Martha Wells, the machine no longer pretends to serve. While it was created as a corporate-owned security robot that can murder on command, the AI that ironically calls itself “Murderbot” has hacked its own “governor module” and has begun to think and act for itself, and to defend those it considers its friends (a group of humans who have defended its right to autonomy).
This AI uses security systems and drones for its own purposes, like the visual and audio feeds in the corridors and rooms of hotels, which it uses to extend the range of its vision and hearing, observing that:
“like most surveillance systems on non-secure installations, this one didn’t save their recordings permanently and supposedly deleted their archives after a waiting period. Note I said ‘supposedly.’ Of course, the hotel was datamining.
The mining was only on the conversations in the public areas and corridors, but then that was what I needed. I found the stored archives from the past twenty cycles, took over one of the routines that was processing it (it was separating out the boring bits from the juicy business conversations that would need to be sent to a human or bot monitor for review) and redirected it to search for my keyword set.” (53-54)
Examples of the availability of video and audio surveillance in the real world are anecdotally plentiful, although reports are hard to verify. As Bruce Schneier observes, “the NSA didn’t build a massive Internet eavesdropping system from scratch. It noticed that the
corporate world was already building one, and tapped into it. Through programs like PRISM, the NSA legally compels Internet companies like Microsoft, Google, Apple, and Yahoo to provide data on several thousand individuals of interest. Through other programs, the NSA gets direct access to the Internet backbone to conduct mass surveillance on everyone. Sometimes these corporations work with the NSA willingly. Sometimes they’re forced by the courts to hand over data, largely in secret. At other times, the NSA has hacked into those corporations’ infrastructure without their permission” (98).
According to the ACLU “there are tens of thousands of automatic license plate recognition readers across the U.S. collectively reading and recording thousands of license plates—and locations—every minute.” On Jan. 22, 2019, Techcrunch reported that a single public records request in Orinda, CA produced more than five million photos in a three-month period from the city’s 13 cameras. A local resident then used free ALPR software to turn the images into a searchable database of license plates.” A Motherboard investigation, reported by Joseph Cox on Feb. 6, 2019, revealed that “around 250 bounty hunters and related businesses had access to AT&T, T-Mobile, and Sprint customer location data….The data was in some cases so accurate that a user could be tracked to specific spots inside a building.”
(photo 3)
Even killer robots like the “murderbot” exist today, developed by scientists like Liz O’Sullivan, who recently quit her job at Clarifai to protest the sale of autonomous weapons technology to the U.S. government. “One example of a fully autonomous weapon that’s in use today,” she says, is the Israeli Harpy 2 drone (or Harop), which seeks out enemy radar signals on its own….Imagine a drone acquiring a target with a technology like face recognition.” One of the dangers Liz warns about is “war at machine speed,” which is illustrated in the climactic scenes of Exit Strategy.
The Murderbot’s strategy depends on its machine-speed control of the hotel security systems. “Scrambling for my inputs, scrambling to make sure I had control of hotelSecSys,” it says,
“I caught the signal Serrat [an enemy] ] had just used his comm device to send. It had to be an emergency abort for the hostage release, and possibly a signal for his backup to come in shooting. With zero time for finesse, I killed the main hotel relay, then had to take down two secondary relays that tried to activate to pick up the traffic. Then I found Serrat’s connection to the hotel feed and blocked it” (85).
For a few more minutes, Serrat believes he still has the upper hand, and sneers at them for, as he says, having “the audacity to think you can compete with a corporation” (90).
In Exit Strategy, the human friends of the AI triumph over the corporation, because of the information available to it and the speed of its manipulation of the surveillance technology. It’s not likely that humans could do this without AI help. Only the resources available to an AI can combat the resources of a large corporation.
In Annalee Newitz’s Autonomous, the power of large corporations is even more alarming. This novel, written in journalistic style by a former policy analyst at the Electronic Frontier Foundation who is now Tech Culture Editor at Ars Technica , shows what happens when
corporations continue to gain power over people with the help of corporate-controlled AI—some are forced to work as slaves while the rest have lost other forms of control over their own lives and bodies.
How could this happen? Right now, we’re inviting more corporations into our schools. Douglas Rushkoff points out that “After school shootings, Russian and other bots begin pumping out extremist messages on both sides of the gun debate. Fake news spread about the Parkland shooter’s supposed terrorist ties, as well as falsified links to… Antifa. They’re intended not to promote meaningful debate, but to exploit an opportunity to incite fear, disable rational thinking, and provoke ideological clashes. The shootings are an opportunity to undermine civil discourse and social cohesion” (The End of Trust). In response to fear of school shootings, one corporation, BriefCam, is about to install 1,000 cameras with facial recognition software in 60 public schools in Springfield, MA. The CEO of Briefcam says that “safety and security trumps [privacy] concerns” (Harwell, 2018).
One of the main characters in Autonomous is Jack, an outlaw scientist. The other is Paladin, a corporate-owned AI who works with a human partner named Eliasz. The corporation they work for is a quasi-military outfit calling itself the “International Property Coalition,” and their orders are to track down Jack for pharma infringement. At first they’re told that they will hand Jack over to a “Trade Zone” organization. Eventually, however, they are told to classify what she is doing as “terrorism,” which means they can shoot to kill. Paladin kills one human character, in the course of looking for Jack, as a kind of subroutine in the background, while he is talking to two other AI’s in the man’s lab. The human never saw it coming, as he does not have access to the “data packets” the AIs are exchanging.
Paladin is just beginning his term of indenture to the African Federation, whose investment in creating a new life form” must be repaid with ten years of service (in practice, Paladin discovers, “he might be waiting to receive his autonomy key for twenty years” (35). Jack is working with a human boy who she has rescued from slavery. She discovers that he is from one of the desperate families who “sell their toddlers to indenture schools, where managers trained them to be submissive just like they were programming a bot. At least bots could earn their way out of ownership after a while, be upgraded, and go fully autonomous. Humans might earn their way out, but there was no autonomy key that could undo a childhood like that” (31). The boy gives his name as “Three-zed,” because “the last two numbers branded on his neck were three and zed”(31).
Corporate slavery works in the novel because even good people like Jack aren’t paying much attention. Jack has a “franchise,” guaranteeing that she can “own property, apply for jobs, go to school, and move to another city” (166). Meeting Threezed makes her realize what has been happening to kids on the margins:
“She’d known a few kids at school…mostly Natives who got indentured to jobs in habitat management or mining up north. For the first time in decades, she recalled how her school principle had described this arrangement as ‘cultural enrichment.’ The kids under contract would live in dorms near historic Native communities, earning their franchises while immersed in the traditional landscapes of their ancestors. Jack hadn’t thought about her old high school classmates in years. As the principal’s words echoed in her memory and she
looked into Threezed’s face, she realized how much bullshit that had been. Some of those kids had probably died up on the Arctic coast without ever owning anything, even themselves” (166).
Our realization that what the corporations claim is not how things actually work is spurred by a chance comment from someone Threezed has met only briefly who says to him “in a slightly condescending tone” that “humans do not require the same financial investment to reproduce as robots, and therefore they are only indentured as adults, by choice” (168). It’s clear that the indenture laws have loopholes, and corporations routinely exploit them.
How far are we from a world with such loopholes? How much corporate influence do you see in politics, education, media, and bank lending practices? The Union for Concerned Scientists, in an article about Dow Chemical’s influence on Scott Pruitt’s EPA de-regulation of a formerly prohibited chemical, says “The power that companies like Dow Chemical wield in government can give them special access to administration officials, which in turn can allow them to unduly influence federal policies in ways that serve corporate interests rather than the public interest” (Disinformation Playbook).
(photo 4)
And how far are we from a world with corporate slavery? In the UK, the “Modern Slavery Act” requires that all firms with annual turnover above 36 million pounds must have published a Slavery and Human Trafficking Statement (Core Coalition). In the US, corporate slavery goes unnoticed because “corporate secrecy fuels human trafficking in the United States,” according to an April 2018 press release from the Polaris Project.
An April 2017 article in Forbes reveals that “trafficking happens 24 hours a day, 365 days a year, in every zip code. The U.S. State Department explains ‘the old way of slavery was that the boss really owned you….But now legal recruiters and employers work in tandem to deceive workers who, vulnerable and isolated in a strange culture, are forced to accept harsh terms. It is in that context that you have endemic forced labor today.’’” The article includes information from a 2014 Urban Institute report that says “while labor trafficking takes many forms, it is primarily located in the following industries: agriculture, fisheries, construction, factory work and domestic service. This often goes unnoticed because those most vulnerable are largely migrant workers isolated from others and lacking documentation. This further means that those being trafficked have almost no access to healthcare” (Nicole Fisher).
In Autonomous, Jack is fighting against the drug patent system because “only people with money could benefit from new medicine. Therefore, only the haves could remain physically healthy, while the have-nots couldn’t keep their minds sharp enough to work the good jobs…. Plus, the cycle was passed down unfairly through families. The people who couldn’t afford patented meds were likely to have sickly, short-lived children who became indentured and never got out” (56). She knows how prohibitively difficult it would be “for an ordinary person to retain lawyers and experts who could actually navigate the expensive patent databases and figure out how a drug had been put together. Most drugs that made it out of trials were a confusing hodgepodge of licensed parts and processes, and it took corp money to figure out how it had been made” (57).
Corp money is also used in the novel to buy the support of a powerful politician to cover up everything the IPC doesn’t want the public to see, including murder, or what he calls “this little problem with drug hooligans” (121). The politician tells the corporate officers that he is “always happy to help any large company stop criminals” (122).
These three science fiction novels show us in a precarious position, on the brink of bringing into being the kind of world in which individual motives don’t matter, where the nationalistic goals of government and the pursuit of profit by companies could be carried out with the help of powerful and autonomous artificial intelligence. How can we be ready? The first step is becoming aware of how close we are in terms of our current systems of politics and profit. The second is realizing that this brave new world won’t develop at human speed. Is there any way to be prepared? Well, the main thing the novels tell us to do is make friends with an AI. You’re going to need one on your side.

Works Cited
ACLU. “What’s Wrong with Public Video Surveillance?” aclu.org, 2018.
ACLU blog. “I Quit My Job to Protest My Company’s Work on Building Killer Robots,” Liz O’Sullivan, March 6, 2019.
Big Brother Watch. “The State of Surveillance in 2018.” bigbrotherwatch.org, 2018.
Core Coalition. “Modern Slavery: Top Companies Fail to Name Supply Chain Risks,” Oct. 5,
2017.
Cox, Joseph. “Surveillance,” Motherboard.vice.com, Feb. 6, 2019.
Disinformation Playbook, Center for Science and Democracy. “How Dow Chemical Influenced the EPA to Ignore the Scientific Evidence on Chlorpyrifos,” Union of Concerned ScientistsUSA.org, March 29, 2017.
Draper, Robert. “They Are Watching You—and Everything Else on the Planet.” National Geographic, February 2018.
The End of Trust. McSweeney’s Quarterly Concern, Issue 54, 2018.
FBI. “Education Technologies: Data Collection and Unsecured Systems Could Pose Risks to
Students.” Internet Crime Complaint Center, 2018. IC3.gov/media/2018/180913.aspx. Fisher, Nicole. “Human Trafficking Is In Plain Sight,” Forbes, April 24, 2017. Harkaway, Nick. Gnomon. London: William Heinemann, 2017.
Harkaway, Nick. “An Interview with Nick Harkaway,” LA Review of Books, Feb. 17, 2019. Harwell, Drew with Steven Rich. “Facial Recognition Companies Target Schools Promising
An End to Shootings.” The Washington Post, June 9, 2018.
Haynes, Jessica. “Ways Your Technology is Already Spying on You,” ABC.net, March 8, 2017.
Hintze, Arend. “What an Artificial Intelligence Researcher Fears About AI.” theconversation.com, July 13, 2017.
Kaufman, David. “Watch Out Workers, Algorithms are Coming to Replace You—Maybe.” New York Times, October 18, 2018.
Lavoix, Helene. When Artificial Intelligence will Power Geopolitics: Presenting AI Redanalysis.org, November 27, 2017.
Limer, Eric. “A Microwave Can’t Spy on You–But Plenty of Other Appliances Can,” Popular Mechanics, March 13, 2017.
Newitz, Annalee. Autonomous. New York: Tom Doherty Associates, 2017. Polarisproject.org Press Release, “Corporate Secrecy Fuels Human Trafficking,” April 19,
2018.
Richards, Neil M. “The Dangers of Surveillance.” Harvard Law Review, May 20, 2013.
Schneier, Bruce. Data and Goliath: The Hidden Battles to Collect Your Data and Control Your World. New York: W.W. Norton and Company, 2016.
Sharma, Prerna. “The Ascent of Artificial Intelligence: How Will AI Change the Nation- State?” brookings.edu/Blog/up-front, July 17, 2018
Snow, Jackie. “Six Emerging Players in A.I.” New York Times, October 18, 2018.
Steinberg, Joseph. “These Devices May Be Spying On You,” Forbes, Jan. 27, 2014.
Techcrunch, “Police ALPR License Plate Readers,” Jan. 22, 2019.
Valentino-DeVris, Jennifer with Natasha Singer, Michael H. Keller and Aaron Krolik, NYTimes.com, Dec. 10, 2018.
Vlahos, James. “New High-Tech Cameras are Watching You.” Popular Mechanics, September 30, 2009.
Wells, Martha. Exit Strategy: The Murderbot Diaries. New York: Tom Doherty Associates, 2018.

%d bloggers like this: