1960, 2003, 2007, 2016, 2020

Drawing Worlds

In 1960 — before the Web, before Macintosh, before Online System — a team of engineers worked for months to make their expensive computer systems not just difficult to use, but utterly useless. Steve Russell and his colleagues imagined using the screen of their machine to represent a cartoon solar system, and to give two people the ability to control their own space ship (a small cluster of lighted dots) in a battle against each other. Their creation Spacewar!1 simulated the gravity around a star, the thrust of rockets on ships pulled by that gravity, and missiles fired by each ship which would destroy the other if they hit. Spacewar! put a simulated world into code runners:

Watch Spacewar!
Play Spacewar!

Right now, on countless screens around the world, people play games that work a lot like Spacewar!:

  • You look at a screen into a world.
  • The world has a shape representing you.
  • You are there to play by moving around in it.
  • The world has gravity.
  • The world has dangerous areas that you enter at risk of death.
  • The world has other shapes representing other people.
  • You compete with them, often by pretend violence.

Russell and his colleagues discovered that animated stick drawings could mimic the physics of a world in such a way as to engage the bodily awareness of the person looking at the screen, even if that world was a silly cartoon of a solar system. Giving that person controls for an object on the screen, even very indirect ones such as keyboards2, further encouraged their physical identification with the agents in the simulated world. When we play a game, we experience ourselves inside the world the computer has created. Given how closely the very first video game anticipated video games today, we can ask to what extent Russell should be called a discoverer rather than a creator of computing technology. Like Turing before him, he demonstrated an important part of what computers are for decades later. The

simulators

have proven enormously important as a way for people to work with information machines, perhaps because Russell’s creation matches a fundamental way our minds work:

Our physical intuitions may be supported by a “physics engine” in the brain akin to the physical simulation engines built into video games. … a variety of physical inference tasks as well as simply viewing physically rich scenes engage a common brain network in frontal and parietal cortices.3

As hardware has become more powerful, simulators have evolved from Spacewar!’s drawings of two-dimensional stick figures to photorealistic three dimensional scenes rendered cinematically. They have even become tools for

experiencing a four-dimensional world

Simulation was present in disguised form from the very beginning. EDVAC, the first working Von Neumann computer, was built to predict how missiles would fly through the air. It presented its predictions as long tables of numbers, making it a mathematical simulation but not a visual one4Page machines invoke simulation in several ways. Most importantly, numberland has close kinship with simulators. Architects, mechanical engineers, and visual effects artists manipulate precise numerical recipes to model the physical world with breathtaking precision. Spreadsheets provide a place for seeing math; computer games use math to make a world we can look around in. We have come full circle. from calculating whether it will rain tomorrow to generating vividly realistic downpours in movies.

Information machines in their endless diversity are variations on a few underlying themes.

Pokables

For versatile information machines to become truly popular tools, someone had to marry the universal appeal of the simulator with our shared desire to talk to each other. For thousands of years, the only way to get a message from one person to another was for a human being to transport it5. In the last two centuries electricity has made it possible to move first words, then our voices between two places. By the early 21st century, the human desire to chat with distant friends was served with hundreds of millions of mobile telephones. A few of those mobile phones grew the ability to send email, take pictures, and even access a crude version of the Web. In 1998, one got the name “smartphone”6. It worked nothing like your people connector, and few people used it. The urge to converse with other people is so important that most adults who could afford one bought a mobile phone, but the devices were all talk.

Around 2003, engineers Greg Christie and Bas Ording tackled the challenge of making a page machine that you operated by touching the screen instead of using the 100 year old keyboard and the 40 year old mouse7. Many earlier attempts had failed to produce popular tools. Christie and Ording worked for Steve Jobs, who had brought page machines to the consumer world. Jobs attributed the failure of keyboardless page machines to their requiring the use of an instrument to touch the screen, rather than accepting a person’s fingers as the pointing instrument. The reminding pictures that page machines rely on require the same precision that Engelbart had built into his first pointing machine. As Jobs phrased the issue:

PC operating systems … really needed the precision of the tip of an arrow of a cursor. … the precision of a finger is much cruder. Therefore you can’t use a PC operating system.8

Christie and Ording made two fundamental changes from earlier efforts. The first was to replace intricate reminding pictures requiring a mechanical pointer with big simple shapes meant for fingers to poke at. We can perhaps imagine lots of inventors making the same choice, if only they had worked with a screen meant for fingers long enough. But Christie and Ording‘s second invention was an unprecedented stroke of genius. They gave the

pokable

pictures on the screen some of the physics of real world objects: in motion they had inertia and drag. Pick up your people connector, find a long list, and scroll it. Do you see how the motion slows to a stop? This behavior is called “rubberband inertial scrolling”. Here is the insight, described in the stiff words of the patent Jobs, Christie, and Ording filed:

Scrolling or translation may be in accordance with a speed of movement of a detected object, such as a speed of movement of a point of contact. [Here follows more physics language, including a formula for velocity.]… In some embodiments, the list is scrolled in accordance with a simulation of an equation of motion having friction. … the change [when scrolling to the end of a list] makes the terminus of the list appear to be elastically attached to an edge of the touch screen display.9

“Simulation of motion”! Like Russell four decades earlier, Christie and Ording built a simulator on top of a code runner10. As Christie and Ording worked on their invention, Jobs separately fretted over how to make a mobile phone that would improve the state of the art. Other engineers working for him proposed a wide range of approaches, none of which satisfied him. Then Christie and Ording showed him the pokables, which moved under his fingers as if they could be prodded, slid, and spun:

When I saw the rubberband inertial scrolling and a few of the other things I thought “My God, we can build a phone out of this!” and I put the [touch screen PC] project on the shelf because the phone was more important. And we [made] the iPhone11.

Christie understood that PCs are fundamentally tools for the few:

One of the biggest challenges is that we need to sell products to people who don’t do what we do for a living … normal people — people with better things to do with their lives than learn how a computer might work — to use the product as well as we can12.

As a form of simulator, pokables give us a world to work with, a world that makes sense to our animal brains. Reminding pictures merely gave us sketches to interpret, sketches of an environment that most of us have never seen and don’t immediately understand13.

Like Macintosh, iPhone had to persuade millions of people to spend their own money to purchase one. The consumer market for information machines shaped it in two seemingly arbitrary ways. Computation-free portable music listeners sold well for decades before iPhone. They leveraged computer power on behalf of millions when Jobs shepherded an information machine for portable music playback, iPod, into consumer success. When the time came to present consumers with what would become the people connector, he relied on both iPod and the clunky “smartphone” coinage14 to market Christie and Ording’s breakthrough:

Steve Jobs’ introduction of iPhone

Turing, Von Neumann, Hopper, Russell, Bush, Engelbart, Berners-Lee, Andreessen, and Page worked for public institutions at the time of their breakthroughs15. We know a great deal about their motivations, their false starts, and their intermediate steps as they created the machines which have changed our world. Christie and Ording worked for Jobs’ Apple in conditions of secrecy. It may be that other engineers deserve credit for the core invention of making a world-in-a-screen that translated human intuitions about physical objects into discrete steps that a code runner can work with. To my knowledge the enormously consequential process of developing pokables as the basis for people connectors16 has not become a matter of public record beyond what we read in patent applications and a few lines in a handful of books and interviews17. Games that simulated a world inside the screen had for decades been the most popular face that code runners and page machines turned to the world. When you first touched iPhone it presented you with a pictured slider that moved under your finger just as a physical slider would. Using iPhone you in effect enjoyed the pleasure of playing a video game with each gesture you made to control it. Christie and Ording’s pokables stand with Hopper’s code and the Engelbart/PARC/Macintosh page machine as landmarks of modern technology: information machines that made computing technology useable for newer and much wider populations than the tools those engineers had inherited. Hopper’s code is a meta-tool for millions of tool makers. The page connectors crafted by Engelbart’s heirs serve hundreds of millions of readers and writers. People connectors make sense to everyone over the age of two.

Almost Everyones

People connectors have two cameras. One points outwards, so you can show your friends what you see. The other points back at you, so you can show your friends what they would see if they were standing next to you, sharing your experience. When you take and send a picture with the second camera, it’s called a selfie. You often share selfies with groups of your friends via online communities that span hundreds of millions of people18. 微信19, WhatsApp, and Facebook20 are three you might know. The companies that build these community-making information machines design them to draw in everyone they can reach. We can call them

almost everyones

Almost everyone wants to chat with their family and friends. Almost everyone can use pokables. Almost everyone finds it worth their money to buy a pocketable device that brings them into online presence with their community, and promptly signs up with 微信, WhatsApp, or Facebook. Almost everyones and people connectors fit each other perfectly. You might be surprised to learn that the history of almost everyones stretches back at least to the 1970s. Versatile information machines anticipate their own future. The birth of Facebook shows how. Facebook began as a simple website where students at one college could look each other up. Like Memex, Online System, the Web itself and PageRank, Facebook was created by a researcher21so that other researchers could connect with each other. The older tools managed knowledge. Facebook managed the social connections among a community with physical ties.

We use the Internet’s world wide address system to get to the websites and app stores where we find almost everyones. Within them we use a very different address system: our Contacts or Friends22. Before the 21st century, contacting someone beyond earshot required using a layered public address system like those for mailing addresses, telephone numbers, and websites. On an almost everyone, we can instantly reach members of our communities by poking their picture or saying their name. We find the people in these information machines just as we would find them at a party. If we know them, we say their name. If we haven’t met them yet, we ask a mutual friend for an introduction.

Page connectors and people connectors can serve in each other’s stead23. We use our people connectors for page connector work, such as reading the Web. We use our page connectors to do people connector work, such as sending our friends messages. Both of them are used as simulators to play games. Many page connectors are used as code runners and even mathematical tools, for example by weather scientists. The tremendous flexibility of these devices makes any specific name seem inadequate: how do you settle on one label for a gadget whose purpose is to become many different tools? On the other hand, as we read the history of these machines and learn their inventor’s purposes from their own words, we see that they were designed neither for a single purpose nor for complete generality, but to have important aspects of both24. In common usage that flexibility earns them the label “computer”, but “versatile information machine” is more precise and more accurate25. Turing called the digital computer “universal”, but every computing device ever built has had physical limits that constrained where in the computational universe it could reach quickly enough to be useful26. The machines that reached millions or billions of people — Macintosh, Netscape, iPhone, and each of their variants, clones, and descendants — combined versatility with dedication to a specific field of use.

We can see a loose analogy between this evolution of information machines by inventors and users and a particular form of natural selection. Feathers evolved in dinosaurs long before they took to the air; countless generations after the first fuzzy walkers, their distant descendants took advantage of the light surface area provided by their tufted covering to glide. That shift in use is called exaptation27. Spacewar! was a silly game devised on very serious machines, code runners exapted for child’s play. Online System served the high-minded pursuit of knowledge; Macintosh’s core users exapted its capacity for managing pages to craft advertisements in glossy magazines meant to capture the glance of a dentistry patient. People connectors and almost everyones have proven wildly exaptable, because we live through our relationships:

WeChat should be understood … as a way of life. … What differentiates WeChat from other products is that it is entirely based on people who know one another … the whole WeChat product can be thought of as a friend circle. WeChat tries to make everything that you can do with your friends easier.28

People connectors unify our physical intuition with our social existence. We live through information machines.

Smarts

Here is one small way in which the “smart” in “smartphone” makes sense. When you raise your people connector to your ear to use it as a phone, some part of it is smart enough to turn off the screen so you don’t poke the pokables with your ear. When you lower the device away from your ear, something notices and makes the screen pokable once more. A particular information machine inside your device, call it a screen disabler, maintains awareness of your intention and responds by setting the screen appropriately.

Computers built for human use have keyboards, speakers, and other sensors and senders to move information between the mind of the person using them and the logic recipes they run. The screen disabler does not exchange information directly with your mind. Instead, it serves on behalf of a mind, maintaining awareness through sensors and responding appropriately to what they perceive. Instead of a person using a machine’s senders to inform themself about the world for some purpose he has in mind, the machine uses a sensor to inform itself about the world for the purpose wired into it by its maker‘s mind. We can call such machines

responsive awareness

Responsive awareness dates back decades29. Lately it has acquired the name “Internet of Things”30, despite its failure to share the Internet’s core purpose of providing world scale addresses. A few years ago, we were told that the future of computing was exemplified by an Internet-connected thermostat. I don’t want to connect the thermostat in my house to billions of computers around the world. I don’t want to think about that thermostat at all. Both the “Internet” and the “Thing” in Internet of Things miscast the work I want taken care of. I want my house to maintain an appropriate temperature31. I want it to be aware of my needs so I can be unaware of them and it.

Responsive awareness is another arrangement of information machines. Because the machines often have limited purposes, many of them are not versatile: the screen disabler will not message your friends. Responsive awareness can also take the form of sensors in one place which send messages to code runners far away, typically via the same underlying network that the Web uses. We owe the “Internet” in Internet of Things to that detail32. Unlike the other misnomers which motivated this essay, “Internet of Things” is so obvious a misfit that most serious discussions of the underlying phenomena begin with an attempt to relabel it. Alternate names include “Enchanted Objects”, “Ubiquitous Computing”, “Smart Homes”, “Tangible Computing”, etcetera.

With that, we have finished our story of how technology has evolved since Turing made his discovery 70 years ago. We can consider the full picture of how computers show us our world.

Notes

1. “The Origin of Spacewar

2. and light pens.

3. Fischer, et al. “Functional neuroanatomy of intuitive physical inference” Proceedings of the National Academy of Sciences 9 August 2016.

4. I owe this observation to a presentation by Michael Tiller (around 3:00).

5. Limited and local exceptions include drums, smoke signals, stray carrier pigeons, etcetera.

6. “The Evolution of the Smartphone

7. Brent Schlender and Rick Tetzeli, Becoming Steve Jobs, Chapter 12

8. “Steve Jobs Full Interview at 2010 D8 Conference

9United States Patent 7,469,381 p.64

10. The technical achievement at the heart of iPhone’s screen is often called “multi-touch”, because a necessary if preliminary capability for the screen was to sense more than one touch at a time. Our continued use of this term closely follows the backwards / outdated example set by “smartphone” and “personal computer”, as discussed in the Introduction.

Bret Victor, an engineer employed at Apple during the first few years of iPhone’s public availability, describes his personal work this way:

I became known for … pervasively direct-manipulation interfaces (where users do their thing by moving and gesturing with meaningful objects, instead of relying on verb buttons and other indirect controls), new forms of creative tools and new ways of learning information.

Victor employs the phrase “pervasively direct-manipulation interface” to denote his own craft rather than iPhone as a whole. In particular, his use of “direct manipulation,” a term of art in software design, does not precisely parallel what I mean by “the physics of real world objects” above. Nonetheless, if we repurposed Victor’s phrase as a synonym for “the iPhone interface”, it would be better than most commonly used names for modern technology — better than PC, smartphone, Internet of Things, cloud, and so on. “Pervasively direct-manipulation interface” also puts the twee imprecision of “pokables” in unflattering relief. Unfortunately its formalism makes it a bit of a mouthful for everyday use. Perhaps you can come up with a replacement that has the strength of each label without their drawbacks.

Regarding Victor’s comment on Apple, see note 17. Victor’s remarkable body of work is, among many other things, a rich investigation of the kinship between numberland and simulators alluded to above. Victor also alerted me to “Miegakure”, the simulation of four dimensions linked above.

11. “Steve Jobs Full Interview at 2010 D8 Conference

12. “Apple engineer: We wanted to make a phone for ’normal people’

13. To appreciate just how wide a gulf there is between these two paradigms, first watch chimpanzees explore pokables. Then consider that after decades of refinement, “across 33 rich countries, only 5% of the population has high [PC]-related abilities, and only a third of people can complete medium-complexity tasks.”

14. To introduce iPhone, Jobs says “the most advanced phones are called ’smartphones’”; I don’t know if he accurately reflected contemporary usage, or if his claim lacked basis. Jobs was a famous fabulist. Four years later the scale of iPhone’s impact had become clear, but Jobs continued to apply remarkably mundane labels to it, such as “post-PC”. See his 2011 strategy outline.

15. Also Russell and Whitehead.

16. Larry Page’s company created a copy of iPhone which has spread even more widely than the original, but it is essentially the same device.

17. Because there is such a sparse public record, the books and interviews take the form of “access journalism”, an activity which blends truth and deception. Becoming Steve Jobs by Brent Schlender and Rick Tetzeli is simultaneously a useful account of iPhone’s development, an example of access journalism, and decent discussion of access journalism which helpfully demonstrates the role that self-deception plays in it. See also Bret Victor’s commentary on his years at Apple. Brian McCullough’s “The History of the iPhone, on its Tenth Anniversary” is an excellent synopsis of what we have.

18. There is much more to the camera than selfies. See “Apple Doesn’t Understand Photography” and “Imaging, Snapchat, and mobile”.

19. Aka “WeChat”.

20. See Ben Thompson’s wonderful “The Facebook Epoch”.

21. Well, a college student, Mark Zuckerberg.

22. Introducing iPhone, Jobs announces “We want to let you use Contacts as never before.”

23. Jobs and his colleagues returned to their original task of creating a page machine you operated with your fingers. The result, known as “iPad” or a “tablet”, keeps the pokables around which people connectors are built, but serves different purposes. The majority of tablets are inexpensive iPad copies used to watch videos and play games. A minority are light, durable page machines used for work in which pokables serve in place of reminding pictures.

24. Dediu and Evans each claimed one of the sides of this paradox in a recent Twitter exchange.

25. Vi Hart: “when people say ‘precise’ but mean ‘accurate’ they’re being precise but not accurate.”

26. If you put my bicycle on a big enough rocket you could send it to the moon, but that doesn’t make it a spaceship.

27. “New specimen of Archaeopteryx provides insights into the evolution of pennaceous feathers” Nature, 3 July 2014.

28. “China’s WeChat Way of Life”, Zara Zhang, unfortunately not publicly available. C.f. Benedict Evans: “FB & Google, structurally, are mirrors of user behavior. There are limits to what it’s possible for them to angle, alter or mask out.” Modern computing technology has so thoroughly incorporated humanity into it that you can’t separate people from it without obliterating what it is.

People connectors marry a very simple simulation with the rich reality of our human relationships. Going the other way, modern video games and virtual reality provide lusciously absorbing simulations of visual and auditory worlds that we use in part to simulate alternate human identities:

The appeal of games is due in part to their ability to provide players with novel experiences that let them “try on” ideal aspects of their selves that might not find expression in everyday life. We found that video games were most intrinsically motivating and had the greatest influence on emotions when players’ experiences of themselves during play were congruent with players’ conceptions of their ideal selves.

Przybylski, et alia: “The Ideal Self at Play: The Appeal of Video Games That Let You Be All You Can Be”, Psychological Science 23.

29. An alternate, unwritten version of this essay argues that the SAGE Air Defense System of the 1950s anticipated much of what we’ve done with computers since.

30. As coined by Kevin Ashton in a very different context.

31. “Miscast the work I want done” parallels the phrase “jobs to be done” from Clayton Christensen’s The Innovator‘s Dilemma. See Horace’s Dediu‘s ouevre for many insightful applications of Christensen’s work to modern computing markets. Dediu and his colleague Bob Moesta illustrate jobs to be done in a recorded interview.

32. The distant code runners are called “the cloud,” as ineffective a name as any we have discussed, and one that deserves replacing immediately.