Probes: Biometrics and Interactivity
by Chris Csikszentmihályi


Not so many years ago, when I was working several freelance jobs and trying to get money to go to graduate school, I noticed a poster around the University of Chicago campus that advertised:

Sleep Study: earn $1200.

Some applicants may be eligible to participate in 4 week study of sleep cycles. Applicant will gradually shift their regular sleep cycle by twelve hours over the four weeks, reversing their regular day and night patterns. Study occurs in your own home, except for the last two days, when subject is expected to play games and other social activities in the lab. No drugs.

An inviting description, except perhaps for the lack of drugs. I called, and received a more complicated and in-depth explanation from a gentleman, who pre-screened me for age, gender, and other personal information. He seemed very cautious, but I understood that they didn't want their subjects pulling out after a serious commitment of time and energy. He was supposed to get back to me, and did, although my machine took the message; I called him back, and missed him. After a few more abortive attempts to communicate by phone, I was eventually scheduled for an appointment at their administrative office, on the eighth floor of the old Hyde Park Bank building, near the University campus.

I walked in, and three women were working in the office. One looked to be an under grad, the other two grad students or post-docs. The under grad summoned me to her desk, and began the process of final screening and registration for the study. Again, she was very plodding and exhaustive with her details. I alternately listened and drifted, occasionally tuning in to the conversation to two post-docs were having.

"cycle will be shifted by twelve hours. Here's a chart, showing the daily sleep and wake periods: Sleep is in gray.... windows will be blacked out with thick curtains, and we'll provide bright lights, although we can't pay for your electric bill, which will probably go up by %5... Yeah, her PI is really demanding. He has the researchers compete by giving them the same project... the last two days you'll be in the lab, taking a variety of tests and playing games. We are trying to understand the effect of sleep cycle changes on attention and judgment, for a variety of situations, like flight control tower operators... but the oil light keeps blinking every time I hit a bump... has been dramatically improved, so if you heard anything about it being uncomfortable, it's been changed, so it's very comfortable. You can hardly feel it."

I was sitting unnaturally straight, and blinking, because I couldn't process what it was she had said would be hardly felt. I interrupted her, "I'm sorry... the anal..."

"The anal probe." Pause.

"No one told you about the anal probe?" She looked over to her colleagues, who had stopped their conversation and were looking over at us with misgiving written on their faces.

"No. An anal probe."

"Why don't I just go on, and explain it..." and soon I knew that the storage device which took electrical samples from the probe was kept in a fanny pack, with a wire running into one's ass. There, a probe measured temperature, acidity, and other muck. The "pack" would be worn for the 4 weeks, everywhere I went. I learned that the probe would be removed to defecate, and during sex, but that to leave it out too long would be noticed by the researchers, and I would be dropped from the study without pay.

It was such a stark option, completely unanticipated, and I quickly weighed the many pros and cons, my mind a hard and fast dogfight of dueling opinions: "You need the money." "Remove the probe for sex? No one would have sex with me if I wore a fanny pack." "This is clearly funded by the military... do I want to help them, even in this small way?" "And if someone did have sex with me, how would I explain the probe?" "What if I swapped with a friend?" All these reflections, and many more, echoed through my mind.

I knew it was research for the military because of the phrase "flight control tower." Not long before this I had been working in a steady job, creating software that measured people. I was part of a consulting firm that had been hired by a big retail client. We were prototyping software that would measure a customer's passage through space, how long they idled near the milk, and where their eyes looked. I had been in touch with many researchers doing similar tasks, and I knew that "Flight control tower" was usually a euphemism for a tank, or a helicopter crew. Cognitive scientists studying micro-gesture and body language through Defense grants often speak disproportionately about "flight control towers." Eventually I left, telling them I need to sit on the idea for a while.

While technologies of measurement will always remain contested and problematic spheres, careful analysis of the debate has left me with one epiphany, one bit of added wisdom with which to view the world. Whenever I pass by a stranger wearing a fanny pack -- usually a network systems administrator, in fact -- I now think, "Anal probe, maybe."


Political Interaction

"Biometric Identification Solutions
Integrated Identification Systems" {http://www.viisage.com/index.html}

The phrase "interactivity" has been bounced around for many years in the fields like infomatics, information design, multimedia, and interactive art, and has assumed many connotations. Typically, the technological term Interactivity is supposed to be a kind of freedom. Interactivity might "free the user" to create their own narrative, or sometimes, free the user from the domination of the author. Chris Dodge, for instance, has called it a

...Paradigmatic shift in the authorship of artistic content, where the fulfillment of artistic content is deferred to the viewer, [and] puts the role of the artist into a unique position. We are now the creators of deferred artistic potentials, leading to self-initiated insights on the part of the viewer, rather than a static expression whose content must be uncovered by the inspecting external agent. (Dodge, 1997)

While other interpretations of interactivity have been promoted, by far the most common is this notion of the emancipation of the viewer. The amount of emancipation is always open to negotiation. When the user is free to click on a door in Myst, this provides a little interactivity. Multiple buttons, as found on video game controllers, provide extra interactivity. One can imagine interactive systems based on spoken commands, although such systems often offer little more complexity of interaction than keyboard-based systems in common use. The potential for interactivity is generally viewed in the technical community to be far greater, and a variety of producers of technology continue to look for new methods of more seamless interaction.


Affective Computing

Affective interactive systems are being researched in several places, including San Francisco State, MIT, UCSD (my own university), and a variety of private institutions. These systems involve mechanical recognition of facial emotion, usually through a neural network. Some researchers develop systems whereby random access video is custom-edited for an audience, depending on their emotional reaction to previous footage. One recent TV blurb showed an AIDS awareness message which was able to algorithmically "crack through" a sullen teenager's refined boredom.
{http://www-white.media.mit.edu/vismod/demos/affect/AC_research/interfaces.html}

That an AIDS awareness message was the "content" in this particular bit of technological promotion is no coincidence. One researcher (Paul Ekman) describes potential applications:
From a practical standpoint, different aspects of expression elucidate whether a listener is empathetic or hostile (important in politics and business), distinguish abusive from nonabusive caretakers (social work), predict divorce in dysfunctional married couples, and may incriminate dissembling witnesses (forensics). (Ekman, 1993)

It is difficult, despite these relatively mainstream cultural goals, to not think of problematic applications for this new media. How much television programming in the United States is currently devoted to AIDS awareness? Imagine if such a news report had, instead, depicted a researcher using this new interactive system with a programmed political ad for David Duke, or a commercial for Microsoft, or a bad sitcom? Technologies, including media technologies, are routinely introduced in artistic, entertainment, or public service roles as a way to ease their birthing. Education, art, and public service are all markets, as well as being considered altruistic domains. New technologies are always marketed to these areas.

This new affective interactive technologies pose an interesting challenge to the notion that "interactive" implies some sort of freedom. They are more reminiscent of subliminal messages in films, or the anti-shoplifting musak used in shopping malls, in that an affect-based system has the potential to bypass a user's conscious response. Where Dodge said that traditional media were "a static expression whose content must be uncovered by the inspecting external agent," that agent is now the media itself. The user is not remotely free, but the system is entirely interactive.


Objects of Measurement

The current interest of algorithmic face recognition has existed for some time (Harmon, 1973), and has been used in domains such as law enforcement and social science research. Automatic analysis and recognition of faces is an extension of what humans do very well from an early age. Early studies showed that humans can recognize a known face with very little information. With only a few pixels (on the order of 10x10), one can distinguish the face of a loved one from a database of thousands of faces. This is, in part, because a large chunk of our brain is hard-wired to the face. When one communicates with another person, it is to an expressive face, and our sense of self usually resides behind the eyes and nose, between the ears, over the mouth. Faces are important.

Faces, because of their proximity to the self, the brain, the pituitary, or whatever homunculus you chose, are especially important because they are also tangible, physical objects which lend themselves well to measurement. Viewed this way, faces are natural subjects (things-in-themselves) which may be grist for scientific inquiry. They are objects, we are born with them (to some degree), and thus they are part of the set of things about humans which may be studied as natural objects. Facial cosmetic surgery is one of many disciplines created through scientific investigation of the physical, natural aspect of faces.

Around this locus revolves thousands of years of interest in sorting and categorizing faces. Cranioscopy might be considered the first modern attempt to sort different types of faces and heads. Dr. Franz Josef Gall, who coined the science, gave this account for the start of his discoveries:

The great facility with which some of his fellow school-fellows could commit their tasks to memory, which to him was a work of intense labor, although in matters of reasoning and judgment he felt himself their superior, often proved a grievous source of mortification, and excited in him a strong desire to know the cause of this difference. He at length remarked, that all the boys gifted with this kind of memory had large and prominent eyes... This observation gave rise to others; it suggested the notion, that other intellectual endowments might also be indicated by the features; and Gall, by degrees, came to imagine that he had discovered a number of external signs, which respectively indicated a decided turn for painting, for music, for mechanical arts, or other objects." (Roget, 1815)

The account is telling, because it shows the transition between Gall's socialized prejudices and his attempt to scientize these prejudices. (If the notion seems a little simple to you, it's not surprising: Gall's brain weighed a measly 1,198 grams, compared with Turgenev's 2000+ (Gould, 1981)) But this hybridization of the aesthetic and the scientific continues unabated for the rest of the century. Gall's cranioscopy led to phrenology, craniology, criminal anthropology, and eventually (because it presupposed discrete, exclusive functions of parts of the brain) the notion of hereditary intelligence.

This ability to translate a gaze into scientific action functions on many levels. Sander Gilman quotes an observed example in daily life, from Eugenics Review, a journal of hereditarians:
But it is not merely that Jews "look Jewish" but that this marks them as inferior: "Who has not heard people characterize such and such a man or woman they see in the streets as Jewish without in the least knowing anything about them? The street arab who calls out "Jew' as some child hurries on to school is unconsciously giving the best and most disinterested proof that there is a reality in the Jewish expression." The gaze of the non-Jew seeing the Jew is immediately translated into action. (Gilman, 1993, quoting Salaman, 1912)

But "disinterested" proof wasn't quite enough. By Salaman's era, hundreds of years of scientific analysis of the face had left it heavily contested, and practitioners of hereditary intelligence subsumed the face into a less problematic natural object of study.


Hereditarian Testing

If discrete parts of the brain had exclusive functions, operating on particular types of problems, it might be possible to test them through the responses of the person who's character they help determine. Thus, what was once measured spatially (the natural object above the neck) might be measured once-removed, through specially constructed tests. Hence the "IQ" test, which was designed to yield a single quotient from a variety of discrete measures of intelligence. The test substitutes the "face" as natural object of study with another object, a written examination, effectively replacing one natural phenomenon with another.

In the beginning of this century, the American military embraced the IQ test, giving it to all new recruits. Two tests were used: The Alpha test was taken by recruits who had some knowledge of written english, while the Beta test measured intelligence pictorially (see image), and was used primarily on immigrants. The cultural specificity of the Beta images are striking, and were so even at the time. (For a wonderful description, see Gould (1981, chap. 5)). Crabs and light bulbs, violins and bowling, a phonograph without a horn: IQ testing clearly measured nothing if not American urbanity.

The IQ test was leveraged for a variety of purposes, including limiting immigration of Southern Europeans. Southern European immigration was rising: Many of these newcomers took the test with little or no knowledge of English or American conventions. They were compared with Nordic immigrants, who had largely been in the country for some time, and often spoke english well. The test substituted for the discrimination of the "street arab" by sorting swarthy and pale efficiently, if statistically, and advocates of the test were unapologetic:
...If one wishes to deny, in the teeth of the facts, the clear superiority of the Nordic race on the ground that the language factor mysteriously aids this group when tested, he may cut out of the Nordic distribution the english speaking Nordics, and still find a marked superiority of the non-english speaking Nordics over the Alpine and Mediterranean groups, a fact which clearly indicated that the underlying cause of the nativity differences we have shown is race, and not language. (Brigham, 1923)

Brigham was, of course, talking about traditionally "english speaking Nordics," i.e. English and Canadians. He had no information regarding the english language proficiency of test-taking Nordics who had lived in the country, on average, for many years longer than the southerners. (This is, essentially, a synopsis of Gould, 1981)

One telling moment in the tragicomic history the IQ test was that after its first major use, in the US military, the linguistic and pictorial tests proved so foreign to their takers that the mean scored at the index mental age of 13, many years below the actual mean age, which of course should have caused serious doubt about the test's scale, scoring, and methodology. It didn't, but instead sparked a minor crises during which many commentators fretted for the future of democracy in a country where the average citizen was almost unable to tend to their own affairs.

While the IQ test has declined in popularity, automatic systems to measure human qualities abound. One test, notable because of its Jungian foundations, is the Meyers Briggs Type Indicator. The MBTI, and a variety of similar tests, are widely used in a plethora of institutional applications. They purport to categorize subjects into 16 meaningful groups. MBTI is used, for instance, to coordinate different members of a management team:
As a member of a team, committee, task force, or other group, you will find the material helpful in understanding your own leadership style, your influence on teammates, how you contribute to team functioning, and how to maximize your effectiveness. You can learn how you and your teammates may irritate and annoy each other and how each of you can work to improve your own contribution and your team's productivity. (Hirsh, 1992)


A friend of mine, starting a job at Silicon Graphics, had to take the MBTI, and was output as an INFP, or "Introverted Feeling with iNtuition." This means, among other things, that he
Irritates other team members by
ï being overly perfectionistic or idealistic
ï appearing out of touch
ï becoming fiercely attached to a value not held by others (Hirsh, 1992)

Employee assignments are doled out on the basis of these tests, and they are often used outright in hiring. Oddly, the only thing they are shown to correlate to are other, similar tests.


Infomatics and Metrics
This tendency to automate recognition of human qualities took a new meaning in the 1970s, as computers became available. One author asks "How can a computer be made to recognize a human face? This question remains unanswered, because pattern recognition by computer is still too crude to achieve automatic identification of objects as complex as faces." (Harmon, 1973, p.71) Where previous systems, like the one by French criminologist Alphonse Bertillon, relied on human coders, computers focused increasing attention on computable algorithms.

Like many computer researchers exploring the human/computing boundary, Harmon saw humans as being essentially like computers: "Although the ultimate question of how a face is recognized remains unanswered, a few promising lines of inquiry have emerged. It has again been clearly shown that the human viewer is a fantastically competent information processor." (p. 82) As with the IQ test, more contemporary tools for measurement are often so convincing that they dramatically reformulate conceptions of the object they are said to measure, moving that definition toward the tool's.


Facing Tomorrow
One current scientific model of the face, authored by Paul Ekman {http://mambo.ucsc.edu/psl/nsf9120868}, at the University of California in San Francisco, constrains the problem by acknowledging only 9 human emotional states. Actors are then trained to use only the muscles involved in those emotional states. Photos of the actors flexing only these muscles (looking not unlike Maori warriors) are then used to "train" a neural network (a computational model built -- very roughly -- as models of the architecture of the human brain). One researcher at my own campus proudly claimed that his network "read facial emotions better than the average undergraduate." He didn't seem to consider that the undergraduates might be confused by the apparent artificiality of the images presented them. Furthermore, the use of the word "read" might seem problematic; "Identified as belonging to one in a set of categories" might be a more descriptive phrase.

It is on this basic premise, and a few others {http://www-white.media.mit.edu/vismod/demos/affect/AC_research/emotions.html}, that affect-based computing systems are built. It should be clear from the history of facial analysis that what is being "read" by the analytical system is a subset of what the authors have "written" into that system. It might be credible to believe that there exists a natural subject, "the human face," which might be measured accurately, to yield 9 essential emotions. Or, perhaps, these numbers are arbitrary: This object called face is a construction authored by a society which has long been interested in faces, and with mixed results.

But a much more subtle process is at work, one that Bruno Latour has called mobilization. In the very process by which Ekman speculates about distinguishing "abusive from nonabusive caretakers," or by which other researchers have generated AIDS awareness programs, affective computing is currently being turned from what might well be described as constructed scientific narratives into things which are not so easily dismissed. Technologies like affective interactive interfaces may well be constructed artifacts of a culture, but they may then have profound effects on that culture which produced them.


Nature Loves a Vacuum
While it is clear that scientific measurement of the quarks and atoms often yields predictable, reprovable results, it is equally true that scientific measurement of the human often yields results which objectify the subject. The notion of mathematically reducing quarks, the simplest known things in the universe, is fairly straightforward. The notion of reducing humans, the most complex known things in the universe, is perhaps another issue.

But this is only part of the probes problematic. The history of facial analysis I've presented is a contextualist reading of science: An observer can note broader cultural influences from without science, echoed and impressed in science's findings and methodologies. In the case of phrenology, notions of race, sanity, difference, and vision were embodied into every scientific experiment, thesis, or policy. Certainly, current sciences of facial analysis echo the preoccupations of our culture. But there are other methods of critique: Gould, for instance, objects to phrenology also because the "science was bad," on a purely heuristic level. Precisely this opposition, these two types of critique form what might be called the dialectical dynamo which powers modern culture.

Problematizing scientific discovery with reference to power and social justice completely bypasses scientific power at the level at which that power is experienced. New technologies are felt on completely personal levels, experienced in highly anecdotal and subtle ways. Contextualist methods of interpretation, and their alternative, scientific methods, work together to obscure issues as complex as affective computing. Such issues tend to fall between these critiques, and lose their phenomenal, almost vital force. Bruno Latour traces the genesis of this dualist schism in his We Have Never Been Modern by comparing the disparate legacies of Hobbes' Leviathan and Boyle's vacuum pump.

Latour is not, per se, a contextualist, but rather regards science as being able to find remarkable, apparently vocal "facts" within the world.
As accustomed as we have become to the idea of a science that "constructs," "fashions," or "produces" its objects, the fact still remains that, after all the controversies, the sciences seem to have discovered a world that came into being without men and without sciences. Galileo may have constructed the phases of Venus, but once that construction was complete her phases appeared to have been "always already present." (Latour, 1996)

These things that wer "always already present" are natural objects -- subjects, devices, and meters -- which can account for the world, can make witness. Of course, this witness must be interpreted, so scientists must then account for, and make witness of the meters,
Yet the scientists declare that they themselves are not speaking; rather, facts speak for themselves. These mute entities are thus capable of speaking, writing, signifying within the artificial chamber of the laboratory or inside the even more rarefied chamber of the vacuum pump. Little groups of gentlemen take testimony from natural forces, and they testify to each other that they are not betraying but translating the silent behavior of objects. With Boyle and his successors, we begin to conceive of what a natural force is, an object that is mute but endowed or entrusted with meaning. (Latour, 1993)

This is the moment at which we find affective computing. Cognitive scientists must view the face as precisely such a natural object. To allow for social mediation of the face (or, at least, a mediation which cannot be filtered through covariance) would disallow its use as an object of study. One method by which individual faces may be made objective is to include many of them to form a statistic: Ekman's study claims to have found the emotions true across all cultures. The face, perhaps the one thing in the world which is not mute, is thus muted.

On the other hand, cultural and science studies will take such acts and do their usual routine, finding the contexts in which these scientists are working, and then trace social, power, and linguistic precedents to science's "found" nature. Sometimes this contextualization is almost too easy, as with recent "cross cultural" studies of attractiveness of women's faces, where investigators excitedly notice a high correlation between American and Japanese ideas of a beautiful mien. "Japanese and caucasian observers showed the same direction of preferences for the same facial composites, suggesting that aesthetic judgements of face shape are similar across differencet cultural backgrounds." (Perrett, May, and Yoshlkawa, 1994) Couldn't they have just watched a little Japanimation? But as the cultural critics do these analyses, they deny the phenomenal power which natural objects, the mute subjects, exert once mobilized. If not for the glowing filament, the chemical action of silver, how would Hollywood colonize foreign cultures? Many cultural critics would argue that the means of colonization, the technological basis of power, is arbitrary. Anyone involved in actively attempting to subvert these processes knows otherwise.

Through this "tag team" process of naturalizing and socializing, alternately mobilizing nature/technology and then denying its importance, modernism is able to "...[Conceive of] every hybrid as a mixture of two pure forms. The modern explanations consisted in splitting the mixtures apart in order to extract from them what came from the subject (or the social) and what came from the object. Next they multiplied the intermediaries in order to reconstruct the unity they had broken and wanted none the less to retrieve through blends of pure forms." (Latour, 1993) The latter reconstruction is the work of the engineers (and artists) who will build affective interfaces, which we will experience as new phenomena in our lived experience. These interfaces will approach us, through images of our faces, as objects, in much the same way that we will them. {http://www-white.media.mit.edu/vismod/demos/affect/jocelyn/tttpage.html}

Things like affective computing continue to be mobilized by slight of hand, where one side of culture (that which deals with objects) and the other (that which deals with the social) dissimulate each other's work. The things which slip effortlessly between these two camps, those monstrous hybrids which are neither natural nor social, are unmonitored and not understood, because each camp ignores much of their existence, acknowledging only the existence of their own camp. "The metamorphoses become explicable, on the contrary, if we redistribute essence to all the entities that make up history. But then they stop being simple, more or less faithful, intermediaries... The serfs have become free citizens once more." (Latour, 1993)

Freeing these technologies, acknowledging their inherent power, and recognizing their status as phenomena allows us to analyze them outside of strict historical context and without regard to their natural a priori status. Affective computing, like the face, becomes an elaborate, complex web of many influences, agents, and histories. It is a new technology, but it is also an old, remarkably human one, as related to fortune tellers as to silicon.

Affective interfaces are not simply a projection of cultural agendas or a successful model, but rather a phenomena which extends between the two. Where switching of modes allowed technologies to escape accountability, we may now ask questions about this technology we may not have before: Does the construction of a model with 9 emotions, while effectual, reduce the subject beyond recognizability? Are these 9 emotions really cross cultural, or have other media technologies extended western expressions of affect? and would technologies based on these affective models encourage homogeneity, in much the way that mass media technologies do? Would daily interaction with affective technologies change the way faces express emotions?

In the meantime, I will continue to watch for fanny packs.




Brigham, C. C. (1923). A Study of American Intelligence. Princeton, N.J., Princeton University Press.

D.I. Perrett, K. A. M., S. Yoshikawa (1994). ìFacial shape and judgements of female attractiveness.î Nature 368(17 March 1994): 239-242.

Dodge, C. (1997). The Abstracted Process: Providing for consistent metephors between content and computation in interactive media art. WRO Media Art Bienalle,

Ekman, P. (1993). Automating Facial Expression. National Science Foundation.

Gilman, S. L., Ed. (1976). The Face of Madness. New York, Brunner/Mazel, Inc.

Gilman, S. L. (1993). Imaging Hysteria: Science at the Fin de Siecle. Unpublished.

Gould, S. J. (1981). The Mismeasure of Man. New York, W. W. Norton & Company, Inc.

Harmon, L. D. (1973). ìThe Recognition of Faces.î Scientific American 229, 5(November):

Hirsh, S. K. (1992). Myers-Briggs Type Indicator Team Member's Guide. Palo Alto, CA, Consulting Psychologists Press, Inc. 27.

Latour, B. (1993). We Have Never Been Modern. Cambridge, MA, Harvard University Press.

Latour, B. (1996). Aramis or the love of technology. Cambridge, MA, Harvard University Press.

Radcliffe N. Salaman, M. D. (1912). ìHeredity and the Jew.î Eugenics Review 3: 190.

Roget, P. M. (1815). Cranioscopy. Encyclopoedia Britannica, 6th ed.