the Computers and Society AGORA agoo.ra G~.~ ~'.~i n. A marketplace in ancient Greece, customarily used as a place of popular assembly.For members of this l~opularassembly, we invite short, substantive communications tightly-focused around a single experience, issue or theoretical point within the scope of this publication. Programsthat Do What TheyThinkYou Want am disturbed by a recent outbreak of program or plans for programs that promise "systems that can be more friendly, useful and efficient". The quote is from "Perceptual Intelligence" by Alex Pentland (Comm ACM 43(3):35- 44, 2000); his is not the only such program, but I will concentrate on it, for expressing my worries. Pentland mentions, as his next sentence, "Or, of course, these new systems could be even more difficult to use than current systems; it depends how we design the interface between the world of humans and the world of this new generation of machines." Yes, they could. Pentland opens his article with "Inanimate things are coming to life. However, these stirrings are not Frankenstein or the humanoid robots dreamed of in artificial intelligence laboratories [Pentland is in the MIT Media Lab; is that noticeably different?]. The new awakening is more like Walt Disney: the simple objects that surround us are gaining sensors, computational powers, and actuators." Yeah, right--and what about those of us who don't like Disneylike objects? I liked Pooh before Disney bought him; but progress can't be stopped, can it? Now, let's think how many systems you've met that interpret things the way you are interpreting them, almost all the time. Start with simple tasks that we've all done, like creating a document. WordPerfect 4.7 had a lot of odd keystroke commands, but at least you could usually find out what it had done, and often you could find out how to make I it do what you intended, instead of what it had assumed. I lost touch with WordPerfect about that time, and switched to Word; as you know, Word (any version) thinks it knows what you want, and does that for you. The trouble is that if that's not what you want, it's nearly impossible to find out what its state is, or how to change it to what you'd like it to do. In my experience, this has affected minor things like indentation, spacing, hyphenation, bullets, and other secondary matters. But its inscrutability is famous, and if I really want the document to be formatted my way, I may have to back out, re-type a lot of stuff, and try to avoid Word's booby-traps. Annoying, but not fatal. My wife suggested other scenarios, for Pentland's "perceptually intelligent systems:" his smart rooms are said to recognize acquaintances, understand speech, and similar humanoid actions. And if my smart room in my perceptual house recognizes that our new neighbors have dropped in to visit, yet fails to realize that they came with shotguns and are starting a home invasion, how will I tell it to change its interpretation and call the police? Or if I conclude that these new neighbors are dangerous psychopaths, in spite of their apparently nice words, which initially fooled me, too? (Of course, I may be paranoid, but that doesn't mean that one of my new neighbors may not be a psychopath...) I have always read Risks-at the back of Comm ACM, and lately have subscribed to Risks Digest. Every issue of the Digest is full of anecdotes about systems that messed up--the smart, automatic toilet room in Tokyo which locked up and wouldn't let the user out; or the X-ray machine that sent lethal doses into patients it was supposed to be just imaging. O f course, these are old stories, about incompletely tested systems--or are they? What tools do we have, to check the trustworthiness of systems that have blended into out environment? What hope do we have that all the unobtrusive new systems will interpret the world pretty much as we do, especially in difficult and unusual situations? And what tools do we have, and how will they be deployed, to make reasonably sure that the interactions among countless systems, separately manufactured and installed (and differently maintained, if at all), will not develop some meta-systematic oscillations or pathologies--as so many less complicated systems, seem frequently to do? None of the articles in that special issue of Comm ACM mention any such problems, or any attempts to anticipate and perhaps avoid them. Do I just need to read other articles published by those same labs? Or are they assuming that, because they are nice guys and gals, and don't intend to make any errors, that all will be well, with all kinds of systems installed by all kinds of firms? Yeah, right. - D o n a l d O. Walter dwalter@pop,bol.ucla.edu Computersand Society, September 2000

/lp/association-for-computing-machinery/programs-that-do-what-they-think-you-want-1rGxAPPj7i