\documentclass[12pt,titlepage,twoside]{article}

\usepackage[verbose,
            a4paper,
            twosideshift=0mm]{geometry}

\title{Why the future doesn't need us\\\vspace{\baselineskip}
  \normalsize{Our most powerful  21st-century technologies --- robotics,
    genetic engineering, and nanotech --- are threatening to make humans
    an endangered species
    }
  } 

\author{Bill Joy\footnote{
    Bill  Joy, cofounder and  Chief Scientist  of Sun  Microsystems, was
    cochair of the presidential commission on the future of IT research,
    and  is coauthor  of \textsf{The  Java Language  Specification}. His
    work  on  the   \textsf{Jini  pervasive  computing  technology}  was
    featured in \textsf{Wired 6.08.}  
    }
  }

\date{Apr 2000\footnote{
    Wired Magazine, 8.04 (\texttt{http://www.wired.com});
    Copyright 1993-2000 The  Cond\'{e} Nast  Publications Inc.  All rights
    reserved;
    Copyright 1994-2000 Wired Digital, Inc. All rights reserved. 
    }
  }

\setlength{\parskip}{.75\baselineskip}

\begin{document}
\maketitle

\newpage

From the moment I became involved in the creation of new technologies, their 
ethical dimensions have concerned me, but it was only in the autumn of 
1998 that I became anxiously aware of how great are the dangers facing 
us in the 21st century. I can date the onset of my unease to the day I 
met Ray Kurzweil, the deservedly famous inventor of the first reading machine 
for the blind and many other amazing things.

Ray and I were both speakers at George Gilder's 
Telecosm conference, and I encountered him by chance in the bar of the 
hotel after both our sessions were over. I was sitting with John Searle, 
a Berkeley philosopher who studies consciousness. While we were talking, 
Ray approached and a conversation began, the subject of which haunts me 
to this day.

I had missed Ray's talk and the subsequent panel that Ray and John had 
been on, and they now picked right up where they'd left off, with Ray saying 
that the rate of improvement of technology was going to accelerate and that 
we were going to become robots or fuse with robots or something like that, 
and John countering 
that this couldn't happen, because the robots couldn't be conscious.

While I had heard such talk before, I had always 
felt sentient robots were in the realm of science fiction. But now, from 
someone I respected, I was hearing a strong argument that they were a near-term 
possibility. I was taken aback, especially given Ray's proven ability to 
imagine and create the future. I already knew that new technologies like 
genetic engineering and nanotechnology were giving us the power to remake 
the world, but a realistic and imminent scenario for intelligent robots 
surprised me.

It's easy to get jaded about such breakthroughs. 
We hear in the news almost every day of some kind 
of technological or scientific advance. Yet this was no ordinary prediction. 
In the hotel bar, Ray gave me a partial preprint of his then-forthcoming 
book {\it The Age of Spiritual Machines,} which outlined a utopia he foresaw 
--- one in which humans gained near immortality by becoming one with robotic 
technology. On reading it, my sense of unease only intensified; I felt sure 
he had to be understating the dangers, understating the probability of a 
bad outcome along this path.

I found myself most troubled by a passage detailing 
a {\it dys}topian scenario:

\section{The New Luddite Challenge} 


\begin{quote}
First let us postulate that the computer scientists succeed in developing 
intelligent machines that can do all things better than human beings can 
do them. In that case presumably 
all work will be done by vast, highly organized systems of machines and 
no human effort will be necessary. Either of two cases might occur. The 
machines might be permitted to make all of their own decisions without 
human oversight, or else human control over the machines might be retained. 


 
If the machines are permitted to make all their own decisions, we can't 
make any conjectures as to the results, because it is impossible to guess 
how such machines might behave. We only point out that the fate of the 
human race would be at the mercy of the machines. It might be argued that 
the human race would never be foolish enough to hand over all the power 
to the machines. But we are suggesting neither that the human race would 
voluntarily turn power over to the machines nor that the machines would 
willfully seize power. What we do suggest is that the human race might 
easily permit itself to drift into a position of such dependence on the 
machines that it would have no practical choice but to accept all of the 
machines' decisions. As society and the problems that face it become more 
and more complex and machines become more and more intelligent, people 
will let machines make more of their decisions for them, simply because 
machine-made decisions will bring better results than man-made ones. Eventually 
a stage may be reached at which the decisions necessary to keep the system 
running will be so complex that human beings will be incapable of making 
them intelligently. At that stage the machines will be in effective control. 
People won't be able to just turn the machines off, because they will be so 
dependent on them that turning them off would amount to suicide. 


 
On the other hand it is possible that human control over the machines may 
be retained. In that case the average man may have control over certain 
private machines of his own, such as his car or his personal computer, 
but control over large systems of machines will be in the hands of a tiny 
elite --- just as it is today, but with two differences. Due to improved 
techniques the elite will have greater control over the masses; and because 
human work will no longer be necessary the masses will be superfluous, 
a useless burden on the system. If the elite is 
ruthless they may simply decide to exterminate the mass of humanity. If 
they are humane they may use propaganda or other psychological or biological 
techniques to reduce the birth rate until the mass of humanity becomes 
extinct, leaving the world to the elite. Or, if the elite consists of 
soft-hearted 
liberals, they may decide to play the role of good shepherds to the rest 
of the human race. They will see to it that everyone's physical needs are 
satisfied, that all children are raised under psychologically hygienic 
conditions, that everyone has a wholesome hobby to keep him busy, and that 
anyone who 
may become dissatisfied undergoes "treatment" to cure his "problem." Of 
course, life will be so purposeless that people 
will have to be biologically or psychologically engineered either to remove 
their need for the power process or make them "sublimate" their drive for 
power into some harmless hobby. These engineered human beings may be happy 
in such 
a society, but they will most certainly not be free. They will have been 
reduced to the status of domestic animals\footnote{The passage Kurzweil
  quotes is  from Kaczynski's  Unabomber Manifesto, which  was published
  jointly, under duress, by {\it The New York Times} and 
  {\it The Washington Post} to attempt to bring his campaign of terror to an 
  end. I agree with David Gelernter, who said about their decision: 
  \begin{quote}
  "It was a tough call for the newspapers. To say yes would be giving in 
  to terrorism, and for all they knew he was lying anyway. On the other hand, 
  to say yes might stop the killing. There was also a chance that someone 
  would read the tract and get a hunch about the author; and that is exactly 
  what happened. The suspect's brother read it, and it rang a bell. 

  "I would have told them not to publish. I'm glad they didn't ask me. I 
  guess." 
  \end{quote}
  ({\it Drawing Life: Surviving the Unabomber.} Free Press, 1997: 120.) 
  }.
\end{quote}

  In the book, you don't discover until you turn the page that the
  author 
  of this passage is Theodore Kaczynski --- the Unabomber. I am no apologist 
  for Kaczynski. His bombs killed three people during a 
  17-year terror campaign and wounded many others. One of his bombs gravely 
  injured my friend David Gelernter, one of the most brilliant and visionary 
  computer scientists of our time. Like many of my colleagues, I felt that 
  I could easily have been the Unabomber's next target.
  
  Kaczynski's actions were murderous and, in my view, criminally insane. 
  He is clearly a Luddite, but simply saying this does not dismiss his argument; 
  as difficult as it is for me to acknowledge, I saw some merit in the reasoning 
  in this single passage. I felt compelled to confront it.

  Kaczynski's dystopian vision describes unintended consequences, a well-known 
  problem with the design and use of technology, and one that is clearly 
  related 
  to Murphy's law --- "Anything that can go wrong, will." (Actually, this is 
  Finagle's law, which in itself shows that Finagle was right.) Our overuse 
  of antibiotics has led to what may be the biggest such problem so far: 
  the emergence of antibiotic-resistant and much more dangerous bacteria. 
  Similar things happened when attempts to eliminate malarial mosquitoes 
  using DDT caused them to acquire DDT resistance; malarial parasites likewise 
  acquired multi-drug-resistant genes.\footnote{Garrett, Laurie. {\it The
  Coming Plague:  Newly Emerging  Diseases in a  World Out  of Balance.}
  Penguin, 1994: 47-52, 414, 419, 452.  
  }

  The cause of many such surprises seems clear: The systems involved are 
  complex, involving interaction among and feedback between many parts. Any 
  changes to such a system will cascade in ways that are difficult to predict; 
  this is especially true when human actions are involved.
  
  I started showing friends the Kaczynski quote from {\it The Age of Spiritual 
    Machines;} I would hand them Kurzweil's book, let them read the quote, and 
  then watch their reaction as they discovered who had written it. At around 
  the same time, I found Hans Moravec's book {\it Robot: Mere Machine to 
    Transcendent Mind.} Moravec is one of the leaders in robotics research, 
  and was a founder of the world's largest robotics research program, at 
  Carnegie Mellon University. {\it Robot} gave 
  me more material to try out on my friends --- material surprisingly supportive 
  of Kaczynski's argument. 
  For example:


\section{The Short Run (Early 2000s)} 

\begin{quote}
Biological species almost never survive encounters with superior competitors. 
Ten million years ago, South and North America were separated by a sunken 
Panama isthmus. South America, like Australia today, was populated by marsupial 
mammals, including pouched equivalents of rats, deers, and tigers. When 
the isthmus connecting North and South America rose, it took only a few 
thousand years for the northern placental species, with slightly more effective 
metabolisms and reproductive and nervous systems, to displace and eliminate 
almost all the southern marsupials. 


 
In a completely free marketplace, superior robots would surely affect humans 
as North American placentals affected South American marsupials (and as 
humans have affected countless species). Robotic industries would compete 
vigorously among themselves for matter, energy, and space, incidentally 
driving their price beyond human reach. Unable to afford the necessities 
of life, biological humans would be squeezed out of existence. 


 
There is probably some breathing room, because we do not live in a completely 
free marketplace. Government coerces nonmarket behavior, especially by 
collecting taxes. Judiciously applied, governmental coercion could support 
human populations in high style on the fruits of robot labor, perhaps for 
a long while. 
\end{quote}


 A textbook dystopia --- and Moravec is just getting wound up. He goes on 
to discuss how our main job in the 21st century will be "ensuring continued 
cooperation from the robot industries" by passing laws decreeing that they 
be "nice,"\footnote{Isaac Asimov described what became the most famous view of ethical rules 
for robot behavior in his book {\it I, Robot} in 1950, in 
his Three Laws of Robotics: 1. A robot may not injure a human being, or, 
through inaction, allow a human being to come to harm. 2. A robot must 
obey the orders given it by human beings, except where 
such orders would conflict with the First Law. 
3. A robot must protect its own existence, as long as such protection does 
not conflict with the First or Second Law. 
} and to describe how seriously dangerous a 
human can be "once transformed into 
an unbounded superintelligent robot." Moravec's view is that the robots 
will eventually succeed us --- that humans clearly face extinction.

I decided it was time to talk to my friend Danny Hillis. Danny became famous 
as the cofounder of Thinking Machines Corporation, which built a very powerful 
parallel supercomputer. Despite my current job title of Chief Scientist 
at Sun Microsystems, I am more a computer architect than a scientist, and 
I respect Danny's knowledge of the information and physical sciences more 
than that of any other single person I know. Danny is also a highly regarded 
futurist who thinks long-term --- four years ago he started the Long Now 
Foundation, which is building a clock designed to last 10,000 years, in 
an attempt to draw attention to the pitifully short attention span of our 
society. (See "Test of Time," {\it Wired} 8.03, page 78.) 

So I flew to Los Angeles for the express purpose of having dinner with 
Danny and his wife, Pati. I went through my now-familiar routine, trotting 
out the ideas and passages that I found so disturbing. Danny's answer --- 
directed specifically at Kurzweil's scenario of humans merging with robots 
--- came swiftly, and quite surprised me. He said, simply, that the changes 
would come gradually, and that we would get used to them.

But I guess I wasn't totally surprised. I had seen a quote from Danny in 
Kurzweil's book in which he said, "I'm as fond of my body as anyone, but 
if I can be 200 with a body of silicon, I'll take it." It seemed that he 
was at peace with this process and its attendant risks, while I was not.

While talking and thinking about Kurzweil, Kaczynski, and Moravec, I suddenly 
remembered a novel I had read almost 20 years ago --- {\it The White Plague,} by 
Frank Herbert --- in which a molecular biologist is driven insane by the 
senseless murder of his family. To seek revenge he constructs and disseminates 
a new and highly contagious plague that kills widely but selectively. (We're 
lucky Kaczynski was a mathematician, not a molecular biologist.) I was 
also reminded of the Borg of {\it Star Trek,} a hive of partly biological, 
partly robotic creatures with a strong destructive streak. 
Borg-like disasters are a staple of science fiction, so 
why hadn't I been more concerned about such robotic dystopias earlier? 
Why weren't other people more concerned about these nightmarish scenarios?

Part of the answer certainly lies in our attitude toward the new --- in our 
bias toward instant familiarity and unquestioning acceptance. Accustomed 
to living with almost routine scientific breakthroughs, we have yet to 
come to terms with the fact that the most compelling 21st-century technologies 
--- robotics, genetic engineering, and nanotechnology --- pose a different 
threat than the technologies that have come before. Specifically, robots, 
engineered organisms, and nanobots share a dangerous amplifying factor: 
They can self-replicate. A bomb is blown up only once --- but one bot can 
become many, and quickly get out of control.

Much of my work over the past 25 years has been on computer networking, 
where the sending and receiving of messages creates the opportunity for 
out-of-control replication. But while replication in a computer or a computer 
network can be a nuisance, at worst it disables a machine or takes down 
a network or network service. Uncontrolled self-replication in these newer 
technologies runs a much greater risk: a risk of substantial damage in 
the physical world.

Each of these technologies also offers untold promise: The vision of near 
immortality that Kurzweil sees in his robot dreams drives us forward; genetic 
engineering may soon provide treatments, if not outright cures, for most 
diseases; and nanotechnology and nanomedicine can address yet more ills. 
Together they could significantly extend our average life span and improve 
the quality of our lives. Yet, with each of these technologies, a sequence 
of small, individually sensible advances leads to an accumulation of great 
power and, concomitantly, great danger.

What was different in the 20th century? Certainly, the technologies underlying 
the weapons of mass destruction (WMD) --- nuclear, biological, and chemical 
(NBC) --- were powerful, and the weapons an enormous threat. But building 
nuclear weapons required, at least for a time, access to both rare --- indeed, 
effectively unavailable --- raw materials and highly protected information; 
biological and chemical weapons programs also tended to require large-scale 
activities.

The 21st-century technologies --- genetics, nanotechnology, and robotics 
(GNR) --- are so powerful that they can spawn whole new classes of accidents 
and abuses. Most dangerously, for the first time, these accidents and abuses 
are widely within the reach of individuals or small groups. They will not 
require large facilities or rare raw materials. Knowledge alone will enable 
the use of them.

Thus we have the possibility not just of weapons of mass destruction but 
of knowledge-enabled mass destruction (KMD), this destructiveness hugely 
amplified by the power of self-replication.

I think it is no exaggeration to say we are on the cusp of the further 
perfection of extreme evil, an evil whose possibility spreads well beyond 
that which weapons of mass destruction bequeathed to the nation-states, 
on to a surprising and terrible empowerment of extreme individuals.




Nothing about the way I got involved with computers suggested to me that 
I was going to be facing these kinds of issues.

My life has been driven by a deep need to ask questions and find answers. 
When I was 3, I was already reading, so my father took me to the elementary 
school, where I sat on the principal's lap and read him a 
story. I started school early, later skipped a grade, and escaped into 
books --- I was incredibly motivated to learn. I asked lots of questions, 
often driving adults to distraction.

As a teenager I was very interested in science and technology. I wanted 
to be a ham radio operator but didn't have the money to buy the equipment. 
Ham radio was the Internet of its time: very addictive, and quite solitary. 
Money issues aside, my mother put her foot down --- I was not to be a ham; 
I was antisocial enough already.

I may not have had many close friends, but I was awash in ideas. By high 
school, I had discovered the great science fiction writers. I remember 
especially Heinlein's {\it Have Spacesuit Will Travel} and Asimov's 
{\it I, Robot,} with its Three Laws of Robotics. I was enchanted by the 
descriptions of space travel, and wanted to have a telescope to look at 
the stars; since I had no money to buy or make one, I checked books on 
telescope-making out of the library and read about making them instead. 
I soared in my imagination.

Thursday nights my parents went bowling, and we 
kids stayed home alone. It was the night of Gene Roddenberry's original 
{\it Star Trek,} and the program made a big impression on me. I came to accept 
its notion that humans had a future in space, Western-style, with big heroes 
and adventures. Roddenberry's vision of the centuries to come was one with 
strong moral values, embodied in codes like the Prime Directive: to not 
interfere in the development of less technologically advanced civilizations. 
This had an incredible appeal to me; ethical humans, not robots, dominated 
this future, and I took Roddenberry's dream as part of my own.

I excelled in mathematics in high school, and when 
I went to the University of Michigan as an undergraduate engineering student 
I took the advanced curriculum of the mathematics majors. Solving math 
problems was an exciting challenge, but when I discovered computers I found 
something much more interesting: a machine into which you could put a program 
that attempted to solve a problem, after which the machine quickly checked 
the solution. The computer had a clear notion of correct and incorrect, 
true and false. Were my ideas correct? The machine could tell me. This 
was very seductive.

I was lucky enough to get a job programming early supercomputers and discovered 
the amazing power of large machines to numerically simulate advanced designs. 
When I went to graduate school at UC Berkeley in the mid-1970s, I started 
staying up late, often all night, inventing new worlds inside the machines. 
Solving problems. Writing the code that argued so strongly to be written.

In {\it The Agony and the Ecstasy,} Irving Stone's biographical novel of 
Michelangelo, 
Stone described vividly how Michelangelo released the statues from the 
stone, "breaking the marble spell," carving from the images in his mind\footnote{Michelangelo wrote a sonnet that begins: 




    \begin{verse}
      Non ha l' ottimo artista alcun concetto\\
      Ch' un marmo solo in s\`{e} non circonscriva\\
      Col suo soverchio; e solo a quello arriva\\
      La man che ubbidisce all' intelleto. 
    \end{verse}

Stone translates this as: 

    \begin{verse}
      The best of artists hath no thought to show\\
      which the rough stone in its superfluous shell\\
      doth not include; to break the marble spell\\
      is all the hand that serves the brain can do. 
    \end{verse}


 
Stone describes the process: "He was not working from his drawings or clay 
models; they had all been put away. He was carving from the images in his 
mind. His eyes and hands knew where every line, curve, mass must emerge, 
and at what depth in the heart of the stone to create the low relief." 


 
({\it The Agony and the Ecstasy.} Doubleday, 1961: 6, 144.) 

}.
In my most ecstatic moments, 
the software in the computer emerged in the same way. Once I had imagined 
it in my mind I felt that it was already there in the machine, waiting 
to be released. Staying up all night seemed a small price to pay to free 
it --- to give the ideas concrete form.

After a few years at Berkeley I started to send out some of the software 
I had written --- an instructional Pascal system, Unix utilities, and a text 
editor called vi (which is still, to my surprise, widely used more than 
20 years later) --- to others who had similar small PDP-11 and VAX minicomputers. 
These adventures in software eventually turned into the Berkeley version 
of the Unix operating system, which became a personal "success disaster" 
--- so many people wanted it that I never finished my PhD. Instead I got 
a job working for Darpa putting Berkeley Unix on the Internet and fixing 
it to be reliable and to run large research applications well. This was 
all great fun and very rewarding. And, frankly, I saw no robots here, or 
anywhere near.

Still, by the early 1980s, I was drowning. The Unix releases were very 
successful, and my little project of one soon had money and some staff, 
but the problem 
at Berkeley was always office space rather than money --- there wasn't room 
for the help the project needed, 
so when the other founders of Sun Microsystems showed up I jumped at the 
chance to join them. At 
Sun, the long hours continued into the early days of workstations and personal 
computers, and I have enjoyed participating in the creation of advanced 
microprocessor technologies and Internet technologies such as Java and 
Jini.

From all this, I trust it is clear that I am not a Luddite. I have always, 
rather, had a strong belief in the value of the scientific search for truth 
and in the ability of great engineering to bring material progress. The 
Industrial Revolution has immeasurably improved everyone's life over the 
last couple hundred years, and I always expected my career to involve the 
building of worthwhile solutions to real problems, one problem at a time.

I have not been disappointed. My work has had more impact than I had ever 
hoped for and has been more widely used than I could have reasonably expected. 
I have spent the last 20 years still trying to figure out how to make computers 
as reliable as I want them to be (they are not nearly there yet) and how 
to make them simple to use (a goal that has met with even less relative 
success). Despite some progress, the problems that remain seem even more 
daunting.

But while I was aware of the moral dilemmas surrounding technology's 
consequences 
in fields like weapons research, I did not expect that I would confront 
such issues in my own field, or at least not so soon.




Perhaps it is always hard to see the bigger impact while you are in the 
vortex of a change. Failing to understand the consequences of our inventions 
while we are in the rapture of discovery and innovation seems to be a common 
fault of scientists and technologists; we have long been driven by the 
overarching desire to know that is the nature of science's quest, not stopping 
to notice that the progress to newer and more powerful technologies can 
take on a life of its own.

I have long realized that the big advances in information technology come 
not from the work of computer scientists, computer architects, or electrical 
engineers, but from that of physical scientists. The physicists Stephen 
Wolfram and Brosl Hasslacher introduced me, in the early 1980s, to chaos 
theory and nonlinear systems. In the 1990s, I learned about complex systems 
from conversations with Danny Hillis, the biologist Stuart Kauffman, the 
Nobel-laureate physicist Murray Gell-Mann, and others. Most recently, 
Hasslacher 
and the electrical engineer and device physicist Mark Reed have been giving 
me insight into the incredible possibilities of molecular electronics.

In my own work, as codesigner of three microprocessor architectures --- SPARC, 
picoJava, and MAJC --- and as the designer of several implementations thereof, 
I've been afforded a deep and firsthand acquaintance with Moore's law. 
For decades, Moore's law has correctly predicted the exponential rate of 
improvement of semiconductor technology. Until last year I believed that 
the rate of advances predicted by Moore's law might continue only until 
roughly 2010, when some physical limits would begin to be reached. It was 
not obvious to me that a new technology would arrive in time to keep 
performance advancing smoothly.

But because of the recent rapid and radical progress in molecular electronics 
--- where individual atoms and molecules replace lithographically drawn 
transistors 
--- and related nanoscale technologies, we should be able to meet or exceed 
the Moore's law rate of progress for another 30 years. By 2030, we are 
likely to be able to build machines, in quantity, a million times as powerful 
as the personal computers of today --- sufficient to implement the dreams 
of Kurzweil and Moravec.

As this enormous computing power is combined with the manipulative advances 
of the physical sciences 
and the new, deep understandings in genetics, enormous transformative power 
is being unleashed. These combinations open up the opportunity to completely 
redesign the world, for better or worse: The replicating and evolving processes 
that have been confined to the natural world are about to become realms 
of human endeavor.

In designing software and microprocessors, I have never had the feeling 
that I was designing an intelligent machine. The software and hardware 
is so fragile and the capabilities of the machine to "think" so 
clearly absent that, even as a possibility, this has always seemed very 
far in the future.

But now, with the prospect of human-level computing power in about 30 years, 
a new idea suggests itself: that I may be working to create tools which 
will enable the construction of the technology that may replace our species. 
How do I feel about this? Very uncomfortable. Having struggled my entire 
career to build reliable software systems, it seems to me more than likely 
that this future will not work out as well as some people may imagine. 
My personal experience suggests we tend to overestimate our design abilities.

Given the incredible power of these new technologies, shouldn't we be asking 
how we can best coexist with them? And if our own extinction is a likely, 
or even possible, outcome of our technological development, shouldn't we 
proceed with great caution?




The dream of robotics is, first, that intelligent machines 
can do our work for us, allowing us lives of leisure, restoring us to Eden. 
Yet in his history of such ideas, {\it Darwin Among the Machines,} George Dyson 
warns: "In the game of life and evolution there are three players at the 
table: human beings, nature, and machines. 
I am firmly on the side of nature. But nature, I suspect, is on the side 
of the machines." As we have seen, Moravec agrees, believing we may well 
not survive the encounter with the superior robot species.

How soon could such an intelligent robot be built? The coming advances 
in computing power seem to make it possible by 2030. And once an intelligent 
robot exists, it is only a small step to a robot species --- to an intelligent 
robot that can make evolved copies of itself.

A second dream of robotics is that we will gradually replace ourselves 
with our robotic technology, achieving near immortality by downloading 
our consciousnesses; it is this process that Danny Hillis thinks we will 
gradually get used to and that Ray Kurzweil elegantly details in {\it The Age 
of Spiritual Machines.} (We are beginning to see intimations of this in 
the implantation of computer devices into the human body, as illustrated 
on thecover of {\it Wired} 8.02.)

But if we are downloaded into our technology, what are the chances that 
we will thereafter be ourselves or even human? It seems to me far more 
likely that a robotic existence would not be like a human one in any sense 
that we understand, that the robots would in no sense be our children, 
that on this path our humanity may well be lost.

Genetic engineering promises to revolutionize agriculture by increasing 
crop yields while reducing the use of pesticides; to create tens of thousands 
of novel species of bacteria, plants, viruses, and animals; to replace 
reproduction, or supplement it, with cloning; to create cures for many 
diseases, increasing our life span and our quality of life; and much, much 
more. 
We now know with certainty that these profound changes in the biological 
sciences are imminent and will challenge all our notions of what life is.

Technologies such as human cloning have in particular raised our awareness 
of the profound ethical and moral issues we face. If, for example, we were 
to reengineer ourselves into several separate and unequal species using 
the power of genetic engineering, then we would threaten the notion of 
equality that is the very cornerstone of our democracy.

Given the incredible power of genetic engineering, it's no surprise that 
there are significant safety issues 
in its use. My friend Amory Lovins recently cowrote, along with Hunter 
Lovins, an editorial that provides 
an ecological view of some of these dangers. Among their concerns: that 
"the new botany aligns the development of plants with their economic, not 
evolutionary, success." 
(See "A Tale of Two Botanies," page 247.)
Amory's 
long career has been focused on energy and resource efficiency by taking 
a whole-system view of human-made systems; such a whole-system view often 
finds simple, smart solutions to otherwise seemingly difficult problems, 
and is usefully applied here as well.

After reading the Lovins' editorial, I saw an op-ed by Gregg Easterbrook 
in {\it The New York Times} (November 19, 1999) about genetically engineered 
crops, under the headline: "Food for the Future: Someday, rice will have 
built-in vitamin A. Unless the Luddites win."

Are Amory and Hunter Lovins Luddites? Certainly not. I believe we all would 
agree that golden rice, with its built-in vitamin A, is probably a good 
thing, if developed with proper care and respect for the likely dangers 
in moving genes across species boundaries.

Awareness of the dangers inherent in genetic engineering is beginning to 
grow, as reflected in the Lovins' editorial. The general public is aware 
of, and uneasy about, genetically modified foods, and seems to be rejecting 
the notion that such foods should be permitted to be unlabeled.

But genetic engineering technology is already very far along. As the Lovins 
note, the USDA has already approved about 50 genetically engineered crops 
for unlimited release; more than half of the world's soybeans and a third 
of its corn now contain genes spliced in from other forms of life.

While there are many important issues here, my own major concern with genetic 
engineering is narrower: that it gives the power --- whether militarily, 
accidentally, or in a deliberate terrorist act --- to create a White Plague.

The many wonders of nanotechnology were first imagined by the Nobel-laureate 
physicist Richard Feynman in a speech he gave in 1959, subsequently published 
under the title "There's Plenty of Room at the Bottom." The book that made 
a big impression on me, in the mid-'80s, was Eric Drexler's {\it Engines of 
Creation,} in which he described beautifully how manipulation of matter 
at the atomic level could create a utopian future of abundance, where just 
about everything could be made cheaply, and almost any imaginable disease 
or physical problem could be solved using nanotechnology and artificial 
intelligences.

A subsequent book, {\it Unbounding the Future: The Nanotechnology Revolution,} 
which Drexler cowrote, imagines some of the changes that might take place 
in a world where we had molecular-level "assemblers." Assemblers could 
make possible incredibly low-cost solar power, cures for cancer and the 
common cold by augmentation of the human immune system, essentially complete 
cleanup of the environment, incredibly inexpensive pocket supercomputers 
--- in fact, any product would be manufacturable by assemblers at a cost 
no greater than that of wood --- spaceflight more accessible than transoceanic 
travel today, and restoration of extinct species.

I remember feeling good about nanotechnology after reading {\it Engines of 
Creation.} As a technologist, it gave me a sense of calm --- that is, 
nanotechnology 
showed us that incredible progress was possible, and indeed perhaps inevitable. 
If nanotechnology was our future, then I didn't feel pressed to solve so 
many problems in the present. I would get to Drexler's utopian future in 
due time; I might as well enjoy life more in the here and now. It didn't 
make sense, given his vision, to stay up all night, all the time.

Drexler's vision also led to a lot of good fun. I would occasionally get 
to describe the wonders of nanotechnology to others who had not heard of 
it. After teasing them with all the things Drexler described 
I would give a homework assignment of my own: "Use nanotechnology to create 
a vampire; for extra credit create an antidote."

With these wonders came clear dangers, of which 
I was acutely aware. As I said at a nanotechnology 
conference in 1989, "We can't simply do our science and not worry about 
these ethical issues"\footnote{First Foresight Conference on Nanotechnology in October 1989, a talk 
titled "The Future of Computation." Published in Crandall, B. C. and James 
Lewis, editors. {\it Nanotechnology: Research and Perspectives.} MIT Press, 
1992: 269. 
See also \texttt{www.foresight.org/Conferences/MNT01/Nano1.html}. 
}. But my 
subsequent conversations with physicists convinced 
me that nanotechnology might not even work --- or, 
at least, it wouldn't work anytime soon. Shortly thereafter I moved to 
Colorado, to a skunk works I had set up, and the focus of my work shifted 
to software for the Internet, specifically on ideas that became Java 
and Jini.

Then, last summer, Brosl Hasslacher told me that nanoscale molecular 
electronics 
was now practical. This was {\it new} news, at least to me, and I think to many 
people --- and it radically changed my opinion about nanotechnology. It sent 
me back to {\it Engines of Creation.} Rereading Drexler's work after more than 
10 years, I was dismayed to realize how little I had remembered of its 
lengthy section called "Dangers and Hopes," including a discussion of how 
nanotechnologies can become "engines of destruction." Indeed, in my rereading 
of this cautionary material today, I am struck by how naive some of Drexler's 
safeguard proposals seem, and how much greater I judge the dangers to be 
now than even he seemed to then. (Having anticipated and described many 
technical and political problems with nanotechnology, Drexler started the 
Foresight Institute in the late 1980s "to help prepare society for anticipated 
advanced technologies" --- most important, nanotechnology.)

The enabling breakthrough to assemblers seems quite likely within the next 
20 years. Molecular electronics --- the new subfield of nanotechnology where 
individual molecules are circuit elements --- should mature quickly and become 
enormously lucrative within this decade, causing a large incremental investment 
in all nanotechnologies.

Unfortunately, as with nuclear technology, it is far easier to create 
destructive 
uses for nanotechnology than constructive ones. Nanotechnology has clear 
military and terrorist uses, and you need not be suicidal to release a 
massively destructive nanotechnological device --- such devices can be built 
to be selectively destructive, affecting, for example, only a certain 
geographical area or a group of people who are genetically distinct.

An immediate consequence of the Faustian bargain in obtaining the great 
power of nanotechnology is that we run a grave risk --- the risk that we 
might destroy the biosphere on which all life depends.

As Drexler explained:


 
"Plants" with "leaves" no more efficient than today's solar 
cells could out-compete real plants, crowding the biosphere with an inedible 
foliage. Tough omnivorous "bacteria" could out-compete real bacteria: They 
could spread like blowing pollen, replicate swiftly, and reduce the biosphere 
to dust in 
a matter of days. Dangerous replicators could easily be too tough, small, 
and rapidly spreading to stop --- at least if we make no preparation. We 
have trouble enough controlling viruses and fruit flies. 


 
Among the cognoscenti of nanotechnology, this threat has become known as 
the "gray goo problem." Though masses of uncontrolled replicators need 
not be gray or gooey, the term "gray goo" emphasizes that replicators able 
to obliterate life might be less inspiring than a single species of crabgrass. 
They might be superior in an evolutionary sense, but this need not make 
them valuable. 


 
The gray goo threat makes one thing perfectly clear: 
We cannot afford certain kinds of accidents with replicating assemblers. 


 Gray goo would surely be a depressing ending to our human adventure on 
Earth, far worse than mere fire or ice, and one that could stem from a 
simple laboratory accident\footnote{In his 1963 novel {\it Cat's Cradle,} Kurt Vonnegut imagined a gray-goo-like 
accident where a form of ice called ice-nine, which becomes solid at a 
much higher temperature, freezes the oceans.
}. Oops.




It is most of all the power of destructive self-replication in genetics, 
nanotechnology, and robotics (GNR) that should give us pause. Self-replication 
is the modus operandi of genetic engineering, which uses the machinery 
of the cell to replicate its designs, and the prime danger underlying gray 
goo in nanotechnology. Stories of run-amok robots like the Borg, replicating 
or mutating to escape from the ethical constraints imposed on them by their 
creators, are well established in our science fiction books and movies. 
It is even possible that self-replication may be more fundamental than 
we thought, and hence harder --- or even impossible --- to control. A recent 
article by Stuart Kauffman in {\it Nature} titled "Self-Replication: Even 
Peptides 
Do It" discusses the discovery that a 32-amino-acid peptide can "autocatalyse 
its own synthesis." We don't know how widespread this ability is, but Kauffman 
notes that it may hint at "a route to self-reproducing molecular systems 
on a basis far wider than Watson-Crick base-pairing."\footnote{Kauffman, Stuart. 
"Self-replication: Even Peptides Do It." Nature, 382, August 8, 1996: 496. 
See \texttt{www.santafe.edu/sfi/People/kauffman/sak-peptides.html}. 
}

In truth, we have had in hand for years clear warnings of the dangers inherent 
in widespread knowledge of GNR technologies --- of the possibility of knowledge 
alone enabling mass destruction. But these warnings haven't been widely 
publicized; the public discussions have been clearly inadequate. There 
is no profit in publicizing the dangers.

The nuclear, biological, and chemical (NBC) technologies used in 20th-century 
weapons of mass destruction were and are largely military, developed in 
government laboratories. In sharp contrast, the 21st-century GNR technologies 
have clear commercial uses and are being developed almost exclusively by 
corporate enterprises. In this age of triumphant commercialism, technology 
--- with science as its handmaiden --- is delivering a series of almost magical 
inventions that are the most phenomenally lucrative ever seen. We are 
aggressively pursuing the promises of these new technologies within the 
now-unchallenged system of global capitalism and its manifold financial 
incentives and competitive pressures.





 
This is the first moment in the history of our planet when any species, 
by its own voluntary actions, has become a danger to itself --- as well as 
to vast numbers of others. 


 
It might be a familiar progression, transpiring on many worlds --- a planet, 
newly formed, placidly revolves around its star; life slowly forms; a 
kaleidoscopic 
procession of creatures evolves; intelligence emerges which, at least up 
to a point, confers enormous survival value; and then technology is invented. 
It dawns on them that there are such things as laws of Nature, that these 
laws can be revealed by experiment, and that knowledge of these laws can 
be made both to save and to take lives, both on unprecedented scales. Science, 
they recognize, grants immense powers. In a flash, they create world-altering 
contrivances. Some planetary civilizations see their way through, place 
limits on what may and what must not be done, and safely pass through the 
time of perils. Others, not so lucky or so prudent, perish. 


That is Carl Sagan, writing in 1994, in {\it Pale Blue Dot,} a book describing 
his vision of the human future in space. I am only now realizing how deep 
his insight was, and how sorely I miss, and will miss, his voice. For all 
its eloquence, Sagan's contribution was not least that of simple common 
sense --- an attribute that, along with humility, many of the leading advocates 
of the 21st-century technologies seem to lack.

I remember from my childhood that my grandmother was strongly against the 
overuse of antibiotics. She had worked since before the first World War 
as a nurse and had a commonsense attitude that taking antibiotics, unless 
they were absolutely necessary, was bad for you.

It is not that she was an enemy of progress. She saw much progress in an 
almost 70-year nursing career; 
my grandfather, a diabetic, benefited greatly from the improved treatments 
that became available in his lifetime. But she, like many levelheaded people, 
would probably think it greatly arrogant for us, now, to be designing a 
robotic "replacement species," when we obviously have so much trouble making 
relatively simple things work, and so much trouble managing --- or even 
understanding --- ourselves.

I realize now that she had an awareness of the nature of the order of life, 
and of the necessity of living with and respecting that order. With this 
respect comes a necessary humility that we, with our early-21st-century 
chutzpah, lack at our peril. The commonsense view, grounded in this respect, 
is often right, in advance of the scientific evidence. The clear fragility 
and inefficiencies of the human-made systems we have built should give 
us all pause; the fragility of the systems I have worked on certainly humbles 
me.

We should have learned a lesson from the making of the first atomic bomb 
and the resulting arms race. We didn't do well then, and the parallels 
to our current situation are troubling.

The effort to build the first atomic bomb was led by the brilliant physicist 
J. Robert Oppenheimer. Oppenheimer was not naturally interested in politics 
but became painfully aware of what he perceived as the grave threat to 
Western civilization from the Third Reich, a threat surely grave because 
of the possibility that Hitler might obtain nuclear weapons. Energized 
by this concern, he brought his strong intellect, passion for physics, 
and charismatic leadership skills to Los Alamos and led a rapid and successful 
effort by an incredible collection of great minds to quickly invent the 
bomb.

What is striking is how this effort continued so naturally after the initial 
impetus was removed. In a meeting shortly after V-E Day with some physicists 
who felt that perhaps the effort should stop, Oppenheimer argued to continue. 
His stated reason seems a bit strange: not because of the fear of large 
casualties from an invasion of Japan, but because the United Nations, which 
was soon to be formed, should have foreknowledge of atomic weapons. A more 
likely reason the project continued is the momentum that had built up --- 
the first atomic test, Trinity, was nearly at hand.

We know that in preparing this first atomic test the physicists proceeded 
despite a large number of possible dangers. They were initially worried, 
based on a calculation by Edward Teller, that an atomic explosion might 
set fire to the atmosphere. A revised calculation reduced the danger of 
destroying the world to a three-in-a-million chance. (Teller says he was 
later able to dismiss the prospect of atmospheric ignition entirely.) 
Oppenheimer, 
though, was sufficiently concerned about the result of Trinity that he 
arranged for a possible evacuation of the southwest part of the state of 
New Mexico. And, of course, there was the clear danger of starting a nuclear 
arms race.

Within a month of that first, successful test, two atomic bombs destroyed 
Hiroshima and Nagasaki. Some scientists had suggested that the bomb simply 
be demonstrated, rather than dropped on Japanese cities --- saying that this 
would greatly improve the chances for arms control after the war --- but 
to no avail. With the tragedy of Pearl Harbor still fresh in Americans' 
minds, it would have been very difficult for President Truman to order 
a demonstration of the weapons rather than use them as he did --- the desire 
to quickly end the war and save the lives that would have been lost in 
any invasion of Japan was very strong. Yet the overriding truth was probably 
very simple: As the physicist Freeman Dyson later said, "The reason that 
it was dropped was just that nobody had the courage or the foresight 
to say no."

It's important to realize how shocked the physicists were in the aftermath 
of the bombing of Hiroshima, on August 6, 1945. They describe a series 
of waves of emotion: first, a sense of fulfillment that the bomb worked, 
then horror at all the people that had been killed, and then a convincing 
feeling that on no account should another bomb be dropped. Yet of course 
another bomb was dropped, on Nagasaki, only three days after the bombing 
of Hiroshima.

In November 1945, three months after the atomic bombings, Oppenheimer stood 
firmly behind the scientific attitude, saying, "It is not possible to be 
a scientist unless you believe that the knowledge of the world, and the 
power which this gives, is a thing which is of intrinsic value to humanity, 
and that you are using it to help in the spread of knowledge and are willing 
to take the consequences."

Oppenheimer went on to work, with others, on the Acheson-Lilienthal report, 
which, as Richard Rhodes says in his recent book {\it Visions of Technology,} 
"found a way to prevent a clandestine nuclear arms race 
without resorting to armed world government"; their suggestion was a form 
of relinquishment of nuclear weapons work by nation-states to an international 
agency.

This proposal led to the Baruch Plan, which was submitted to the United 
Nations in June 1946 but never adopted (perhaps because, as Rhodes suggests, 
Bernard Baruch had "insisted on burdening the plan with conventional 
sanctions," thereby inevitably dooming it, even though it would "almost 
certainly have been rejected by Stalinist Russia anyway"). Other efforts to 
promote sensible steps toward internationalizing nuclear power to prevent an 
arms race ran afoul either of US politics and internal distrust, or distrust 
by the Soviets. The opportunity to avoid the arms race was lost, and very 
quickly.

Two years later, in 1948, Oppenheimer seemed to have reached another stage 
in his thinking, saying, 
"In some sort of crude sense which no vulgarity, no humor, no overstatement 
can quite extinguish, the physicists have known sin; and this is a knowledge 
they cannot lose."

In 1949, the Soviets exploded an atom bomb. By 1955, both the US and the 
Soviet Union had tested hydrogen bombs suitable for delivery by aircraft. 
And so the nuclear arms race began.

Nearly 20 years ago, in the documentary {\it The Day After Trinity,} Freeman 
Dyson summarized the scientific attitudes that brought us to the nuclear 
precipice:

"I have felt it myself. The glitter of nuclear weapons. It is irresistible 
if you come to them as a scientist. To feel it's there in your hands, to 
release this energy that fuels the stars, to let it do your bidding. To 
perform these miracles, to lift a million tons of rock into the sky. It 
is something that gives people an illusion of illimitable power, and it 
is, in some ways, responsible for all our troubles --- this, what you might 
call technical arrogance, that overcomes people when they see what they 
can do with their minds."\footnote{Else, Jon. {\it The Day After Trinity: J. Robert Oppenheimer and The Atomic 
Bomb} (available at 
\texttt{www.pyramiddirect.com}). 
}

Now, as then, we are creators of new technologies and stars of the imagined 
future, driven --- this time by great financial rewards and global competition 
--- despite the clear dangers, hardly evaluating what it may be like to try 
to live in a world that is the realistic outcome of what we are creating 
and imagining.




In 1947, {\it The Bulletin of the Atomic Scientists} began putting a Doomsday 
Clock on its cover. For more than 50 years, it has shown an estimate of 
the relative nuclear danger we have faced, reflecting the changing 
international 
conditions. The hands on the clock have moved 15 times and today, standing 
at nine minutes to midnight, reflect continuing and real danger from nuclear 
weapons. 
The recent addition of India and Pakistan to the list of nuclear powers 
has increased the threat of failure of the nonproliferation goal, and this 
danger was reflected by moving the hands closer to midnight in 1998.

In our time, how much danger do we face, not just from nuclear weapons, 
but from all of these technologies? How high are the extinction risks?

The philosopher John Leslie has studied this question and concluded that 
the risk of human extinction is at least 30 percent\footnote{This estimate is in Leslie's book {\it The End of the World: The Science and 
Ethics of Human Extinction,} where he notes that the probability of 
extinction 
is substantially higher if we accept Brandon Carter's Doomsday Argument, 
which is, briefly, that "we ought to have some reluctance to believe that 
we are very exceptionally early, for instance in the earliest 0.001 percent, 
among all humans who will ever have lived. This would be some reason for 
thinking that humankind will not survive for many more centuries, let alone 
colonize the galaxy. Carter's doomsday argument doesn't generate any risk 
estimates just by itself. It is an argument for {\it revising} the estimates 
which we generate when we consider various possible dangers." (Routledge, 
1996: 1, 3, 145.) 
}, 
while Ray Kurzweil believes we have "a better than even chance of making it 
through," with the caveat that he has "always been accused of being an 
optimist." Not only are these estimates not encouraging, but they do not 
include the probability of many horrid outcomes that lie short of extinction.

Faced with such assessments, some serious people are already suggesting 
that we simply move beyond Earth as quickly as possible. We would colonize 
the galaxy using von Neumann probes, which hop from star system to star 
system, replicating as they go. This step will almost certainly be necessary 
5 billion years from now (or sooner if our solar system is disastrously 
impacted by the impending collision of our galaxy with the Andromeda galaxy 
within the next 3 billion years), but if we take Kurzweil and Moravec at 
their word it might be necessary by the middle of this century.

What are the moral implications here? If we must move beyond Earth this 
quickly in order for the species to survive, who accepts the responsibility 
for the fate of those (most of us, after all) who are left behind? And 
even if we scatter to the stars, isn't it likely that we may take our problems 
with us or find, later, that they have followed us? The fate of our species 
on Earth and our fate in the galaxy seem inextricably linked.

Another idea is to erect a series of shields to defend against each of 
the dangerous technologies. The Strategic Defense Initiative, proposed 
by the Reagan administration, was an attempt to design such a shield against 
the threat of a nuclear attack from the Soviet Union. But as Arthur C. 
Clarke, who was privy to discussions about the project, observed: "Though 
it might be possible, at vast expense, to construct local defense systems 
that would 'only' let through a few percent of ballistic missiles, the 
much touted idea of a national umbrella was nonsense. Luis Alvarez, perhaps 
the greatest experimental physicist of this century, remarked to me that 
the advocates of such schemes were 'very bright guys with no common sense.'"

Clarke continued: "Looking into my often cloudy crystal ball, I suspect 
that a total defense might indeed be possible in a century or so. But the 
technology involved would produce, as a by-product, weapons so terrible 
that no one would bother with anything as primitive as ballistic missiles."
\footnote{Clarke, Arthur C. "Presidents, Experts, and Asteroids." {\it Science,} June 
5, 1998. Reprinted as "Science and Society" in {\it Greetings, Carbon-Based 
Bipeds! Collected Essays, 1934-1998.} St. Martin's Press, 1999: 526. 
}

In {\it Engines of Creation,} Eric Drexler proposed that 
we build an active nanotechnological shield --- a form 
of immune system for the biosphere --- to defend against dangerous replicators 
of all kinds that might escape from laboratories or otherwise be maliciously 
created. But the shield he proposed would itself be extremely dangerous 
--- nothing could prevent it from developing autoimmune problems and attacking 
the  biosphere itself\footnote{And,  as  David Forrest  suggests in  his
  paper "Regulating Nanotechnology Development", available at 
  \texttt{www.foresight.org/NanoRev/Forrest1989.html}:
  \begin{quote}
    "If we used strict liability as an alternative to regulation it 
    would be impossible for any developer to internalize the cost of the risk 
    (destruction  of the  biosphere), so  theoretically the  activity of
    developing 
    nanotechnology should never be undertaken." Forrest's analysis leaves us 
    with only government regulation to protect us --- not a comforting thought. 
  \end{quote}
}.


Similar difficulties apply to the construction of shields against robotics 
and genetic engineering. These technologies are too powerful to be shielded 
against in the time frame of interest; even if it were possible 
to implement defensive shields, the side effects of their development would 
be at least as dangerous as the technologies we are trying to protect against.

These possibilities are all thus either undesirable or unachievable or 
both. The only realistic alternative I see is relinquishment: to limit 
development of the technologies that are too dangerous, by limiting our 
pursuit of certain kinds of knowledge.

Yes, I know, knowledge is good, as is the search for new truths. We have 
been seeking knowledge since ancient times. Aristotle opened his Metaphysics 
with the simple statement: "All men by nature desire to know." We have, 
as a bedrock value in our society, long agreed on the value of open access 
to information, and recognize the problems that arise with attempts to 
restrict access to and development of knowledge. In recent times, we have 
come to revere scientific knowledge.

But despite the strong historical precedents, if open access to and unlimited 
development of knowledge henceforth puts us all in clear danger of extinction, 
then common sense demands that we reexamine even these basic, long-held 
beliefs.

It was Nietzsche who warned us, at the end of the 19th century, not only 
that God is dead but that "faith in science, which after all exists undeniably, 
cannot owe its origin to a calculus of utility; it must have originated 
 {\it in spite of} the fact that the disutility and dangerousness of the 'will 
to truth,' of 'truth at any price' is proved to it constantly." It is this 
further danger that we now fully face --- the consequences of our truth-seeking. 
The truth that science seeks can certainly be considered a dangerous substitute 
for God if it is likely to lead to our extinction.

If we could agree, as a species, what we wanted, where we were headed, 
and why, then we would make our future much less dangerous --- then we might 
understand what we can and should relinquish. Otherwise, we can easily 
imagine an arms race developing over GNR technologies, as it did with the 
NBC technologies in the 20th century. This is perhaps the greatest risk, 
for once such a race begins, it's very hard to end it. This time --- unlike 
during the Manhattan Project --- we aren't in a war, facing an implacable 
enemy that is threatening our civilization; we are driven, instead, by 
our habits, our desires, our economic system, and our competitive need 
to know.

I believe that we all wish our course could be determined by our collective 
values, ethics, and morals. If we had gained more collective wisdom over 
the past few thousand years, then a dialogue to this end would be more 
practical, and the incredible powers we are about to unleash would not 
be nearly so troubling.

One would think we might be driven to such a dialogue by our instinct for 
self-preservation. Individuals clearly have this desire, yet as a species 
our behavior seems to be not in our favor. In dealing with the nuclear 
threat, we often spoke dishonestly to ourselves and to each other, thereby 
greatly increasing the risks. Whether this was politically motivated, or 
because we chose not to think ahead, or because when faced with such grave 
threats we acted irrationally out of fear, 
I do not know, but it does not bode well.

The new Pandora's boxes of genetics, nanotechnology, and robotics are almost 
open, yet we seem hardly to have noticed. Ideas can't be put back in a 
box; unlike uranium or plutonium, they don't need to be mined and refined, 
and they can be freely copied. Once they are out, they are out. Churchill 
remarked, in a famous left-handed compliment, that the American people 
and their leaders "invariably do the right thing, after they have examined 
every other alternative." In this case, however, we must act more presciently, 
as to do the right thing only at last may be to lose the chance to 
do it at all.




As Thoreau said, "We do not ride on the railroad; it rides upon us"; and 
this is what we must fight, in our time. The question is, indeed, Which 
is to be master? Will we survive our technologies?

We are being propelled into this new century with 
no plan, no control, no brakes. Have we already gone too far down the path 
to alter course? I don't believe so, but we aren't trying yet, and the 
last chance to assert control --- the fail-safe point --- is rapidly approaching. 
We have our first pet robots, as well as commercially available genetic 
engineering techniques, and our nanoscale techniques are advancing rapidly. 
While the development of these technologies proceeds through 
a number of steps, it isn't necessarily the case --- as happened in the Manhattan 
Project and the Trinity test --- that the last step in proving a technology 
is large and hard. The breakthrough to wild self-replication in robotics, 
genetic engineering, or nanotechnology could come suddenly, reprising the 
surprise we felt when we learned of the cloning of a mammal.

And yet I believe we do have a strong and solid basis for hope. Our attempts 
to deal with weapons of mass destruction in the last century provide a 
shining example of relinquishment for us to consider: the unilateral US 
abandonment, without preconditions, of the development of biological weapons. 
This relinquishment stemmed from the realization that while it would take 
an enormous effort to create these terrible weapons, they could from then 
on easily be duplicated and fall into the hands of rogue nations or terrorist 
groups.

The clear conclusion was that we would create additional threats to ourselves 
by pursuing these weapons, and that we would be more secure if we did not 
pursue them. We have embodied our relinquishment of biological and chemical 
weapons in the 1972 Biological Weapons Convention (BWC) and the 1993 Chemical 
Weapons Convention (CWC)\footnote{Meselson, Matthew. 
"The Problem of Biological Weapons." Presentation to the 1,818th Stated 
Meeting of the American Academy of Arts and Sciences, January 13, 1999. 
(minerva.amacad.org/archive/bulletin4.htm) 
}

As for the continuing sizable threat from nuclear weapons, which we have 
lived with now for more than 50 years, the US Senate's recent rejection 
of the Comprehensive Test Ban Treaty makes it clear relinquishing nuclear 
weapons will not be politically easy. But we have a unique opportunity, 
with the end of the Cold War, to avert a multipolar arms race. Building 
on the BWC and CWC relinquishments, successful abolition 
of nuclear weapons could help us build toward a habit of relinquishing 
dangerous technologies. (Actually, by getting rid of all but 100 nuclear 
weapons worldwide --- roughly the total destructive power of World War II 
and a considerably easier task --- we could eliminate this extinction threat
\footnote{Doty, Paul. "The 
Forgotten Menace: Nuclear Weapons Stockpiles Still Represent the Biggest 
Threat to Civilization." {\it Nature,} 402, December 9, 1999: 583. 
}.)

Verifying relinquishment will be a difficult problem, but not an unsolvable 
one. We are fortunate to have already done a lot of relevant work in the 
context of the BWC and other treaties. Our major task will be 
to apply this to technologies that are naturally much more commercial than 
military. The substantial need here is for transparency, as difficulty 
of verification is directly proportional to the difficulty of distinguishing 
relinquished from legitimate activities.

I frankly believe that the situation in 1945 was simpler than the one we 
now face: The nuclear technologies were reasonably separable into commercial 
and military uses, and monitoring was aided by the nature of atomic tests 
and the ease with which radioactivity could be measured. Research on military 
applications could be performed at national laboratories such as Los Alamos, 
with the results kept secret as long as possible.

The GNR technologies do not divide clearly into commercial and military uses; 
given their potential in the market, it's hard to imagine pursuing them only 
in national laboratories. With their widespread commercial pursuit, enforcing 
relinquishment will 
require a verification regime similar to that for biological weapons, but 
on an unprecedented scale. This, inevitably, will raise tensions between 
our individual privacy and desire for proprietary information, and the 
need for verification to protect us all. We will undoubtedly encounter 
strong resistance to this loss of privacy and freedom of action.

Verifying the relinquishment of certain GNR technologies will have to occur 
in cyberspace as well as at physical facilities. The critical issue will 
be to make the necessary transparency acceptable in a world of proprietary 
information, presumably by providing new forms of protection for intellectual 
property.

Verifying compliance will also require that scientists and engineers adopt 
a strong code of ethical conduct, resembling the Hippocratic oath, and 
that they have 
the courage to whistleblow as necessary, even at high personal cost. This 
would answer the call --- 50 years after Hiroshima --- by the Nobel laureate 
Hans Bethe, one of the most senior of the surviving members of the Manhattan 
Project, that all scientists "cease and desist from work creating, developing, 
improving, and manufacturing nuclear weapons and other weapons of potential 
mass destruction."\footnote{See also Hans Bethe's 1997 letter to President Clinton, at 
\texttt{www.fas.org/bethecr.htm}. 
} In the 21st century, this requires 
vigilance and personal 
responsibility by those who would work on both NBC and GNR technologies 
to avoid implementing weapons of mass destruction and knowledge-enabled 
mass destruction.




Thoreau also said that we will be "rich in proportion to the 
number of things which we can afford to let alone." 
We each seek to be happy, but it would seem worthwhile to question whether 
we need to take such a high risk of total destruction to gain yet more 
knowledge and yet more things; common sense says that there is 
a limit to our material needs --- and that certain knowledge is too dangerous 
and is best forgone.

Neither should we pursue near immortality without considering the costs, 
without considering the commensurate increase in the risk of extinction. 
Immortality, while perhaps the original, is certainly not the only possible 
utopian dream.

I recently had the good fortune to meet the distinguished author and scholar 
Jacques Attali, whose book {\it Lignes d'horizons} ({\it Millennium,} in the 
English 
translation) helped inspire the Java and Jini approach to the coming age 
of pervasive computing, as previously described in this magazine. In his 
new book {\it Fraternit\'{e}s,} Attali describes how our dreams of utopia 
have changed over time:


\begin{quote}
At the dawn of societies, men saw their passage on Earth as nothing more 
than a labyrinth of pain, at the end of which stood a door leading, via 
their death,  to the  company of  gods and to  {\it Eternity.}  With the
Hebrews and 
then the Greeks, some men dared free themselves from theological demands 
and dream of an ideal City where {\it Liberty} would flourish. Others, noting 
the evolution of the market society, understood that the liberty of some 
would entail the alienation of others, and they sought {\it Equality}.
\end{quote}

Jacques helped me understand how these three 
different utopian goals exist in tension in our society today. He goes 
on to describe a fourth utopia, {\it Fraternity,} whose foundation is altruism. 
Fraternity alone associates individual happiness with the happiness of 
others, affording the promise of self-sustainment.

This crystallized for me my problem with Kurzweil's dream. A technological 
approach to Eternity --- near immortality through robotics --- may not be the 
most desirable utopia, and its pursuit brings clear dangers. Maybe we should 
rethink our utopian choices.

Where can we look for a new ethical basis to set 
our course? I have found the ideas in the book 
{\it Ethics for the New Millennium,} 
by the Dalai Lama, to be very helpful. As is perhaps well known but little 
heeded, the Dalai Lama argues that the most important thing 
is for us to conduct our lives with love and compassion for others, and 
that our societies need to develop a stronger notion of universal 
responsibility 
and of our interdependency; he proposes a standard of positive ethical 
conduct for individuals and societies that seems consonant with Attali's 
Fraternity utopia.

The Dalai Lama further argues that we must understand what it is that makes 
people happy, and acknowledge the strong evidence that neither material 
progress nor the pursuit of the power of knowledge is the key --- that there 
are limits to what science and the scientific pursuit alone can do.

Our Western notion of happiness seems to come 
from the Greeks, who defined it as "the exercise of vital 
powers along lines of excellence in a life affording them scope."\footnote{Hamilton, Edith. {\it The Greek Way.} W. W. Norton \& Co., 1942: 35. 
}


Clearly, we need to find meaningful challenges 
and sufficient scope in our lives if we are to be happy in whatever is 
to come. But I believe we must find alternative outlets for our creative 
forces, beyond the culture of perpetual economic growth; this growth has 
largely been a blessing for several hundred years, but 
it has not brought us unalloyed happiness, and we must now choose between 
the pursuit of unrestricted and undirected growth through science and 
technology and the clear accompanying dangers.




It is now more than a year since my first encounter with Ray Kurzweil and 
John Searle. I see around me cause for hope in the voices for caution and 
relinquishment and in those people I have discovered who are as concerned 
as I am about our current predicament. I feel, too, a deepened sense of 
personal responsibility --- not for the work I have already done, but for the 
work that I might yet do, at the confluence of the sciences.

But many other people who know about the dangers still seem strangely silent. 
When pressed, they trot out the "this is nothing new" riposte --- as if 
awareness of what could happen is response enough. They tell me, There are 
universities filled with bioethicists who study this stuff all day long. 
They say, All this has been written about before, and by experts. They 
complain, Your worries and your arguments are already old hat.

I don't know where these people hide their fear. As an architect of complex 
systems I enter this arena as 
a generalist. But should this diminish my concerns? 
I am aware of how much has been written about, talked about, and lectured 
about so authoritatively. But does this mean it has reached people? Does 
this mean we can discount the dangers before us?

Knowing is not a rationale for not acting. Can we doubt that knowledge 
has become a weapon we wield against ourselves?

The experiences of the atomic scientists clearly show the need to take 
personal responsibility, the danger that things will move too fast, and the 
way in which a process can take on a life of its own. We can, as they did, 
create insurmountable 
problems in almost no time flat. We must do more thinking up front if we 
are not to be similarly surprised and shocked by the consequences of our 
inventions.

My continuing professional work is on improving 
the reliability of software. Software is a tool, and as 
a toolbuilder I must struggle with the uses to which 
the tools I make are put. I have always believed that making software more 
reliable, given its many uses, will make the world a safer and better place; 
if I were to come to believe the opposite, then I would be morally obligated 
to stop this work. I can now imagine such a day may come.

This all leaves me not angry but at least a bit melancholic. Henceforth, 
for me, progress will be somewhat bittersweet.




Do you remember the beautiful penultimate scene in Manhattan where Woody 
Allen is lying on his couch and talking into a tape recorder? He is writing 
a short story about people who are creating unnecessary, neurotic problems 
for themselves, because it keeps them from dealing with more unsolvable, 
terrifying problems about the universe.

He leads himself to the question, "Why is life worth living?" and to consider 
what makes it worthwhile for him: Groucho Marx, Willie Mays, the second 
movement of the Jupiter Symphony, Louis Armstrong's recording of "Potato 
Head Blues," Swedish movies, Flaubert's Sentimental Education, Marlon Brando, 
Frank Sinatra, the apples and pears by C\'{e}zanne, the crabs at Sam Wo's, 
and, finally, the showstopper: his love Tracy's face.

Each of us has our precious things, and as we care for them we locate the 
essence of our humanity. In the end, it is because of our great capacity 
for caring that I remain optimistic we will confront the dangerous issues 
now before us.

My immediate hope is to participate in a much larger discussion of the 
issues raised here, with people from many different backgrounds, in settings 
not predisposed to fear or favor technology for its own sake.

As a start, I have twice raised many of these issues 
at events sponsored by the Aspen Institute and have separately proposed 
that the American Academy of Arts and Sciences take them up as an extension of 
its work with the Pugwash Conferences. (These have been held since 1957 
to discuss arms control, especially of nuclear weapons, and to formulate 
workable policies.)

It's unfortunate that the Pugwash meetings started only well after the 
nuclear genie was out of the bottle --- roughly 15 years too late. We are 
also getting a belated start on seriously addressing the issues around 
21st-century technologies --- the prevention of knowledge-enabled mass 
destruction --- and further delay seems unacceptable.

So I'm still searching; there are many more things to learn. Whether we 
are to succeed or fail, to survive or fall victim to these technologies, 
is not yet decided. I'm up late again --- it's almost 6 am. I'm trying to 
imagine some better answers, to break the spell and free them from the 
stone.


\end{document}