\chapter{Reconfiguration Agents}
\label{chap:reconfig-agents}
\index{agents|(textbf}
\index{dynamic (re)configuration}

Management of software configuration in large scale networks is a major
problem in modern computer systems.  Scalability conflicts with dynamic
flexibility since it is difficult to manage dynamic changes in a large number
of machines concurrently. Scalability\index{scalability} is a topic that
received very little attention from previous research in dynamic
configuration.

Internet services such as video on demand, on-line bookstores, and search
engines must maintain a high level of parallelism in order to fulfill the
millions of requests they receive every hour. World-wide services with
high-bandwidth requirements such as on-line sales of software must be readily
available in many different countries. Services must be replicated or split
across machines in different locations, optimizing bandwidth and CPU
utilization, and minimizing user response time.

Since software evolves rapidly and availability is a major requirement in
these systems, the natural solution to cope with all these requirements is to
use dynamic reconfiguration. The problem is that reconfiguring and updating
the software in a distributed collection of servers using conventional tools
is troublesome. Besides, it may lead to wasted bandwidth as duplicated
information is sent through the same Internet lines.

A significant limitation of the traditional administrative tools for
reconfiguration is their scalability. In existing systems
supporting dynamic reconfiguration, the administrator must establish a
point-to-point connection between the administration node and the process that
is going to be reconfigured.  Thus, if a system administrator needs to upgrade
a certain component of an on-line service composed of ten replicas located in
different continents, it is necessary to connect to each replica separately,
upload the new implementation of the component, and reconfigure the
replica. As the latency in long-distance Internet connections tends to be
high, this process is extremely laborious and tiresome.

The solution we propose is to allow administrators to organize the nodes of
their Internet systems in a hierarchical manner for reconfiguration purposes.
The administrator specifies the topology of the distributed application as a
directed graph and creates a mobile \emph{reconfiguration agent} which is
injected into the network.  The reconfiguration agent then visits the nodes of
this graph of interconnected nodes. Each node replicates and forwards the
agent to neighboring nodes, processes the reconfiguration commands locally,
and collects the reconfiguration results, sending them back to the neighboring
agent source. Agents can carry not only reconfiguration and inspection
commands, but also the code for new components to be distributed throughout
the network.

Using this approach, the administrator can organize the reconfiguration
hierarchy to optimize the data flow between distant application nodes. The
reconfiguration commands are executed in parallel in the various nodes,
improving response time. If desired, the graph may contain different levels of
redundancy so that the system can tolerate the failure of some of the nodes in
the reconfiguration network.

\section{Implementation}

Our implementation is based on the \dTAO reflective ORB which is described in
detail in Section \ref{sec:dtao}. \dTAO exports two interfaces for dynamic
reconfiguration: an IDL interface that can be used by CORBA clients and a
simple ASCII interface based on commands send over TCP/IP. The latter can be
accessed via simple telnet connections and uses the commands specified
in the Distributed Configuration Protocol (DCP)\index{DCP} (see Section \ref{sec:dtao} and
\cite{DCP}). DCP includes commands for inspection, reconfiguration, and code
distribution.

To support agents, \dTAO was enhanced with mechanisms for receiving,
processing, and forwarding agents. The system also includes
\emph{ConfigAgentBuilder}, a graphical administrative front-end, written in
Java, for specifying reconfiguration graphs and for assembling and sending
reconfiguration agents \cite{kon99:agents-TR}.

The administrator selects those ORBs that will be part of the reconfiguration
graph (see Figure \ref{fig:OS}) and, using the mouse, draws directed edges
connecting the graph nodes (see Figure \ref{fig:RT}). Each time a new ORB is
selected from the list on the left-hand side of Figure \ref{fig:OS}, a new
node is added to the graph in Figure \ref{fig:RT}.
\begin{figure}[tb]
\mbox{
\subfigure[Selecting the graph nodes]
{\label{fig:OS}%
\leavevmode \epsfig{file=figs/OS.ps,width=7.5cm}}
\subfigure[Creating the graph edges]{%
\label{fig:RT}%
\leavevmode \epsfig{file=figs/RT.ps,width=7.5cm}}
}
\caption{The \dTAO \emph{ConfigAgentBuilder} Interface}
\end{figure}

Once the reconfiguration graph is established, a graphical interface assists
the administrator in building a script of DCP commands that are codified into a
reconfiguration agent. Finally, the administrator instructs the graphical
front-end to send the agent to an initial node in the graph. Figure
\ref{fig:DCPCommands} shows the composition of an agent with three DCP
commands: \texttt{list\_categories}, \texttt{list\_loaded\_implementations}, 
and \texttt{list\_implementations}.
\begin{figure}[tb]
\mbox{
\subfigure[Composing an agent]
{\label{fig:DCPCommands}%
    \leavevmode \epsfig{file=figs/DCPCommands.ps,width=7.5cm}}
\subfigure[Information collected by an inspection agent]
{\label{fig:results}%
\leavevmode \epsfig{file=figs/results.ps,width=7.5cm}}
}
\caption{Sending and Receiving Agents}
\end{figure}

Each agent contains 
\begin{enumerate}
\item a copy of the reconfiguration graph (so that different
agents can operate on different graphs at the same time),
\item a script of DCP\index{DCP} commands to be interpreted on each node in the graph, and 
\item a unique sequence number which is used to avoid processing duplicated
copies of the same agent on the same node.
\end{enumerate}

Upon receiving an agent, the ORB first checks whether it has been processed
before by looking at its local history of processed agents. If the agent has
not been processed, it distributes the agent to its neighbors (according to
the reconfiguration graph included in the agent), processes the DCP commands
locally, and waits for the results of the agents sent to its neighbors.  The
ORB, then, collects the replies from its neighbors, groups them with the
results of its own reconfiguration, and sends all of the information back to
the node from which it received the agent.  Each node waits for the results
from its agents for a certain period of time. A timeout is computed taking
into consideration the number of nodes the agent has already traversed.

The results returned by all the nodes in the reconfiguration graph are finally
displayed to the administrator through the graphical front-end. The system
also draws attention to the nodes that did not reply within a given timeout.
Figure \ref{fig:results} shows the results returned by an agent containing
three DCP inspection commands (also called an \emph{inspection agent}).
Our agent dissemination system can be seen as a simple, lightweight
implementation of reliable multicast \cite{reliable-multicast} over TCP/IP.
We intentionally adopted this simple and reliable solution for agent
transmission because it lets us concentrate on the dynamic configuration
aspects of the problem. However, depending upon the topology of the
reconfiguration network (for example, if the nodes have many output edges), it
may be worthwhile to use other underlying protocols such as IP-Multicast
\cite{deering:multicast-routing} for distributing
agents. In that case, however, the complexity of the middleware would be much
higher since it would have to guarantee reliability over a non-reliable
protocol.


\subsection{Java Agents}

Our implementation supports agents with simple reconfiguration scripts based
on the DCP protocol. On the one hand, this allowed us to write an extremely
lightweight interpreter that processes the agents efficiently. This is very
appropriate for PDAs and embedded systems with limited resources.  On the
other hand, it limits the expressiveness of the reconfiguration agents. By
using a more powerful language for reconfiguration such as OCL \cite{olan98}
or a general-purpose language like Java or Emerald \cite{jul:emerald95}, the
administrator would be able to write more sophisticated agents that could use the
agent and ORB state to make decisions about their actions.  To achieve that, we
designed an infrastructure for Java agents that was implemented by Masters
students in our group \cite{kon00:asama}.

To support this implementation, we encapsulated Sun's Java Virtual Machine
into a component that can be dynamically loaded into the \dTAO domain; this
component exports an interface that contains operations for loading and
interpreting Java bytecode.  In addition to the reconfiguration topology
graph, that now can be modified on-the-fly, the Java agent carries its dynamic
state from node to node and contains Java bytecode that is interpreted inside
a sand-box so that its actions can be limited. By using the
\class{DynamicConfigurator} IDL interface (see Section
\ref{sec:reconfiguration-interfaces}), the Java bytecode issues inspection and
reconfiguration commands to the ORB domain. Based on the result of these
commands, the agent can
\begin{enumerate}
\item update the state it carries from node to node, 
\item change its behavior to adapt to the environment and user preferences on
each node, and
\item modify the reconfiguration graph, deciding to migrate to a
different set of nodes.
\end{enumerate}
In that manner, our framework offers both simple agents based on DCP\index{DCP} scripts
for environments with limited resources and powerful Java agents that can
carry state and modify its behavior along the way.

\section{Security}
\index{security}

Although the introduction of mobile agents promotes significant benefits, it
also brings additional security concerns \cite{agents-security98}.  Luckily,
the problem of mobile agents security has been studied extensively in recent
years and good results are starting to appear. When using mobile agents for
dynamic configuration, it is
important to 
\begin{enumerate}
\item restrict the sources of mobile code to trusted, authenticated
entities \cite{kon00:asama,kon99:agents-TR2}, 
\item limit the mobile code's resource
consumption \cite{bredin:market}, and 
\item limit its capabilities by confining
it to secured environments like sand boxes for Java byte code
\cite{java95:sandbox} and the Janus sand box for native code \cite{janus96}.
\end{enumerate}

\cite{vitek99:javaseal} shows how to isolate agents from one another.
However, another important aspect, the protection of the mobile agents from
the execution environments, is still mostly unresolved.  One can find partial
solutions in approaches like time-limited black-box security~\cite{blackbox98}
and the clueless agents~\cite{clueless98} concept.  The approach adopted in
our group is to establish secure connections between the nodes distributing
the agent using a role-based access control framework. The details are beyond
the scope of this thesis and are described in \cite{kon00:asama} and \cite{kon99:agents-TR2}.

\section{Fault-Tolerance and Consistency}
\index{fault-tolerance}
\index{consistency}

Our experience with distributed computer systems has shown that a wide range
of applications do not require strict consistency between
the components in the different nodes. When deploying the multimedia
distribution system described in Section \ref{sec:reflector}, for example, we
were able to use different versions of application components in different
nodes simultaneously. In many cases, even if a particular component update was
successful in part of the nodes and failed in another part, the distributed
system could continue working together gracefully until all the updates were
achieved manually.

To keep the system small and efficient, we opted not to provide strong
guarantees about the execution of the reconfiguration commands in the current
implementation.  On a single node, some reconfiguration commands may fail
while others succeed; it is the responsibility of the agent to catch the
exceptions raised by the failed commands and treat them accordingly. Also,
reconfiguration commands may succeed on some nodes and fail on others. To cope
with these errors, the system detects the failures and the graphical front-end
(shown in Figure \ref{fig:results}) displays, to the administrator, where they
occurred and the respective diagnosis messages. Then, using a separate Java
GUI developed by Luiz Claudio Magalh\~aes and called \Doctor (the Dynamic ORB
Configuration Tool\index{Doctor} in Figure \ref{fig:doctor}), the
administrator can connect to the node where the failure occurred, inspect its
state and reconfigure the node manually. By navigating through ORB and
application components interactively, the administrator can investigate the
causes of the errors, and fix them.

\begin{figure}[htb]
  \begin{center}
    \leavevmode \epsfig{file=figs/doctor.ps,width=12cm}
    \caption{The \Doctor Configuration Tool}
    \label{fig:doctor}
  \end{center}
\end{figure}

On the other hand, it is possible to achieve the effects of atomic
transactions\index{atomicity|(} using the functionality we are offering through the Java
agents. On an individual node, the Java agent can be programmed to store the
previous state of the system before starting to make any changes. Should an
error occur during the reconfiguration process on that node, the Java code can
detect the error messages and bring the configuration back to the initial
state\footnote{Depending on the components being reconfigured, it may be
necessary to save the component internal state before the configuration occurs
so that it can be restored in case it is necessary to roll back to the initial
state. This can be achieved by requiring each component to implement
operations for capturing and restoring its internal state as described in
Section \ref{sec:cc-dynconf}.}.
%it is done, for example, in Surgeon \cite{surgeon93}.}.

To achieve the effects of atomic transactions on the distributed system as a
whole, one can use a different technique. According to our experience,
reconfiguration failures happen, typically, when a new component is loaded
(and the dynamic loader cannot link it) or when the new component is
initialized (and it fails to locate the functions or
allocate the resources it needs). Thus, we instruct the administrator to
separate its reconfiguration in two phases resembling the two-phase commit
protocol for atomic transactions \cite{gray93:transactions}. First, the
administrator sends an agent to load all the required components and to
initialize them by executing the operation \method{load\_implementation()}
(and, if needed, \method{configure\_implementation()}).  If an error in any
node occurs, the administrator chooses between sending a new agent to unload
the new components from all the nodes or using \Doctor to fix the individual
errors.  If the component implementations are loaded and initialized
successfully in all the nodes, the administrator sends a new agent to attach
them to the proper applications and ORB components, which would correspond to
the \emph{commit} phase of the two-phase commit protocol.

We found this to be sufficient for most of the application scenarios with
which we work. However, some other applications require more strict guarantees
of the ACID properties for atomic transactions \cite{gray93:transactions}.  For
example, if we want to reconfigure the marshalling and unmarshaling
components in all the ORBs in a distributed system, it is important to
guarantee an \emph{all-or-nothing} property in the transactions. If an ORB
replaces its marshalling mechanism and one of the ORBs to which it sends
requests does not change its unmarshaling mechanism, they might no longer be
able to communicate.

To support that, one could extend our infrastructure by using standard
transaction mechanisms for distributed systems \cite{hicss5} and recent
protocols for mobile agent fault-tolerance
\cite{silva98:agent-fault-tolerance}. The deployment of these mechanisms for
the reconfiguration of active components is not straightforward. It is an
interesting research topic. In this case, the atomic transactions must
deal with active objects (i.e., data + code + running threads) rather than
simply data, as is the case with more traditional uses of atomic transactions
such as file systems \cite{quicksilver} and distributed databases
\cite{gray93:transactions}.

\index{atomicity|)}
\index{agents|)}

