\chapter{Experimental Results}
\label{chap:experiments}

In this chapter we present the results of experiments carried out to evaluate
the performance of the three major elements of our architecture: the Automatic
Configuration Service, the component configurator (in this case, used to
support dynamic reconfiguration of the Reflector Network), and the mobile
agents infrastructure (used for reconfiguration, inspection, and code
distribution).

\section{Automatic Configuration Service}
\index{automatic configuration!experiments}

The Automatic Configuration Service is implemented as a library that can be
linked to any application. A program enhanced with this service becomes
capable of fetching components from a remote Component
Repository\index{component repository} and dynamically loading and assembling
them into its local address-space. The library requires only 157Kbytes of memory
on Solaris 7, which makes it possible to use it even on machines with limited
resources such as a PalmPilot. In fact, we expect that services similar to
this will be extensively used in future mobile systems to configure software
automatically according to location and user requirements.

To evaluate the performance of the Automatic Configuration Service, we
instrumented the Reflector application described in Section
\ref{sec:reflector} to measure the time for fetching, dynamic linking, and
configuring its constituent components.

\subsection{Loading Multiple Components}

Figure \ref{fig:autoconf-performance} shows the total time for the service to
load from one to eight components of 19.2Kbytes each. These experiments were
carried out on two Sparc Ultra-60 machines running Solaris~7 and connected by
a 100Mbps Fast Ethernet network. The Component Repository was executed on one
of the machines and the Reflector with the Automatic Configuration Service on
the other. Each value is the average of five runs of the experiment. The
vertical bars in the subsequent graphs and the numbers in parentheses in Table
\ref{tab:autoconf-performance} represent the standard deviation.

\begin{figure}[htb]
\centerline{\epsfig{file=figs/autoconf.eps,width=10cm}}
\caption{Automatic Configuration Service Performance}
\label{fig:autoconf-performance}
\end{figure}

Table \ref{tab:autoconf-performance} shows, in more detail, how the service
spends its time when loading a single 19.2Kbyte component. The current version
of the Automatic Configuration Service fetches the
prerequisites\index{prerequisites} file from the remote Component
Repository\index{component repository} and saves it to the local disk. The
same is done with the file containing the component code. Then, it uses \dTAO
to perform the local dynamic linking of the component into the process runtime.

\begin{table}[htbp]
\begin{center}
\begin{tabular}{|l|c|c|}
\hline
Action                                              & Time  (ms) & \% of the total \\
\hline\hline
fetching prerequisites from Component Repository    &     2 (0)  &    8 \\
\hline
saving prerequisites to local disk                  &     1 (0)  &    4 \\ 
\hline
fetching component from Component Repository        &     4 (0)  &   17 \\ 
\hline                   
saving component to local disk                      &     1 (0)  &    4 \\ 
\hline                   
local dynamic linking                               &     5 (0)  &   21 \\ 
\hline                   
autoconf protocol additional operations             &  11 (0.7)  &   46 \\ 
\hline                   
Total                                               &  24 (0.7)  &  100 \\ 
\hline
\end{tabular}
\end{center}
\caption{Discriminated Times for Loading a 19.2Kbyte Component}
\label{tab:autoconf-performance}
\end{table}

The table also shows the additional time spent by the service (row labeled as
``autoconf protocol additional operations'') to detect if there are more
components or prerequisite files to load, to parse the prerequisite file, and
to reify dependencies.  This overhead accounts for 46\% of the total time
required to load the component, which suggests that it would be desirable to
improve this part of the service by optimizing the implementation of the
\class{SimpleResolver} (see Section
\ref{sec:autoconfig-concrete}). We believe that an optimized version of the
\class{SimpleResolver} could lead to improvements in the order of 20\% for
components of this size.

In the experiments described in this section, the component code and
prerequisite files were cached in the memory of the machine executing the
Component Repository. When the Component Repository program needs to read both
files from its local disk, there is an additional overhead of approximately 20
milliseconds.

\subsection{Loading Components of Different Sizes}

To evaluate how the time for loading a single component varies with the component
size, we created a program that generates components of different
sizes. According to its command-line arguments, this program generates C++
source code containing a given number of functions (which include code to
perform simple arithmetic operations) and local and global variables. Using
this program, we created components whose DLL sizes vary from 12 to 115 Kbytes.
Figure \ref{fig:autoconf-performance-line} shows the time for the Automatic
Configuration Service to load a single component as the component size increases.

\begin{figure}[htb]
\centerline{\epsfig{file=figs/componentSizes.eps,width=10cm}}
\caption{Times for Loading Components of Different Sizes}
\label{fig:autoconf-performance-line}
\end{figure}

Figure \ref{fig:autoconf-performance-bar-absolute} shows the absolute times spent in
each step of the process\footnote{These steps are the same as those presented
in Table \ref{tab:autoconf-performance}.}. We can notice that the time spent in the item
labeled ``autoconf protocol'' is approximately constant\footnote{This is expected
since the messages processed in this step do not carry component code and
therefore are not affected by the size of the component.}. Hence, as the
component size increases, its relative importance decreases. This can be
noticed in Figure \ref{fig:autoconf-performance-bar-percentage}, which shows the
same data in a different form. In this case, the figure shows the percentage
of the total time spent in each of the steps of the process.

\begin{figure}[!htb]
\centerline{\epsfig{file=figs/componentSizesAbsolute.ps,width=12cm}}
\caption{Discriminated Times for Loading Components of Different Sizes}
\label{fig:autoconf-performance-bar-absolute}
\end{figure}

\begin{figure}[!hbt]
\centerline{\epsfig{file=figs/componentSizesPercentage.ps,width=12cm}}
\caption{Discriminated Percentual Times for Loading Components of Different Sizes}
\label{fig:autoconf-performance-bar-percentage}
\end{figure}

As the size of the component increases, the time for fetching the code from
the remote repository to the local machine becomes the dominant factor. It is
important to remember that these data were captured in a fast local
network. If the access to the repository requires the use of a lower bandwidth
connection, then this step would clearly be the most important with respect to
performance. This suggests that ``component caching''  is an important topic for
future research.

\subsection*{Discussion}

Although there is still much room for improvements and performance
optimizations in the protocols used by the Automatic Configuration Service,
the results presented here are very encouraging.  They demonstrate that it is
possible to carry out automatic configuration of a distributed component-based
application within a tenth of a second, which is what we intended to
prove.\index{automatic configuration}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Dynamic Reconfiguration Using Component Configurators}
\label{sec:experiments-dynreconf}
\index{component configurator!experiments}
\index{Reflector!experiments|(}

In Section \ref{sec:dynconf-qos} we explained how to use component
configurators to manage dependencies and support dynamic reconfiguration of a
distributed application such as the Multimedia Distribution System. In this
section, we present experimental results showing that dependence management
using component configurators can be performed efficiently.

We conducted experiments to evaluate the performance of dynamic reconfiguration
using component configurators in the Reflector system as described in
Section \ref{sec:reflector-reconfig}.  In particular, we were interested in
evaluating the impact of the automatic reconfiguration of the reflector
network on the quality of the multimedia streams received by end-users. We
carried out this set of experiments on seven Sun Ultra machines\footnote{Our
testbed consisted of two dual-processor Sun Ultra-60 machines with 360 MHz
processors and five Sun Ultra-5 machines with 333 MHz processors.}  connected
by a 100Mbps Fast Ethernet. As depicted in Figure \ref{fig:testbed}, three of
these machines executed our Reflector program. Another machine executed
TestServer, a program we created to synthesize an RTP\index{RTP} stream \cite{rtp:00} with
bandwidth and packet rate specified in its command line. A fifth machine executed
TestClient, a program that receives the packets from a Reflector and logs the
packet arrival times into a file.  The sixth machine executed the CORBA Name
Service and the last one, the Component Repository.\index{component repository}

\begin{figure}[htb]
\centerline{\epsfig{file=figs/testbed.eps,width=12cm}}
\caption{Reconfiguration Experiment Testbed}
\label{fig:testbed}
\end{figure}

As explained in Section \ref{sec:reflector-reconfig}, when the
Reflector B goes down, the system reconfigures itself automatically and adopts
the new topology shown in Figure \ref{fig:afterReconfig}. When the Reflector B
recovers, the system reconfigures itself back to the topology shown in
Figure \ref{fig:testbed}.

\begin{figure}[htb]
\centerline{\epsfig{file=figs/afterReconfig.eps,width=12cm}}
\caption{Reconfiguration when Reflector B Goes Down}
\label{fig:afterReconfig}
\end{figure}

We evaluate the impact of these reconfigurations on the quality of service as
perceived by end-users by using the information in the log file to compute the
packet inter-arrival time at the TestClient. The following figures plot packet
inter-arrival time (in seconds) over time (in seconds). Each experiment lasted
between 15 and 18 seconds.

Figure \ref{fig:noreconf1200} shows the packet inter-arrival times at our
testbed client when no reconfigurations take place. The TestServer
sends an RTP/UDP stream of 1.2Mbps at 30 packets per second.

\begin{figure}[htb]
\centerline{\epsfig{file=figs/reflectorGraf4.ps,width=10cm}}
\caption{No Reconfigurations. Bandwidth = 1.2Mbps, Packet Rate = 30pps}
\label{fig:noreconf1200}
\end{figure}

The TestServer program uses an adaptive algorithm to keep its average output
bandwidth as close as possible to the output bandwidth specified at its
command line. Once every $K$ seconds, the program checks its output bandwidth
and modifies its packet inter-transmission time to adapt to the changes in its
measured output bandwidth\footnote{The measurement of the output bandwidth is
based on the values returned by the \method{send} system call.}. The packet
size is fixed throughout each experiment and it is computed by dividing the
bandwidth by the packet rate.

In our first experiments, we set $K = 1$ and the desired packet rate to 30
packets per second. The TestServer, then, tries to send one packet every
33.3 milliseconds. However, since the Solaris clock tick is set to 10
milliseconds, it does not let a program sleep for exactly 33.3
milliseconds. Hence, our adaptive program ends up sending one packet every 30
or 40 milliseconds which explains most of the variations throughout the
experiment in Figure \ref{fig:noreconf1200}.  The small and the localized
variations (such as the one at 8.4$s$) are due to changes in the network and
machine loads caused by other applications.

Figure \ref{fig:badreconf1200} shows the impact of reconfigurations on our
first implementation of the fault-tolerance mechanism described in Section
\ref{sec:reflector-reconfig}. The continuous arrows point to the instants in
which the Reflector B is killed and the Reflector network topology switches
from the configuration in Figure \ref{fig:testbed} to the one in Figure
\ref{fig:afterReconfig}. The dashed arrows point to the instants in which the
Reflector B recovers and the topology switches back to the one in
Figure~\ref{fig:testbed}.

\begin{figure}[htb]
\centerline{\epsfig{file=figs/reflectorGraf1.ps,width=10cm}}
\caption{Reflector Reconfigurations. Bandwidth = 1.2Mbps, Packet Rate = 30pps}
\label{fig:badreconf1200}
\end{figure}

One can see that when we killed the Reflector, there was a peak in the
inter-arrival time at the client. This happened because, in that implementation,
as soon as the reflector received the termination signal, it stopped
forwarding packets and sent events to its neighbors announcing that it was
going to shutdown. This caused a delay until the other reflectors were able to
reconfigure their inputs and outputs. In a wide-area network, the delays would
be even larger and the quality of service as perceived by the end-user would
be degraded significantly.

We solved this problem by creating a new thread to manage the reconfiguration.
While the new thread contacts the Reflector's sources and destinations to
announce the end of its service, the old thread continues to perform the
Reflector's normal packet-forwarding operations. The Reflector only shuts
itself down after it receives a confirmation from its neighbors that its
service is no longer needed\footnote{If this confirmation does not arrive, the
Reflector administrator or a controller program still has the chance to force
the termination of the Reflector program.}.

One can observe another problem in the graph of Figure
\ref{fig:badreconf1200}; as the reflector recovers, there is a sudden change
in the packet inter-arrival time, which approaches zero for a couple of
packets in a row. This happens because the end-user receives some repeated
packets that are sent both by the ``backup'' connection being deactivated and
the new, ``recovered'' connection.
This problem of  duplicated packets is solved very easily by implementing a
new subclass of \texttt{Connection} that uses the RTP sequence number to drop
repeated packets.

Figure \ref{fig:goodreconf1200} shows the result of running the same
experiment after implementing these two improvements. One can see that the
reconfigurations take place without affecting the quality of service as
perceived by the end-user (one can say that because there is no correlation
between the arrows in Figure \ref{fig:goodreconf1200} and the variations in
packet inter-arrival times).  Furthermore, since the old Reflector keeps a
thread performing the Reflector's normal operations until the reconfiguration
is completed, the quality of service is not degraded even in wide-area
networks where the latency to contact the neighboring Reflectors may be large.

\begin{figure}[htbp]
\centerline{\psfig{file=figs/reflectorGraf5.ps,width=10cm}}
\caption{Reflector Reconfigurations. Bandwidth =  1.2Mbps, Packet Rate = 30pps}
\label{fig:goodreconf1200}
\end{figure}

The lack of any correlation between the Reflector reconfigurations and the
quality of service is even more clear in Figure \ref{fig:goodreconfbusy1200}
that shows a similar experiment carried out in a period of high
network and machine loads. In this case, we set $K$ to 10 so that the
TestServer would keep its transmission rate constant for longer periods.

\begin{figure}[htbp]
\centerline{\psfig{file=figs/reflectorGraf6.ps,width=10cm}}
\caption{High Network and CPU Load. Bandwidth = 1.2Mbps,
Packet Rate = 30pps}
\label{fig:goodreconfbusy1200}
\end{figure}

Finally, Figure \ref{fig:vat} shows one more experiment in which we used the
Reflector network to distribute an RTP\index{RTP} stream generated by the vat
audioconference tool for the MBone \cite{mbone-book96}.  The acoustic
perception of five users listening to the audio stream in our laboratory
confirms what one sees in the graph: the reconfigurations do not affect the
quality of service of the audio application.
\begin{figure}[htbp]
\centerline{\psfig{file=figs/reflectorGraf7.ps,width=10cm}}
\caption{Reflector Reconfigurations. Vat Bandwidth = 45kbps, Packet Rate = 25pps}
\label{fig:vat}
\end{figure}

\index{Reflector!experiments|)}

\subsection*{Discussion}

The experiments described in this section addressed only one of the many ways
in which component configurators can be helpful. Nevertheless, they show how
the reification of dependencies can facilitate the implementation of reliable
systems with little or no performance penalty. In fact, we believe that, in
some cases, a proper dependence management can actually enable performance
optimizations \cite{sane:optimization}.


\section{Mobile Agents for Reconfiguration, Inspection, and \\ Code Distribution}
\label{sec:experiments-agents}
\index{agents!experiments|(}

In order to evaluate the response time and relative performance gains
made possible by our mobile agent based reconfiguration infrastructure, we
established an intercontinental testbed with the collaboration of the
Departments of Computer Science at the Rey Juan Carlos University in Madrid,
Spain and at the Campinas State University in Campinas, Brazil. The testbed
consisted of the following three groups of machines.
\begin{enumerate}
\item 2 Sun Ultra-60 and 1 Sun Ultra-5 machines running Solaris 7 in the
\url{cs.uiuc.edu} domain.
\item 3 Pentium-based 333MHz PCs running Linux RedHat 6.1 in the \url{escet.urjc.es}
domain.
\item 3 Pentium-based 300MHz PCs running Linux RedHat 6.1 in the
\url{ic.unicamp.br} domain.
\end{enumerate}
The machines inside each group were connected by 100Mbps Fast Ethernet
networks, while the groups were connected among themselves through the public
Internet.  We executed several instances of a test application running on top
of \dTAO and injected different kinds of agents in this network.

To avoid drastic oscillations in the available Internet bandwidth and latency,
and to minimize undesired interference, we carried out the experiments during
the night\footnote{When we ran the experiments during times of high network
traffic and congestion, the performance numbers were even more in favor of our
mobile agents approach.}. We ran all the experiments between 11:30PM on
Tuesday November 23, 1999, and 1:00 am in the following day (USA CST). Thus,
the starting time in Brazil and Spain were 3:30 am and 6:30 am, respectively,
and the Internet traffic was relatively low. Table
\ref{tab:net} shows the average bandwidth (measured by transferring a 100Kbyte
file via FTP five times) and latency (measured with the \method{ping} command)
between our lab in Illinois and the remote ones.

\begin{table}[htbp]
\begin{center}
\begin{tabular}{|l|c|c|}
\hline
                    & \url{escet.urjc.es} & \url{ic.unicamp.br} \\
\hline\hline
available bandwidth & 76 Kbytes/sec       & 32 Kbytes/sec     \\
\hline
round-trip latency  & 170 $ms$            & 270 $ms$          \\
\hline
\end{tabular}
\end{center}
\caption{Average Bandwidth and Latency from \texttt{cs.uiuc.edu}}
\label{tab:net}
\end{table}


Table \ref{tab:inspection} shows the total round-trip time to execute
five DCP\index{DCP} inspection commands (\method{list\_categories},
\method{list\_imple\-mentations}, \method{list\_loaded\_implementations},
\method{list\_domain\_components}, and \method{get\_comp\_info}) using different
techniques. In all the subsequent tables and graphs, each number is the
arithmetic mean of ten runs of each experiment and the numbers in parentheses
(vertical bars in the graphs) are the standard deviations.
\begin{table}[htbp]
\begin{center}
\begin{tabular}{|l|c|c|}
\hline
                                & 9 ORBs      & 1 ORB \\
\hline\hline                                          
Agents using distribution tree  &  847(33)    &           \\
\hline
Agents point-to-point           &  2127(1044) & 362(10)   \\
\hline
Client/Server                   &             & 1128(141) \\
\hline
\end{tabular}
\end{center}
\caption{Elapsed Time (in $ms$) for the Execution of Five Different Inspection Commands}
\label{tab:inspection}
\end{table}
The first column shows the elapsed times for the execution of the agents in 9
instances of the \dTAO ORB, 3 in each domain. The second column shows the
times for execution in a single ORB in a remote node.
The first line shows the time for agents following the distribution tree
shown in Figure~\ref{fig:adt}.  The second line shows the times for agents
sent directly from a single node in \url{cs.uiuc.edu} for parallel execution
on the other eight nodes as shown in Figure~\ref{fig:ptp}.
Finally, the third line shows the time for executing the same five inspection
commands by using five consecutive requests in a traditional client/server
model rather than using the agent model.

\begin{figure}[htb]
\mbox{
\subfigure[Using a tree]
{\label{fig:adt}%
    \leavevmode \epsfig{file=figs/distribution.ps,width=8cm}}
\subfigure[Using point-to-point connections]
{\label{fig:ptp}%
\leavevmode \epsfig{file=figs/ptp.ps,width=8cm}}
}
\caption {Agent Distribution Topology}
\end{figure}

Looking at Table \ref{tab:inspection}, we can see that by using a well-thought
distribution tree, we obtained a 60\% performance improvement in relation to a
point-to-point distribution scheme (847 $ms$ against 2127 $ms$). Using an
agent containing the five commands, instead of a separate request for each
command, led to a 68\% performance improvement (362 $ms$ against 1128 $ms$).

To measure how the benefits of using a distribution tree as the one in Figure 
\ref{fig:adt} varies with the size of the agent, we sent a series of
agents carrying the code for components to be installed in the remote
nodes. As shown in Figure \ref{fig:graf1}, as the size of the component being
uploaded increases, the relative gain of using a distribution tree instead of
point-to-point connections increases significantly. This shows clearly the benefits
of using the mobile agents infrastructure for code distribution.

\begin{figure}[htbp]
  \begin{center}
    \leavevmode \epsfig{file=figs/graf1.eps,width=11cm}
    \caption{Agent Uploading a New Component to Nine Nodes}
    \label{fig:graf1}
  \end{center}
\end{figure}

Figure \ref{fig:graf2} shows the elapsed time for executing agents carrying
from one to eight reconfiguration commands. 

\begin{figure}[htbp]
  \begin{center}
    \leavevmode \epsfig{file=figs/graf2.eps,width=12cm}
    \caption{Reconfiguration Agent Visiting Nine Nodes}
    \label{fig:graf2}
  \end{center}
\end{figure}



Finally, Figure \ref{fig:graf3} shows the comparative performance between the
agent approach and conventional client/server requests as the number of
requests increases from one to eight. The client/server times were measured by
building an application that sends the commands contained in the agent as
separate requests transmitted via TCP/IP connections. Since the overhead of
processing this kind of agent (around 1$ms$) is negligible compared to the
latency of long distance Internet lines, the performance of both approaches
when sending a single command is roughly the same. As we increase the number
of commands in each experiment, the total completion time for the agent barely
varies while the completion time for the client/server application increases
rapidly as expected.

\begin{figure}[htbp]
  \begin{center}
    \leavevmode \epsfig{file=figs/graf3.eps,width=12cm}
    \caption{Agents Versus Conventional Client/Server}
    \label{fig:graf3}
  \end{center}
\end{figure}

\subsection*{Discussion}

Although our implementation could still be improved significantly with more
tuning and optimizations, these results are very encouraging.  They
demonstrate clearly that mobile agents can provide extreme performance
improvements for the management and reconfiguration of wide-area distributed
systems.

\index{agents!experiments|)}