\chapter{Application Scenarios}
\label{chap:applications}

The work on this thesis was motivated initially by our research on the
\twoK operating system \cite{kon00:hpdc}. During the early phases of that
research, we identified proper dependence management as a key factor in the
development of robust operating system and middleware architectures. We begin
this chapter describing \dTAO, a fundamental part of \twoK, and explaining how
the \dTAO internal architecture applies the ideas defended in this thesis.

Section \ref{sec:reflector} focuses on a Multimedia Distribution System we
developed some years ago. It shows how we enhanced the system recently using
the infrastructure for automatic configuration and component configurators
to enable dynamic reconfiguration in the occurrence of failures.

In Section \ref{sec:other-apps}, we refer to systems being developed by other
researchers that are applying some of the ideas presented in this thesis.

\section{\dTAO}
\label{sec:dtao}
\index{dynamicTAO@\dTAO|(textbf}

One of the major constituent elements of the \twoK distributed operating
system \cite{twoK:wroops98,2K:home,kon00:hpdc} is a reflective middleware
infrastructure based on CORBA. After carefully studying existing CORBA Object
Request Brokers, we came to the conclusion that the TAO ORB
\cite{schmidt:tao-patterns99} would be the best starting point for developing
this infrastructure.

TAO is a portable, flexible, extensible, and configurable ORB based on design
patterns. It uses the \emph{Strategy} design pattern \cite{ralph:patterns} to
encapsulate different aspects of the ORB internal engine. A configuration file
is used to specify the strategies the ORB uses to implement aspects such as
concurrency, request demultiplexing, scheduling, and connection management. At
ORB startup time, the configuration file is parsed and the selected strategies
are loaded.

TAO is primarily targeted for static hard real-time applications such as
avionics systems \cite{tao:avionics}. Thus, it assumes that, once the ORB is
initially configured, its strategies will remain in place until it completes
its execution. There is no support for on-the-fly reconfiguration.
 
On-the-fly adaptation\index{adaptation} is extremely important for a wide
range of applications including those dealing with multimedia, mobile
computers, and dynamically changing environments.  To support this kind of
dynamic adaptations, we developed \dTAO \cite{dTAO:icdcs99,kon00:dtao}, an
extension of TAO that enables on-the-fly reconfiguration of its
strategies. \dTAO exports an interface for loading and unloading modules into
the ORB runtime, and for inspecting the ORB configuration state. The
architecture can also be used for dynamic reconfiguration of user applications
running on top of the ORB and even for reconfiguring non-CORBA applications.

\subsection{A Reflective ORB}
\label{sec:reflective}
\index{reflection!reflective ORB}

\dTAO is our first complete implementation of a CORBA reflective ORB.
As pointed out in \cite{singhai97:reflectiveORB,singhai98:qw}, a
\emph{reflective} system is a system that gives a program access to its
definition and evaluation rules, and defines an interface for altering
them. In an ORB, client requests represent the ``program'' to be evaluated by
the system, the ORB implementation represents the ``evaluator'', and
``evaluation'' is simply the execution of the remote method invocation.  A
reflective ORB makes it possible to redefine its evaluation semantics.

\dTAO allows inspection and reconfiguration of its internal engine and
allows ORB and application developers to specify reconfiguration policies
inside customized subclasses of the \class{ComponentConfigurator} class. It
exports an interface for (1) transferring components across the distributed
system, (2) loading and unloading modules into the ORB runtime, and (3)
inspecting and modifying the ORB configuration state.

The reification of \dTAO's internal structure is achieved through a collection of
component configurators.\index{component configurator}
Each process running the \dTAO ORB contains a component
configurator instance called \texttt{DomainConfigurator}. It is responsible
for maintaining references to instances of the ORB and to servants running in
that process.  In addition, each instance of the ORB contains a customized
component configurator called \texttt{TAOConfigurator}.

\begin{figure}[htb]
  \begin{center}
    \leavevmode \epsfig{file=figs/Configurators.eps,width=10cm}
    \caption{Reifying the \dTAO Structure}
    \label{fig:Configurators}
  \end{center}
\end{figure}

\texttt{TAOConfigurator} contains hooks\index{hooks} to which implementations of \dTAO 
strategies are attached. Hooks work as ``mounting points'' where specific
strategy implementations are made available to the ORB.  We currently support
hooks for different kinds of strategies such as Concurrency, Security\index{security}, and
Monitoring. The association between hooks and component implementations can be
changed at any time, subject to safety constraints.

Figure \ref{fig:Configurators} illustrates this reification mechanism in a
process containing a single instance of the ORB.  If necessary, individual
strategies can use component configurators to reify their dependencies upon
ORB instances and other strategies. These configurators may also store
references to client connections that depend on the strategies. With this
information, it is possible to manage strategy reconfiguration
consistently\index{consistency} as we explain in section \ref{sec:consistency}.

The \dTAO architectural framework is depicted in Figure \ref{fig:framework}.
The \emph{Persistent Repository} stores category implementations in the
local file system.  It offers methods for manipulating (e.g., browsing,
creating, deleting) categories and the implementations of each category. Once
a component implementation is stored in the local repository, it can be
dynamically loaded into the process runtime.
\begin{figure}[htb]
  \begin{center}
    \leavevmode \epsfig{file=figs/framework.eps,width=12cm}
    \caption{\dTAO Components}
    \label{fig:framework}
  \end{center}
\end{figure}

A \emph{Network Broker} receives reconfiguration requests from the network and
forwards them to the \emph{Dynamic Service Configurator}. The latter contains
the \texttt{DomainConfigurator} (shown in Figure~\ref{fig:Configurators}) and
supplies common operations for dynamic configuration of components at
runtime. It delegates some of its functions to specific component
configurators (e.g., \class{TAOConfigurator} or a certain \class{ServantConfigurator}).

We minimized the changes to the standard ACE/TAO distribution by delegating
 some of the basic configuration tasks to components of the ACE framework such
 as the \texttt{ACE\_Service\_Config} (used to process startup configuration files
 and manage dynamic linking) and the \texttt{ACE\_Service\_Repository} (to manage
 loaded implementations) \cite{ACE-Service-Config}.

This architectural framework enables the development of different kinds of
persistent repositories and network brokers to interact with the  Dynamic
Service Configurator. Thus, it is possible to use different naming schemes
when storing category implementations and different communication protocols
for remote configuration as described below.

We built the \dTAO components using the ACE wrappers \cite{ACE-Wrappers}
for operating system services. Thus, \dTAO runs on the several different
platforms to which ACE was ported.


\subsection{Reconfiguration Interfaces}
\label{sec:reconfiguration-interfaces}

\dTAO supports three distinct forms of reconfiguration interfaces. In general
terms, they all provide the same functionality, but each of them has
characteristics that makes it more or less appropriate for certain
situations. A description of the interfaces follows.

\begin{enumerate}
\item The \textbf{DCP Broker} is a customized subclass of the \emph{Network
Broker} shown in Figure \ref{fig:framework}. It listens on a TCP port, 
waiting for connection requests from remote clients. Once a connection is
established, a client can send inspection and reconfiguration commands using
DCP, our Distributed Configuration Protocol\index{DCP|textbf} \cite{DCP}. This
interface is particularly good for debugging and for fast interaction with an ORB,
since the user can access the configuration interface simply by establishing a
telnet connection to the DCP Broker.
\item The \textbf{Reconfiguration Agent Broker} is also a customized subclass
of the \emph{Network Broker}. It is useful for configuring a
distributed collection of ORBs as we described in Chapter \ref{chap:reconfig-agents}.
\item The \textbf{DynamicConfigurator} is a CORBA object that exports an IDL
interface with operations equivalent to those offered by the DCP
protocol. It is the most convenient of the three interfaces for programmatic
interactions, since all the communication aspects are hidden by the CORBA
middleware.
\end{enumerate}

We now use the DynamicConfigurator IDL specification presented in
Figure~\ref{fig:DynamicConfigurator} to explain the functionality of the \dTAO
reconfiguration interfaces\footnote{To make
Figure~\ref{fig:DynamicConfigurator} more clear, we omitted the exceptions
that each operation can raise.}.

\begin{figure}[htbp]
{
\small
\begin{center}
\begin{boxedverbatim}
interface DynamicConfigurator
{
 typedef sequence<string> stringList;   
 typedef sequence<octet> implCode;

 stringList list_categories ();
 stringList list_implementations (in string categoryName);
 stringList list_loaded_implementations ()
 stringList list_domain_components ();
 stringList list_hooks (in string componentName);        
 string     get_impl_info      (in string implName);
 string     get_comp_info      (in string componentName);
 string     get_hooked_comp    (in string componentName, 
                                in string hookName);
 string     get_latest_version (in string categoryName);

 long load_implementation (in  string categoryName, 
                           in  string impName, 
                           in  string params
                           in  Configuration::Factory factory,
                           out Configuration::ComponentConfigurator cc);
 void hook_implementation (in string loadedImpName, 
                           in string componentName, 
                           in string hookName);

 void suspend_implementation   (in string loadedImpName);
 void resume_implementation    (in string loadedImpName);
 void remove_implementation    (in string loadedImpName);
 void configure_implementation (in string loadedImpName, 
                                in string message);

 void upload_implementation   (in string categoryName, 
                               in string impName, 
                               in implCode binCode);
 void download_implementation (in string categoryName, 
                               inout string impName, 
                               out implCode binCode);
 void delete_implementation   (in string categoryName, 
                               in string impName);
};
\end{boxedverbatim}
\end{center}
}
\caption{The \class{DynamicConfigurator} Interface}
\label{fig:DynamicConfigurator}
\end{figure}

The \class{DynamicConfigurator} interface specifies the operations that can be
performed on \dTAO abstractions, namely, categories, implementations, hooks,
and configurable components.  The first nine operations in the interface are
used to inspect the dynamic structure of that domain and retrieve information
about the different abstractions.  A \emph{category} represents the type of a
component; each category typically contains different
\emph{implementations}, i.e., dynamically loadable code stored in the
Persistent Implementation Repository. 
For example, a category called \texttt{Concurrency}
contains the three threading models that \dTAO currently supports:
\texttt{Reactive\_Strategy}, \texttt{Thread\_Strategy}, and 
\texttt{Thread\_Pool\_Strategy}.

Once an implementation is loaded into the system runtime, it becomes a
\emph{loaded implementation} and can be associated with a logical
\emph{component} in the ORB domain. Finally, components have \emph{hooks}
that are used to represent inter-component dependence; if a component $A$
depends upon component $B$, then this dependence is represented by attaching
$B$ to a hook in $A$.

\texttt{load\_implementation} dynamically loads and starts an implementation
from the persistent repository. \texttt{hook\_implementation} attaches it to a
hook in one of the components in the domain.

The next four methods allow operations on loaded implementations. It is
possible to suspend and resume their main threads, remove them from the process,
and send them component-specific reconfiguration messages.

\texttt{upload\_implementation} allows an external entity to send
an implementation to be stored in the local Persistent Repository, so that it
can be linked to a running process and attached to a hook. Conversely,
\texttt{download\_implementation} allows a remote entity to retrieve an
implementation from the local Persistent Repository. Finally,
\texttt{delete\_implementation} is used to delete implementations stored at
the ORB Persistent Repository.

Consider now the scenario in which a user wants to change the threading model
at runtime by using an implementation of the Concurrency strategy called
\emph{Thread\_Pool\_Strategy}.  Assuming that the user wants to start with a
thread pool of size 20, the required configuration steps are the following.
\begin{enumerate}
\item Load the implementation into memory: \\
{\small \texttt{version = load\_implementation("Concurrency","Thread\_Pool\_Strategy","20", 0, cc)}}
\item Attach the implementation to the \emph{Concurrency} hook in TAO: \\
{\small \texttt{hook\_implementation("Concurrency"+version,"TAO","Concurrency\_Strategy")}}
\end{enumerate}
After the new implementation is attached, the ORB starts using it. In section 
\ref{sec:consistency}, we discuss what happens if a different concurrency
strategy is in use. 

Figure \ref{fig:ExCode} shows C++ code that uses the Dynamic
Configurator to retrieve and print some information about the ORB internal
configuration. The code obtains a reference to the \class{DynamicConfigurator} object 
through the ORB's \method{resolve\_initial\_references()} method.

\begin{figure}[htb]
\begin{center}
{\small
\begin{boxedverbatim}
 CORBA::Object_var           dcObj;
 DynamicConfigurator_var     dynConf;
 CORBA::ORB_var              orb;

 orb     = CORBA::ORB_init (argc, argv);
 dcObj   = orb->resolve_initial_references ("DynamicConfigurator");
 dynConf = DynamicConfigurator::_narrow (dcObj.in ());

 stringList *list = dynConf->list_implementations ("Concurrency");

 printf ("Available concurrency strategies:");
 printStringList (list);

 char *ret = dynConf->get_hooked_comp ("TAO", "Concurrency_Strategy");

 printf ("Now, using the <%s> concurrency strategy.", ret);
\end{boxedverbatim}
}
\end{center}
\caption{Inspecting the ORB Internal State}
\label{fig:ExCode}
\end{figure}


% MAYBE INCLUDE EXAMPLE FROM MIDDLEWARE SLIDE HERE! (THE SECURITY ONE).

\subsection{Consistency}
\label{sec:consistency}
\index{consistency}

Reconfiguring a running ORB while it is servicing client requests
is a difficult task that requires careful consideration. There are
two major problems.

Consider the case in which \dTAO receives a request for replacing one of its
strategies ($S_{old}$) by a new strategy ($S_{new}$).  TAO strategies are
implemented as C++ objects that communicate through method invocations. The
first problem is that, before unloading $S_{old}$, the system must guarantee
that no one is running $S_{old}$ code and that no one is expecting to run
$S_{old}$ code in the future. Otherwise, the system could crash. Thus, it is
important to assure that $S_{old}$ is only unloaded after the system can
guarantee that its code will not be called.

The second problem is that some strategies need to keep state
information. When a strategy $S_{old}$ is being replaced by $S_{new}$, part of
$S_{old}$'s internal state may need to be transfered to $S_{new}$. Both
problems can be addressed with the help of the \class{TAOConfigurator}.

Consider, for example, the three concurrency strategies supported by
\dTAO: single-threaded reactive, thread-per-connection, and 
thread-pool.  If the user switches from the reactive or thread-per-connection
strategies to any other concurrency strategy, nothing special needs to be
done. \dTAO may simply load the new strategy, update the proper
\class{TAOConfigurator} hook, unload the old strategy, and
continue. Old client connections  will complete with the concurrency policy
dictated by the old strategy. New connections will utilize the new policy.

However, if one switches from the thread-pool strategy to another strategy, we
must take special care. The thread-pool strategy we developed maintains a pool
of threads that is created when the strategy is initialized. The threads are
shared by all incoming connections to achieve a good level of concurrency
without having the runtime overhead of creating new threads. A problem arises
when one switches from this strategy to another strategy: the code of the
strategy being replaced cannot be immediately unloaded. This happens because,
since the threads are reused, they return to the thread-pool strategy code
each time a connection finishes. This problem can be solved by a
\class{ThreadPoolConfigurator} keeping information about which threads are
handling client connections and destroying them as the connections are
closed. When the last thread is destroyed the thread-pool strategy signals
that it can be unloaded.

Another problem occurs when one replaces the thread-pool strategy by a new
one. There may be several incoming connections queued in the strategy
waiting for a thread to execute them. The solution is to use the
\emph{Memento} pattern \cite{ralph:patterns} to encapsulate the old strategy
state in an object that is passed to the new strategy. An object is used to
encapsulate the queue of waiting connections. The system simply passes this
object to the new strategy, which then takes care of the queued connections.


\subsection{\dTAO and this Thesis}

It is interesting to note that, although the implementation of a reflective
ORB is not the main topic of this thesis, \dTAO is associated with the novel
ideas presented in this thesis in many distinct ways. 

The \dTAO internal structure is reified and managed with the C++
implementation of the component configurator framework. Our implementation of
the CORBA version of the component configurator was developed using \dTAO as
the underlying ORB. Finally, we developed the Component
Repository\index{component repository} and the Automatic Configuration Service
using \dTAO's \class{DynamicConfigurator} IDL interface depicted in Figure
\ref{fig:DynamicConfigurator}.

\dTAO is a good example of how the component configurator framework can be
used to represent inter-component dependence and how software developers can
use it to specify policies for safe dynamic configuration. 

\index{dynamicTAO@\dTAO|)}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Scalable Multimedia Distribution}
\label{sec:reflector}
\index{Reflector|(textbf}
\index{multimedia distribution system|see{Reflector}}

In \cite{icast98:vosaic} we describe the design and implementation of a
flexible multimedia distribution system and demonstrated that it is possible
to use the existing Internet to distribute low and medium bandwidth multimedia
to thousands of simultaneous users.  Our early experiments, however, pointed
out to difficulties in managing such a large-scale system and keeping it
available with an acceptable quality of service. It showed the necessity for a
better support for dynamic reconfiguration, distribution of code updates, and
provision of fault-tolerance for QoS-sensitive applications.

We addressed these problems in a new version of the multimedia distribution
system \cite{kcn:00:communications} built on top of the architecture presented
in this thesis.

\subsection{The Reflector}

The multimedia distribution system's key element is the \emph{Reflector}.  It
acts as a relay, receiving input data packets from a list of trusted sources
and forwarding these packets to other Reflectors or to programs executed by
end-users (also called \emph{end-user clients}). The distribution system is
composed of a network of Reflectors that collaborate with each other to
distribute the multimedia data over local-area, metropolitan-area, and
wide-area networks.

The Reflector is a user-level program that, at the application level, performs
activities similar to those performed by network routers at the hardware
level (which is where protocols such as IP-Multicast
\cite{deering:host-extensions} are implemented). Since it is implemented in
software, not hard-wired into the router, the Reflector is more flexible, easy
to deploy and evolve, and can be dynamically customized to users,
applications, and environments.

Reflector data packets are encoded with RTP\index{RTP|textbf}, a user-level
protocol for real-time applications \cite{rtp:00} defined by the Internet
Engineering Task Force (IETF). RTP packets can be transmitted over different
kinds of low-level transport protocols such as TCP/IP, UDP/IP, and
IP-Multicast.

The Reflector Network topology is determined by each Reflector's
configuration. This information specifies input and output connections, access
privileges, maximum allowed number of users, etc. It is stored in a database
controlled by the Reflector administrator. Figure \ref{fig:reflector-net}
depicts a generic Reflector network that distributes two video streams in
different channels. In this figure, two capture stations send their video
streams to ``master'' Reflectors; the streams may traverse several
``intermediate'' Reflectors until they reach ``public'' Reflectors to which
end-user clients can connect and receive the video streams\footnote{The
classification of Reflectors as ``master'', ``intermediate'', and ``public''
Reflectors is merely pedagogical, they execute exactly the same program.}. All
the Reflectors in the figure are initiated with exactly the same code. Using
the Automatic Configuration Service described in Chapter
\ref{chap:autoconfig-service}, the system then customizes each
Reflector according to their individual requirements.

\begin{figure}[htb]
\centerline{\psfig{file=figs/reflector2streams.eps,width=12cm}}
\caption{A Reflector Network Distributing Two Video Streams}
\label{fig:reflector-net}
\end{figure}

\subsection{Data Distribution Protocols}
\label{sec:connection}

In order to support different types of inter-reflector communication protocols
transparently, the Reflector framework encapsulates the concept of a network
connection into an abstract C++ class named \texttt{Connection} that defines
the basic interface for all types of network connections used by the
Reflector. This abstract class implements some of its own methods, but the
majority of the connection-related code is implemented by its subclasses:
\texttt{TCPConnection}, \texttt{UDPConnection}, \texttt{MulticastConnection},
and the like.

Figure \ref{fig:concrete-reflector-net} depicts a concrete example of a
highly-heterogeneous Reflector network. In this example, the network
distributes two audio-visual streams. The first comes from a mobile camera,
mounted on a helicopter, that sends its stream to a ``master'' Reflector over
a wireless link. This kind of link presents a high rate of packet loss not
related to congestion, which makes protocols like TCP perform poorly
\cite{tcp-wireless96}. Thus, it is desirable to use a protocol optimized for
this kind of situation like, for example, WTCP \cite{wtcp99}. The second
stream is sent to its ``master'' Reflector through a dedicated ISDN line. To
optimize the bandwidth, one can use UDP as the communication protocol
since its overhead is lower than that of TCP and the link offers a low loss
rate.

\begin{figure}[htb]
\centerline{\psfig{file=figs/concrete-reflector-net.eps,width=14cm}}
\caption{A Heterogeneous Reflector Network}
\label{fig:concrete-reflector-net}
\end{figure}

Continuing with the example, Reflector C sends its streams to Reflector D over
the public Internet through a transatlantic satellite link.  Even
though this is a high-bandwidth link, its loss rate may be high, so it is more
appropriate to use TCP.  Reflectors B and E offer the multimedia streams
to dial-up clients over conventional phone lines; Reflector B uses TCP while
Reflector E, on the other side of the Atlantic, uses the VDP adaptive
algorithm.  Finally, Reflectors A and D introduce the video streams into two
distant points of the global MBone and Reflector F uses multicast to
distribute the streams in a local Ethernet network.

Selecting the appropriate protocol for each situation, the Reflector
administrators improve the quality of service offered to the end-users,
optimizing the utilization of the available network bandwidth and minimizing
packet losses.

Most of the Reflector's code deals with objects of type \texttt{Connection}
and is not aware of the \texttt{Connection}'s underlying implementation. The
actual connection type is specified when the connection is created and does
not need to be known after that.

This approach allows programmers to plug in customized \texttt{Connection}
subclasses by providing their own implementation of the \texttt{Open},
\texttt{Close}, \texttt{Bind}, \texttt{Connect}, \texttt{Send}, and
\texttt{Receive} methods, adding specialized functionality.

In this manner, it is possible to incorporate, into the Reflector,
\texttt{Connection} subclasses that implement different transport protocols
(such as the VDP QoS-aware\index{QoS} adaptive protocol for the Internet \cite{vosaic}
and the xbind QoS-aware reservation protocol for ATM networks
\cite{xbind99:atm-qos}). Developers also use this mechanism to implement
\texttt{Connection} subclasses that perform various operations on the data,
such as encryption, transcoding, mixing, downsampling, and the like). Finally,
one can create composite \texttt{Connection} types by combining existing
ones. For example, one can create a \texttt{CryptoMulticast} connection type
-- that encrypts the data and sends them out using Multicast -- by combining a
\texttt{Crypto} connection with a \texttt{Multicast} connection.

\subsection{Experience and Lessons Learned}
\label{sec:lessons}

This technology was utilized in the live broadcast of NASA's JPL Pathfinder
mission \cite{nasa:pathfinder}. During this broadcast -- which lasted for
several months -- more than one million live video sessions were delivered to
dozens of different countries across the globe by a network of more than 30
reflectors spread across five continents. The Reflectors ran in five different
operating systems (Solaris, Linux, Irix, FreeBSD, and Windows) and transmitted
their streams over different kinds of network links. End-users could watch the video
stream simply by pointing their web browsers to selected locations, causing
their browsers to download a Java applet containing the video client. The applet
connected automatically to the reflector, received the video stream, decoded it,
and displayed it to the user in real-time.

During this broadcast, we experienced three major problems:

\begin{enumerate}
\item As the code had not been tested on such a large scale and on so many
different platforms, we found many programming errors both in the Reflector
code and in the client applet. Surprisingly, fixing the error was, sometimes,
easier than updating the code in the dozens of machines that formed
the distributed system. The same problem occurred when a new version of the
Reflector, with added functionality, was released. System administrators had
to manually connect to dozens of machines, upload the new code, shutdown the
old version, and start the new one.

\item In many cases, we had to reconfigure the reflector network by
dynamically changing the distribution topology, or by setting new values to
the reflector configuration parameters (e.g., maximum number of users, number
of multimedia channels, etc.). The configuration information for the
reflectors was stored in a centralized location. After updating this
centralized database, we had to connect to each of the reflectors and instruct
them to download their updated configuration. This process was tiresome and
error-prone.

\item The only mechanism the Reflector provided to support fault-tolerance
\index{fault-tolerance} was to send redundant streams from different sources
to the same Reflector (see \cite{icast98:vosaic} for details). This mechanism
leads to a large waste of bandwidth. The redundant streams are always
transmitted even though they are seldom used.
\end{enumerate}

With this experience, we learned that a user-level Reflector system is,
indeed, a powerful tool for managing large-scale multimedia distribution. It
gives Reflector administrators tight control over the distribution, allowing
for better control of the quality of service. It achieves that through the
definition of the distribution topology, the selection of appropriate
communication protocols, and the possibilities for limiting the number of
clients according to the available resources.

We also learned, however, that it is important to provide better mechanisms
for distributed code updates, dynamic reconfiguration, and fault-tolerance,
which motivated us to develop the architecture described in this thesis.

\subsection{Dynamic Configuration of QoS-Sensitive Systems}
\label{sec:dynconf-qos}
\index{dynamic (re)configuration}
\index{QoS}

The synergistic relationships between dynamic configuration and QoS are
clear. Dynamic configuration allows the use of the best policies for each
situation. For example, a mobile computer displaying a video clip to its user
could use a protocol optimized for wireless connections (e.g., WTCP
\cite{wtcp99}) when the computer is using a wireless link, but dynamically 
reconfigure itself to use a TCP connection when the computer is hooked to a
wired Ethernet connection.

However, if the reconfiguration process itself affects the quality of service
negatively, it may not be worthwhile to do any reconfiguration at all.  Going
back to the example, if the dynamic reconfiguration to the TCP connection is
so expensive that the video is interrupted for several seconds, it is better
to keep using the wireless link, even when the wired link becomes available.

Therefore, while developing the new version of the Reflector system, our major
goal was to deploy our architecture for dependence management to support
dynamic configuration and fault-tolerance \emph{without} affecting quality of
service negatively.

\subsubsection{Automatic Configuration}
\label{sec:auto-config-implement}
\index{automatic configuration}

To solve the problem of maintaining the Reflector instances up-to-date as the
code of the Reflector program evolves, and to customize each Reflector
according to its role, we used the Automatic Configuration Service described
in Chapter \ref{chap:autoconfig-service}.

The first major change we had to do in the Reflector implementation to
accommodate the new design was the adoption of a component-oriented model. We
reorganized the implementation of the Reflector program, breaking it into
dynamically loadable components. Surprisingly, this proved to be not so
difficult, thanks to the original object-oriented design of the Reflector that
was based on loosely coupled objects interacting via well-defined
interfaces. This component-based model facilitates the customization of the
Reflector program at startup, allowing the use of the Automatic Configuration
Service to select the components that are best suited for executing the
Reflector in a given node. It also facilitates the dynamic reconfiguration of
running Reflectors to adapt to changes in the environment and to install new
versions of components on-the-fly.

Figure \ref{fig:bootstrapping} presents a schematic overview of the
Reflector bootstrapping process in which the Reflector configures itself with
the help of our infrastructure.

\begin{figure}[htb]
  \begin{center}
    \leavevmode \epsfig{file=figs/bootstrapping.eps,width=12cm}
    \caption{Bootstrapping a Reflector}
    \label{fig:bootstrapping}
  \end{center}
\end{figure}

At startup time, each Reflector contacts the CORBA Name Service to locate the
Component Repository\index{component repository} that is compatible with the
operating system and hardware platform on which the Reflector is running
(steps 1 and 2 in Figure \ref{fig:bootstrapping}). From the Component
Repository, it retrieves its specific prerequisite specification file, which
may contain information about its resource requirements and the list of
components that must be dynamically loaded before the Reflector starts its
activities.

The Automatic Configuration Service library (denoted as ACS in Figure
\ref{fig:bootstrapping}) is linked to the Reflector process. Using the
prerequisite file fetched from the Component Repository, the ACS requests the
appropriate components from the remote repository (step 5), stores them in the
local disk (step 6)\footnote{To minimize startup time and network load, the
components fetched from the Component Repository may be cached locally.}, and
dynamically loads them into memory using the \dTAO facilities (step 7).

Once all components are loaded, the Reflector uses the resource requirement
information in the prerequisite file to request the reservation of CPU and
memory from an underlying QoS-aware\index{QoS} resource management
service\index{resource management}. This communication with the resource
management service (step 8) is the only part of the bootstrapping scheme that
was not implemented in our prototype.  QoS-aware resource management service
can be provided by systems like SMART \cite{smart97} or the Dynamic Soft
Real-Time Scheduler (DSRT) \cite{nahrstedt98:dsrt}. In the latter case,
DSRT\index{DSRT} can use the prerequisite specification to perform QoS-aware
admission control, negotiation, reservation, and scheduling.

Next, the Reflector registers itself with the Name Service, so that it can be
easily located by other system entities, and opens all the input and output
connections using the specified protocols (steps 9 and 10). Although our
prototype reads the information about the input and output connections from a
local configuration file, it could be easily modified to retrieve this
information from a file stored in the Component Repository.\index{component
repository}

Administrators of the Reflector system can look at the available broadcast and
videoconference sessions by using a graphical user interface that interacts
with the CORBA Name Service. For example, Figure \ref{fig:2kBrowser} is a
screen shot that illustrates the use of this interface. It shows three
independent Reflector networks: the first called \texttt{VirtualMeetingRoom}
that could be used for audio and videoconferencing, possibly divided according
to interest groups; the second called \texttt{News} could contain several news
channels; and the third called \texttt{OlympicGames} could contain audio and
video broadcast channels related to Olympic events.

\begin{figure}[htb]
  \begin{center}
    \leavevmode \epsfig{file=figs/2kBrowser.ps,width=12cm}
    \caption{A Sample Screen Shot of the Name Service GUI}
    \label{fig:2kBrowser}
  \end{center}
\end{figure}

The administrator can also see that there are three available component
repositories for three different kinds of architectures and can use the GUI to
upload, download, and remove Reflector components from the
repositories. Finally, the administrator can see that the
\texttt{OlympicGames} network is composed of the five reflectors shown in the
upper right-hand window. The CORBA IOR (Interoperable Object Reference)
of the Reflector at \texttt{delirius.cs.uiuc.edu} is shown in the bottom
right-hand window.


\subsubsection{Fault-Tolerance}
\label{sec:reflector-reconfig}
\index{fault-tolerance|(} 

An important aspect of quality of service is that of reliability. It is
difficult to ensure that the system will work properly in the presence of
network failures, node failures, software failures, and temporary system
shutdowns for maintenance. In addition, keeping the desired level of quality
of service irrespective of all these disruptive events is a major challenge
for modern QoS-sensitive systems. The Reflector architecture addresses this
problem by using architecturally-aware dynamic reconfiguration based on the
component configurator\index{component configurator} framework described in
Chapter \ref{chap:comp-config}.

In multimedia distribution, the goal with respect to fault-tolerance is to
maximize the system's availability without relying on redundant data
transmission, which leads to waste of bandwidth.  To achieve that, our
prototype supports the dynamic reconfiguration of the Reflector
inter-connections when failures occur. The distributed system has knowledge
of its own structure and is able to establish alternative routes to deliver
the multimedia streams to its users. Furthermore, whenever possible, the
system performs these reconfigurations without affecting the quality of
service perceived by its users.

\paragraph{Fault-Recovery Models}

In order to build an alternate distribution topology, the system must store
enough information so that alternate routes can be found when failures occur. 
The question is: where to maintain this information?

We considered initially a solution in which all the information regarding fault
recovery would be placed in a centralized database accessible by every
Reflector. When a Reflector $R_1$ detects that one of its inputs failed or that it 
is silent for
too long, it would contact a \emph{configuration server} with access to
the centralized database and request the address of a reflector $R_2$ from which
it could receive the missing streams. The configuration server would return
this information and contact $R_2$, reconfiguring it to send data to
$R_1$. The advantage of this approach is that very little information is
stored on the Reflectors, all the fault-recovery information and policies are
centralized in a single location, facilitating the manipulation of this
information by a single entity: the configuration server.

The second solution is to store fault-recovery information on
the Reflectors themselves. That is, each Reflector would store, for each set
of multimedia channels, a list of alternate Reflectors that could be used as
backups.  The advantage of this approach is that it does not lead to a single
point of failure and does not impose an extra load on a possibly already
overloaded configuration server. This solution may be more difficult to
implement, but it tends to be more scalable.

We believe that the optimal solution to this problem is one that encompasses
both models. On the one hand, each Reflector should have some knowledge about
its local surroundings and should be able to perform reconfigurations by
communicating with its immediate neighbors, without being a burden to the
centralized configuration server. On the other hand, the configuration server
should maintain global knowledge about the system topology. This centralized,
global knowledge should be used not only as backup, in case the Reflector's
localized information is not enough to keep the system functioning, but also
to perform optimizations, such as dynamic changes in the network topology to
improve the quality of service and promote load balancing.

Our architecture adopts the hybrid model described above.  It distributes the
knowledge throughout the Reflector network and makes each Reflector aware of
its dependence\index{dependence management} relationships with other
Reflectors.  Thus, the Reflectors are able to make reconfiguration decisions
by themselves, without relying on a centralized entity. In addition to this,
the global system topology is maintained in the configuration service so that
a Reflector administrator or an ``intelligent'' software module can perform
global optimizations in the distribution network.

The prototype supports fault-tolerance\index{fault-tolerance} by using a
subclass of \class{ComponentConfigurator} to represent the dependencies
between Reflectors.  When failures occur, the system uses the dependence
information to locate alternate routes and keep the system functioning. The
\class{ComponentConfigurator} implementation stores the dependencies as a list
of CORBA IORs, which allows for prompt communication no matter where the
objects are located.

The subclass of \class{ComponentConfigurator} we use is called
\class{ReflectorConfigurator} and contains the policies for reshaping the network
topology in case of failures and encapsulates all the code to deal with these
reconfigurations. This approach proved to be very effective in keeping a clear
separation of concerns in the Reflector code. The classes that deal with the
Reflector's normal operation are totally unaware of the
\class{ReflectorConfigurator} and of any code that deals with reconfiguration.
This clean separation also makes it easy to plug different kinds of 
\class{ReflectorConfigurator}s to support different reconfiguration policies.

\paragraph{Triggering Reconfigurations}

In this implementation, four kinds of events can trigger dynamic
reconfiguration:
\begin{enumerate}
\item A Reflector shutdown message sent by the Reflector
administrator or a \texttt{kill} command executed by the local system administrator. 
\item Errors (or ``bugs'') in the Reflector or library code that lead to a
segmentation fault or bus error.
\item A reconfiguration order sent by the Reflector administrator.
\item Sudden machine crashes or network disconnections.
\end{enumerate}

In the first two cases, the Reflector captures those events using signal
handlers installed with the UNIX \texttt{signal} function or the Windows
\texttt{SetConsoleCtrlHandler} function. In the UNIX implementation, for
example, the administrator can kill a Reflector by pressing
\texttt{Ctrl-C} on the terminal executing the Reflector, by sending a shutdown
message to the reflector using a telnet connection, or by using the
\texttt{kill} command. 
%(Control-c or ASCII ETX ) generates a  SIGINT signal.
The Reflector captures the events generated by \texttt{Ctrl-c}, \texttt{kill}, 
segmentation faults, and bus errors by implementing signal handlers for 
the SIGINT, SIGTERM, SIGSEGV, and SIGBUS signals, respectively.

In the third case, the Reflector contacts the configuration service to
retrieve its new configuration information and reprocesses it, reconfiguring
its input and output connections.

Finally, the fourth case is the only one in which it is not possible to keep
the client multimedia stream uninterrupted without relying on redundant
streams to the same Reflector. The solution in this case is to detect when the
input for a given channel has failed or is silent for too long and then locate an
alternative input either by using the local list of alternatives or by
contacting the configuration server.

Our current implementation focuses on supporting dynamic reconfiguration in the
presence of the first two kinds of events. We now describe this process in
more detail.

\paragraph{The Reconfiguration Process}

When an administrator (or other system entity) requests to kill a Reflector,
the system executes a special event handler called
\texttt{abandonReflectorNetwork ()}. This handler takes the following three
actions.

\begin{enumerate}
\item Unregisters the Reflector from the Name Service.
\item Using CORBA, sends a FINISHED event to the
\texttt{ReflectorConfigurator}s of all the sources (inputs) of this
Reflector; the event carries a list of the clients of the finishing reflector.
\item Using CORBA, sends a FINISHED event to the
\texttt{ReflectorConfigurator}s of all the clients (outputs) of this
Reflector; the event carries a list of the sources of the finishing reflector.
\end{enumerate}

When a \texttt{ReflectorConfigurator} receives a FINISHED event from a source
Reflector, it adds all the Reflectors in the list of sources of the finishing
Reflector to its list of inputs\footnote{This action may lead to redundant
inputs for the same channel. This can be avoided by extending the code of the
\texttt{ReflectorConfigurator} so that it adds all the new sources to its list
of input alternatives and chooses only one of them to be its new input.}.

Conversely, when a \texttt{ReflectorConfigurator} receives a FINISHED event
from a client Reflector, it adds all the Reflectors in the list of clients of
the finishing Reflector to its output list.

Figure \ref{fig:net1-before} shows a sample Reflector network where Reflector
C has two inputs and two outputs. When C is killed and the
reconfiguration process described above completes, the new configuration
becomes the one in Figure \ref{fig:net1-after}.

\begin{figure}[htb]
\centerline{\psfig{file=figs/net1-before.eps,width=12cm}}
\caption{A Distribution Network with Five Reflectors}
\label{fig:net1-before}
\end{figure}

\begin{figure}[htb]
\centerline{\psfig{file=figs/net1-after.eps,width=12cm}}
\caption{The Distribution Network After Reflector C Abandons the Network}
\label{fig:net1-after}
\end{figure}

In order to be able to carry out the reconfiguration without any glitches in
the multimedia streams and without affecting the system quality of service, we
had to adopt the multithreaded solution described in section
\ref{sec:experiments-dynreconf}.

\subsubsection{The Recovery Process}

When a Reflector starts its execution for the first time or when it is
restarted after being shutdown for some reason, it executes an initialization
process. In this process, in addition to performing the actions described in 
\ref{sec:auto-config-implement}, it performs the following three actions.

\begin{enumerate}
\item Registers the Reflector with the Name Service.
\item Using CORBA, sends a STARTED event to the
\texttt{ReflectorConfigurator}s of all the clients (outputs) of this
Reflector; the event carries a list of the sources of the new reflector.
\item Using CORBA, sends a STARTED event to the
\texttt{ReflectorConfigurator}s of all the sources (inputs) of this
Reflector; the event carries a list of the clients of the new reflector.
\end{enumerate}

Upon receiving a STARTED event from a new source Reflector, the client
Reflector opens a new input connection to the new Reflector. If it is also
receiving input from one of the sources of the new Reflector, it closes that
input connection as soon as the data from the new source is available.  An
analogous process happens upon receiving a STARTED event from a new client
Reflector.

These mechanisms allow the distribution system to recover its original
topology after a faulty reflector restarts. Therefore, if the system
configuration is the one in Figure \ref{fig:net1-after} and the Reflector
C recovers, then the configuration switches back to the one in Figure
\ref{fig:net1-before}.

Note that we do not have an automatic mechanism for restarting faulty
Reflectors. This requires the administrator's intervention, which seems to be
natural since a Reflector goes out of service either because of an
administrator's command or because of a failure in the system. In both cases,
the administrator's attention is advisable. Alternatively, if desired, the
Reflector can be added to the list of daemons that are executed by the
operating system when a machine boots, eliminating the need for manual
intervention when a machine crashes and restarts.

In section \ref{sec:experiments-dynreconf}, we present the results of several
experiments with the Reflector prototype. The performance evaluation shows
that, with the help of our infrastructure for dependence management, the
Reflector system is able to carry out dynamic reconfiguration of the
distribution topology without any negative impact on the quality of service
perceived by end-users. This dynamic reconfiguration provides a solid base for
fault-tolerance.\index{fault-tolerance} 

\index{fault-tolerance|)} 
\index{Reflector|)textbf}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Other Applications}
\label{sec:other-apps}

In addition to the use cases described in section \ref{sec:cc-use-cases},
three ongoing projects carried out by other researchers in our group are using
the products of this thesis.

\begin{enumerate}
\item \emph{Gaia}\index{Gaia} \cite{roman00:gaia}, a middleware infrastructure for
management of active spaces\index{active spaces} will use both the Automatic
Configuration Service and component configurators to manage dynamic,
heterogeneous devices in ubiquitous computing scenarios.  As a traditional
operating system manages the resources of a machine, Gaia manages the
resources in an active space. Examples of active spaces include active offices
and active lecture rooms enhanced with various computing and multimedia
devices. Interactions between the user and the devices in the space are
managed by a framework inspired in the Model-View-Controller paradigm
\cite{mvc88}. Gaia uses component configurators to manage the dependencies
among the Model, the Controller, and the multiple Views.

%In the context of Active Spaces the application model has to be redefined to
%accomodate heterogeneity, dynamic adaptation and large collections of
%devices that can be part of the application itself. As part of this
%redefinition we introduce a new model for applications. This model is
%inspired in the Model-View-Controller and introduces a fourth element called
%the Model Adapter. An application running in an Active Space consists of a
%model which contains the data, a view which presents the data in some way
%(in a display, using the speakers, modifying physical properties of the
%physical space, etc.), a controller that modifies the view or the model and
%finally a model adapter that adapts the data exported by the model to the
%data expected by the view. Note that this model adapter can imply complex
%data transformations.

%Models can have more than one view and model adapter attached. In the same
%way, a view can have more than one controller. This scenario requires the
%use of an observer pattern to preserve consistency whenever changes are
%introduced in the model, model adapter or view. We implement this observer
%pattern by using the component configurator. Components are the building
%blocs for any abstraction in an active space. Each component has an
%associated component configurator to enforce dependencies.

\item \emph{2KFS}\index{twokfs@\emph{2KFS}}, the \twoK distributed file 
service \cite{hess:adapt:00} will use the Automatic Configuration Service to
dynamically load components, implementing different algorithms for caching and
data transcoding. The component configurator framework will be used to manage
dependencies among the distributed \emph{2KFS} components.

\item $2K^Q$ is a reconfigurable, component-based QoS\index{QoS} framework 
which partitions the end-to-end QoS setup process into distributed QoS
compilation and runtime QoS instantiation phases
\cite{nahrstedt00:iwqos}. Dynamic instantiation is based on \dTAO and the
Automatic Configuration Service.
\end{enumerate}
