\chapter{Measurement Setup}
\label{chap:setup}

This chapter describes the setup used for configuring, and controlling the
experiments inside the lab. Furthermore,it is described how measurement data are
collected, processed and stored. The general software architecture is shown in
\autoref{fig:components}. The components are distributed across several
different platforms.

Measurement data for multiple parameters were collected from the network. The
variables and their data sources are listed in \autoref{tab:params}. The packet
captures can be substituted with outputs on the serial output. For telling the
energy consumption of an implementation, a \ac{PM} of one sample per second is
sufficient.

\begin{table}[h]
  \centering
  \caption{Measured variables at each node}
  \begin{tabular}{ll}
    \toprule
    Variable & Source \\
    \midrule
    Network latency & PCAP, serial output \\
    Delivery ratio & PCAP, serial output \\
    Control overhead & PCAP, serial output \\
    Number of DIO, DAO & PCAP, serial output \\
    DODAG state & serial output \\
    Routing table & serial output \\
    Node reset times & HN event log \\
    RSSI & HN radio receiver \\
    Energy consumption & HN power monitor \\
  \end{tabular}
  \label{tab:params}
\end{table}

When evaluating such a large set of variables from multiple different sources,
the amount of generated data and if enough processing power and storage space is
available has to carefully considered. \autoref{tab:amountdata} shows the
estimated amounts of data generated for each variable. For 50 runs this merely
amounts to 500 MByte of raw data, which can be further compressed using text
compression, since they, especially in the case of the serial log, contain many
repeating strings.

\begin{table}
  \centering
  \caption{Estimated amount of data generated by an experiment with 44 nodes,
    lasting 600 seconds}
  \begin{tabular}{lrr}
    \toprule
    Source & Single (bytes) & Total (KBytes)\\
    \midrule
    Serial output & 300 & 7920 \\
    Event log & 40 & 1056 \\
    Consumption log & 46 & 1214 \\
    RSSI log & 46 & 1214 \\
    \midrule
    Total & & 11404 \\
  \end{tabular}
  \label{tab:amountdata}
\end{table}

\section{Testlab Nodes}

As previously mentioned in \autoref{sec:architecture}, each node inside the
testlab is consists of the \ac{ON} that runs the firmware supplied by the user
and the \ac{HN} that controls and monitors the \ac{ON}. In this section, the
software running on the node and how data is collected from it is further
described.

\subsection{Open Node}

Each \ac{ON} inside the test lab runs a version of the \emph{Contiki} operating
system. The node at the root of the \ac{DAG} is programmed to act as a sink for
the \ac{UDP} traffic that is periodically emitted the other nodes that act as
sources. Each such packet contains a sequence number. The reception of these
packets at the sink and the emission from the source are logged to the serial
output of the node node. This makes it possible to detect whether a packet has
been received when comparing the log of the sink and that of the respective
sender.

Further entries inside the log on the serial output from each node include the
number of emitted \ac{DIO} and \ac{DAO} messages, changes of the routing table and changes
of the preferred parent.

Different versions of the firmware have been produced for the \ac{DAG} root and
all other nodes in the network. The firmware of the root creates a new
\ac{DAG} with the node itself at the root. All other nodes will attempt to join
this \ac{DAG}.

\subsection{Porting the Firmware}

The firmware supplied to the \ac{ON} comes in different versions which each have
different features enabled. The default configuration of \emph{Contiki} 2.7 with
added device support for the \emph{M3} node is provided by \fitlab. The hardened
implementation has been developed based on an older version of \emph{Contiki}, that
also did not include device support for the \emph{M3} node. In preparation for
the evaluation, the hardened implementation has been ported to this newer version
of \emph{Contiki} and some changes and additions had to be made to take into
account changed software interfaces and device specific differences.

Some build options specific to the \emph{Z1} where required for the hardened
implementation to function. These options have been added as necessary.

A missing consistency check has been discovered when restoring the routing state
from flash memory and has then been added that erases the state from the flash
if an inconsistency is discovered.

When testing the hardened implementation, it came to attention that a restart of
the \ac{DAG} root will result in it becoming unable to reconfigure as the
\ac{DAG} root, leading to the network being unusable until the state is purged
from the flash memory of the root node. This is caused by the root node
restoring the previously recorded \ac{DIO} messages from its flash and replaying
them onto the \ac{RPL} module before it configures itself as the root of a
\ac{DAG}. This leads to it joining an existing \ac{DAG} discovered from the
\ac{DIO} messages. This \ac{DAG} is the same \ac{DAG} as was the node previously
the root of. The node joins this \ac{DAG} and selects one of its nodes as a
preferred parent, setting its own rank to a rank larger than its parent. It is
likely that the new parent node previously had selected the former root node as
a parent. In this case, the new parent node of the former root node discovers
that its own parent has a larger rank than In this case, the new parent node of
the former root node discovers that its own parent has a larger rank than
itself. This triggers the former root node to be dropped as the parent at this
node. Thus the tree looses its root and becomes unusable but also unable to
recover, since the root node believes to be already part of a \ac{DAG} and will
not create a new \ac{DAG} with itself as the root node.

\subsection{Host Node}

The \ac{HN} collects the log output from the \ac{ON} and offers it to
connecting clients of a \ac{TCP} network socket. The \ac{HN} also can record
\acp{PCAP} in the direct vicinity of the \ac{ON} and forwards them in a similar
manner to the serial output. Alternatively, the \ac{HN} can record the local
\ac{RSSI}. For each event, including start, stop and reset of the attached
\ac{ON}, a corresponding entry is appended to an event log. The \ac{HN} can also
record accurate values for the power consumption, voltage and current from the
power management unit of the \ac{ON}.

For the purpose of measuring the energy consumption of the node in
reaction to a transient node failure, a high \ac{AM} can be selected since
\ac{RPL} takes some time to react to the change in the network, so the state
will persist for a multiple second. This way it is possible to reduce
the noise component and acquire more accurate data.

The \ac{HN} stores both the consumption data and the \ac{RSSI} in the
\ac{OML}. These log files are stored locally and periodically collected and
stored by the site server.

\section{Site Server}

At each site, a shared server is provided running a multi-user system
(\emph{Linux}). The server periodically queries the events, \ac{RSSI} and
consumption of each \ac{HN} and for each node in the form of log files, stores
them inside the home directory of the user who owns the experiment. The forwarded
serial output of the nodes and the output of the network sniffer are not stored
directly on the shared server because a large amount of data might be generated
this way. Instead, scripts are provided to aggregate the serial output and
stream of network packages into two aggregate streams of messages. These
streams can then be forwarded and stored for later analysis on the local
computer.

\section{Local Computer}

The local computer submits the experiment to the \ac{API}, controls the execution
of the experiment and collects the results from the site server. The data is
stored, processed and later displayed to the user.

\subsection{Orchestration Scripts}

For the purpose of configuring the experiment through the \ac{API}, an
extendable set of scripts has been created. The \texttt{run} script submits the
configuration of the experiment to the \ac{API}, monitors the state of the
experiment, sets up forwardings for the aggregated serial output and packet
captures and loads and calls into hook functions that can be specified per
experiment. The control flow of the script is shown in \autoref{fig:orchest}.

\begin{figure}[h]
  \centering
  \begin{tikzpicture}
    \tikzumlset{fill class=tubsLightBlue20, fill object=tubsLightBlue20, fill component=tubsLightBlue40}
    \begin{umlseqdiag}
      \umlobject[class=script]{run}
      \begin{umlcallself}[op=pre]{run}
      \end{umlcallself}
      \begin{umlcallself}[op=wait running]{run}
      \end{umlcallself}
      \umlobject[class=hook]{during}
      \begin{umlcall}[op=subshell,type=asynchron]{run}{during}
        \umlobject[class=Subshell]{track}
        \begin{umlcall}[op=subshell,type=asynchron]{run}{track}
          \begin{umlcallself}[op=serial aggregator,type=asynchron]{track}
            \begin{umlcallself}[op=sniffer aggregator,type=asynchron]{track}
            \end{umlcallself}
          \end{umlcallself}
        \end{umlcall}
      \end{umlcall}
      \umlobject[class=hook]{post}
      \begin{umlcallself}[op=wait terminated]{run}
      \end{umlcallself}
      \begin{umlcall}{run}{post}
      \end{umlcall}
      \begin{umlcallself}[op=save results]{run}
      \end{umlcallself}
    \end{umlseqdiag}
  \end{tikzpicture}
  \caption{A Sequence diagram of the orchestration scripts}
  \label{fig:orchest}
\end{figure}

First, the script calls the \texttt{pre} hook function from a script file
associated with the experiment, this function can be used for setting up the
environment at the local computer with settings specific to the experiment. Then
the script sends send the configuration of the experiment together with the
associated firmware files to the \ac{REST}-\ac{API} and waits for the experiment
to be started on the test-lab and stores the ID announced by the \ac{API}.

After the \ac{API} has marked the experiment as \texttt{Running}, the script
proceeds by calling into the \texttt{during} function of the experiment. This
function can be used for manipulating the experiment during its executing. In
the case of the experiments run in the evaluation, this function takes care of
resetting a node at a random time during a phase with resets and flashing the different versions of the firmware at the
required time.

After the \texttt{during} function completes, the orchestration script copies
any further results from the side server to the local computer.

\subsection{Aggregation and Storage}

For the storage and analysis of the measurement data, the local computer connects
to the shared server. The aggregated serial output and \acp{PCAP} are
transported to the local computer on one \ac{SSH} connections and each stored
into a separate files. The local computer also collects the log files containing
the events, energy consumption and \ac{RSSI} from the shared server and stores them for
later analysis.

After all data of an experiment has been collected, the contents of the
different files is pre-processed according to the criteria specified for the
evaluation and this data is then stored into a
\emph{SQLite3} database for later analysis. The main reason for pre-processing
the data is to be able to perform the analysis within adequate computation time.

The pre-processed data is then fetched from the database using \ac{SQL}. Numeric
processing of the data is done using \emph{SciPy}
\footnote{\url{https://www.scipy.org/}}, a software library for scientific
computation that is mostly written in the programming language \emph{Python}
but binds to functions written in lower level programming languages that contain
optimized code for certain tasks. The results of the analysis are then displayed
using \emph{Matplotlib}, a \emph{Python} library for displaying structured
data-sets.


\begin{landscape}

\begin{figure}
  \centering

  \begin{tikzpicture}
    \tikzumlset{fill component=tubsLightBlue20, fill port=tubsLightBlue40}

    \begin{umlcomponent}[x=-0.5,y=15,fill=white]{Testlab Node}
    \end{umlcomponent}
    \begin{umlcomponent}[x=3.5,y=15,fill=white]{Testlab Node}
    \end{umlcomponent}
    \begin{umlcomponent}[x=7.5,y=15,fill=white]{Testlab Node}
    \end{umlcomponent}

    \begin{umlcomponent}[x=-1,y=14,fill=white]{Testlab Node}
    \end{umlcomponent}
    \begin{umlcomponent}[x=3,y=14,fill=white]{Testlab Node}
    \end{umlcomponent}
    \begin{umlcomponent}[x=7,y=14,fill=white]{Testlab Node}
    \end{umlcomponent}

    \begin{umlcomponent}[y=1]{Testlab Node}
      \begin{umlcomponent}{Open Node}
        \umlbasiccomponent{UDP Sink / Source}
        \umlbasiccomponent[y=3]{Logger}
        \umlbasiccomponent[y=6]{Powertrace}
      \end{umlcomponent}

      \begin{umlcomponent}[x=6]{Host Node}
        \umlbasiccomponent{Consumption}
        \umlbasiccomponent[y=2.5]{RSSI}
        \umlbasiccomponent[y=5]{Event Log}
        \umlbasiccomponent[y=7.5]{Sniffer}
        \umlbasiccomponent[y=10]{Forwarded Serial}
      \end{umlcomponent}
    \end{umlcomponent}

    \begin{umlcomponent}[y=1,x=14.5]{Shared Server}
      \umlbasiccomponent[y=0]{API}
      \umlbasiccomponent[y=2.5]{Log files}
      \umlbasiccomponent[y=5]{Sniffer Aggregator}
      \umlbasiccomponent[y=7.5]{Serial Aggregator}
    \end{umlcomponent}

    \begin{umlcomponent}[y=1,x=20,fill=tubsLightGreen20]{Local Computer}
      \umlbasiccomponent[y=12]{Orchestration}
      \umlbasiccomponent[y=8]{Database}
      \umlbasiccomponent[y=4]{Analysis}
      \begin{umlcomponent}[y=0]{Presentation}
        \umlactor[scale=0.5]{User}
      \end{umlcomponent}

      \umlassemblyconnector[interface=Parser]{Database-north-port}{Orchestration-south-port}
      \umlassemblyconnector[interface=SQL]{Analysis-north-port}{Database-south-port}
      \umlassemblyconnector[interface=Python]{Presentation-north-port}{Analysis-south-port}

    \end{umlcomponent}

    \umlbasiccomponent[x=14.5,y=14,fill=tubsLightBlue20]{API Server}
    \umlassemblyconnector[interface=REST,with port]{Orchestration-west-port}{API Server}
    \umlassemblyconnector[interface=SSH,with port]{API Server-south-port}{Shared Server-north-port}

    \umlassemblyconnector[interface=SSH,with port]{Orchestration}{Shared Server}
    \umlassemblyconnector[interface=ON Con,with port]{Host Node}{Open Node}
    %\umlHVHassemblyconnector[interface=Configuration,with port]{API}{Host Node}
    \umlHVHassemblyconnector[with port,arm2=+2cm]{Log files}{Consumption}
    \umlassemblyconnector[interface=Log Collection,with port]{Log files}{RSSI}
    \umlHVHassemblyconnector[with port,arm2=+2cm]{Log files}{Event Log}
    \umlassemblyconnector[interface=TCP,with port]{Serial Aggregator}{Forwarded Serial}
    \umlassemblyconnector[interface=TCP,with port]{Sniffer Aggregator}{Sniffer}
  \end{tikzpicture}
  \caption{Software components used in the evaluation}
  \label{fig:components}
\end{figure}

\end{landscape}