Hop Hop Hop Hop STOP!

While playing around with serval in various different network setups in core-network I found some odd behaviour. Getting to run serval to run in the simulator is describe in this article. In this blog post I am going to describe what I have observed.

The network setup

A simple chain of ServalNodes connected pair-wise, each machine with two interfaces - one neighbour on each interface.

N1 <-> N2 <-> N3 <-> ... <-> N17 <->N18

Chained serval setup in core-network.

The problem(?)

To see how the routing and mdp based services perform we did some testing. First connections were verified by using mdp ping.

root@n1:/tmp/pycore.46656/n1.conf# /home/meshadmin/serval-dna/servald mdp ping F40716A16538D25EA01134139056F02D23F9246583012048B4DBA4BBB46A594E
MDP PING F40716A16538D25EA01134139056F02D23F9246583012048B4DBA4BBB46A594E: 12 data bytes
F40716A16538D25EA01134139056F02D23F9246583012048B4DBA4BBB46A594E: seq=1 time=194ms hops=64 ENCRYPTED SIGNED
F40716A16538D25EA01134139056F02D23F9246583012048B4DBA4BBB46A594E: seq=2 time=198ms hops=64 ENCRYPTED SIGNED
F40716A16538D25EA01134139056F02D23F9246583012048B4DBA4BBB46A594E: seq=3 time=184ms hops=64 ENCRYPTED SIGNED
^C--- F40716A16538D25EA01134139056F02D23F9246583012048B4DBA4BBB46A594E ping statistics ---
3 packets transmitted, 3 packets received (plus 0 duplicates, 0 ignored), 0.0% packet loss
round-trip min/avg/max/stddev = 184/192.000/198/5.888 ms (3 samples)

So n18 is reachable from n1, slow ping but for 17 hops it is still okay.
If we try to do the same with an mdp trace, after all there might be different network paths, we end up with the following:

root@n1:/tmp/pycore.46656/n1.conf# /home/meshadmin/serval-dna/servald mdp trace F40716A16538D25EA01134139056F02D23F9246583012048B4DBA4BBB46A594E
Tracing the network path from 8410960D885656669C1B4C4AA56E4339B171E9285A254A1863A80FF7F483A141 to F40716A16538D25EA01134139056F02D23F9246583012048B4DBA4BBB46A594E
INFO: Local date/time: 2016-01-25 11:58:43 +0100
INFO: Serval DNA version: START-3478-g8e223b5
ERROR:network_cli.c:317:app_trace()  overlay_mdp_send returned -1, Timeout waiting for reply to MDP packet (packet was successfully sent).

Trying the same on n17 - only 16 hops away - we get the following:

root@n1:/tmp/pycore.46656/n1.conf# /home/meshadmin/serval-dna/servald mdp trace 614222015D22DBCD65FD79B8C311E12DDE6CDF086E71B0D9CD9746AFCBE02E71
Tracing the network path from 8410960D885656669C1B4C4AA56E4339B171E9285A254A1863A80FF7F483A141 to 614222015D22DBCD65FD79B8C311E12DDE6CDF086E71B0D9CD9746AFCBE02E71
0:8410960D885656669C1B4C4AA56E4339B171E9285A254A1863A80FF7F483A141
1:D6A5E4F4B6EAFE3A22B82742280322695601F2C5C9DF8A8AD6C5067F04E1E139
2:D158F542C7001C55350C67A6B975B08C2B8D2A63A60EC32FEC1712D2206F5C32
3:75A61C0DF5A46BEEEEB28F18C956ACA8C696DED4CBD4C4FE31A54AE2CB26D077
4:424B43C7E6CC80BEEE761F100217824CCEF201ECEA55D14B7F4B205FFA4FF630
5:2E0FCBB04D6C5DDE8D145206A786683164A4443E0AD214E79F69D79EFA1B1D0D
6:1B669680A1555490BC271881362C443E43131ADCB94BA285F6BA1B3B71DC9853
7:9CBAFAB66A0808573ED0FCA42376BC6B527AB39DEB89FFB344EE9381BD105936
8:B9B737D8CFFD5FBFDC5A20505159EC2798D60C4CC4F39B8B0AB6D4240B931C20
9:2AD0422C040079FF320C934D2E98DC9D154896CA1E492FBB3DF720FCFE7FD009
10:BA5288F511246B8C5D77B12B78C6D46DA92D1B28008F57CDAA9F278B9EF54C2B
11:C42A672FB4D19AA59DE47200CC581220CEC36CBE935F8FEF46027CA2604CA072
12:9A3B340EAF09679FF5FCAF720418DA7E2D959018A2C55227EB95C9928B53A52C
13:8EF8016470B665CED119B3CB2F76406952567E2DC16C74AC352B831F1614DB3C
14:57E8A11D20C8970733A5A0776FEA370C0440136D7D7873C1255A26A13924D155
15:79314E660CE67A6DD19E5CAB209CE619EA973A13387D8FE32A35685C1971796E
16:614222015D22DBCD65FD79B8C311E12DDE6CDF086E71B0D9CD9746AFCBE02E71
17:79314E660CE67A6DD19E5CAB209CE619EA973A13387D8FE32A35685C1971796E
18:57E8A11D20C8970733A5A0776FEA370C0440136D7D7873C1255A26A13924D155
19:8EF8016470B665CED119B3CB2F76406952567E2DC16C74AC352B831F1614DB3C
20:9A3B340EAF09679FF5FCAF720418DA7E2D959018A2C55227EB95C9928B53A52C
21:C42A672FB4D19AA59DE47200CC581220CEC36CBE935F8FEF46027CA2604CA072
22:BA5288F511246B8C5D77B12B78C6D46DA92D1B28008F57CDAA9F278B9EF54C2B
23:2AD0422C040079FF320C934D2E98DC9D154896CA1E492FBB3DF720FCFE7FD009
24:B9B737D8CFFD5FBFDC5A20505159EC2798D60C4CC4F39B8B0AB6D4240B931C20
25:9CBAFAB66A0808573ED0FCA42376BC6B527AB39DEB89FFB344EE9381BD105936
26:1B669680A1555490BC271881362C443E43131ADCB94BA285F6BA1B3B71DC9853
27:2E0FCBB04D6C5DDE8D145206A786683164A4443E0AD214E79F69D79EFA1B1D0D
28:424B43C7E6CC80BEEE761F100217824CCEF201ECEA55D14B7F4B205FFA4FF630
29:75A61C0DF5A46BEEEEB28F18C956ACA8C696DED4CBD4C4FE31A54AE2CB26D077
30:D158F542C7001C55350C67A6B975B08C2B8D2A63A60EC32FEC1712D2206F5C32
31:D6A5E4F4B6EAFE3A22B82742280322695601F2C5C9DF8A8AD6C5067F04E1E139
32:8410960D885656669C1B4C4AA56E4339B171E9285A254A1863A80FF7F483A141

Conclusion

16 hops are more than enough, nobody should ever need more... well, apparently ping can do more, didn't test other mdp based services. Delay tolerant services such as rhizome are obviously not effected and perfectly spread data around.

Rhizome spreading data.

Running serval in core-network simulator

For testing purposes it can be really helpful to run your software in a simulator with various different network setups. Here is a short introduction how to use serval with core-network.

Running serval wifi-mesh mixed with static wired nodes.

Installation

Installation of core-network in Ubuntu (here 14.04) is straight forward:

sudo apt-get install core-network

That's it!
For other platform consult the official installation instructions on the core homepage.

The next step is getting serval running as a core-network service. Therefore we need of course serval itself. In this tutorial I am logged in as user meshadmin and have compiled servald with "./configure --prefix=/home/meshadmin/serval-conf".

A set of sample configuration files can be found on github, detailed instructions are following here.

First we need to enable the loading of custom services in /etc/core/core.conf by adding the following line:

custom_services_dir = /home/meshadmin/.core/myservices

In this directory we need to put a small python script describing our service. The contents of this serval.py should look something like this:

''' serval service.
'''

import os

from core.service import CoreService, addservice
from core.misc.ipaddr import IPv4Prefix, IPv6Prefix

class ServalService(CoreService):
    ''' servald as a service.
    '''
    # a unique name is required, without spaces
    _name = "ServalService"
    # you can create your own group here
    _group = "Mesh"
    # list of other services this service depends on
    _depends = ()
    # per-node directories
    _dirs = ("/home/meshadmin/serval-conf/etc/serval","/home/meshadmin/serval-conf/var/log", "/home/meshadmin/serval-conf/var/log/serval", "/home/meshadmin/serval-conf/var/run/serval", "/home/meshadmin/serval-conf/var/cache/serval","/home/meshadmin/serval-conf/var/cache/serval/sqlite3tmp","/home/meshadmin/serval-conf/var/cache/serval/blob")
    # generated files (without a full path this file goes in the node's dir,
    #  e.g. /tmp/pycore.12345/n1.conf/)
    _configs = ('/home/meshadmin/serval-conf/etc/serval/serval.conf', "mesh-start.sh", )
    # this controls the starting order vs other enabled services
    _startindex = 50
    # list of startup commands, also may be generated during startup
    #_startup = ('/home/meshadmin/serval-dna/servald start',)
    _startup = ('bash mesh-start.sh',)
    # list of shutdown commands
    _shutdown = ('/home/meshadmin/serval-dna/servald stop')

    @classmethod
    def generateconfig(cls, node, filename, services):
        ''' Return a string that will be written to filename, or sent to the
            GUI for user customization.
        '''
    if filename == "/home/meshadmin/serval-conf/etc/serval/serval.conf":
            cfg = "debug.rhizome=true\n"
            cfg += "debug.verbose=true\n"
        cfg += "interfaces.0.match=eth*\n"
        cfg += "interfaces.0.socket_type=dgram\n"
        cfg += "interfaces.0.type=ethernet\n"
    elif filename == "mesh-start.sh":
        cfg ="#!/bin/sh\n"
        cfg +="/home/meshadmin/serval-dna/servald start\n"
        cfg +="sleep $[ ( $RANDOM % 10 )  + 1 ]s\n"
        cfg +="for i in `ifconfig | grep \"inet addr:10.\" | cut -d\":\" -f 2 | cut -d\".\" -f1,2,3`\n"
        cfg +="do\n"
        cfg +="/home/meshadmin/serval-dna/servald scan $i.255\n"
        cfg +="done\n"
    else:
        cfg = ""
        return cfg

# this line is required to add the above class to the list of available services
addservice(ServalService)

This service starts a new instance of servald, pre-configured to use any ethernet interface available and starts a discovery scan on all local interfaces after a random sleep. Each instance gets its own fresh copy of the instance directory! Core uses ethernet devices for all types of networks, even WiFi - there configuring servald for all ethernet interfaces is enough.

Note: You need to edit the paths according to your setup!

To use this service it needs to be added to the __init__.py file in the myservices directory:

__all__ = ["sample", "serval"]

That is basically it but to make life easier here are a few helpers I also use.
Adding to the standard node configurations is done by editing ~/.core/nodes.conf. This is where I added the following two entries:

7 { ServalNode pc.gif pc.gif {DefaultRoute ServalService}  netns {simple serval node} }
8 { ServalMesh mdr.gif mdr.gif {OLSRORG ServalService}  netns {simple serval mesh node} }

The first entry, ServalNode, creates a simple node only with our serval service and a default route enabled - classic laptop/pc icon. ServalMesh is a pre-configured node with serval and standard OLSR as a mesh routing protocol - using the standard router icon.

Note: You need olsr installed prior to using it here (sudo apt-get install olsrd) and change the IP config in core to use a /24 or something like that instead of /32 because otherwise serval won't find any peers despite the fact that olsr routing works and you can ping (icmp not mdp!) all addresses.

For live inspection defining custom widgets is also very helpful. This can be done by editing ~/.core/widgets.conf and adding a few lines like these:

15 { {serval peer count} {/home/meshadmin/serval-dna/servald peer count} }
16 { {serval id self} {/home/meshadmin/serval-dna/servald id self} }
17 { {serval rhizome list} /home/meshadmin/serval-tests/listfiles }
18 { {serval mesh scan} /home/meshadmin/serval-tests/mesh-scan.sh }

The first widget displays the number of peers currently known of the inspected serval instance while the second entry shows the SID of the selected node. Widgets 17 and 18 both use helper scripts that can be found here and display the number of files in rhizome store and trigger a new neighbour scan.

Note: Again change the paths to fit your machine.

The last and final step after adding all these files is to restart the core-network service.

sudo service core-daemon restart

Now you are ready to go!

UPDATE: Changed github repo links for core-serval and serval-tests to new location.