* File-sharing (FS) Subsystem::
* REGEX Subsystem::
* REST Subsystem::
+* RPS Subsystem::
@end menu
@node Developer Introduction
@c ** FIXME: Where is the Java tutorial?
@itemize @bullet
@item @xref{Top, Introduction,, gnunet-c-tutorial, The GNUnet C Tutorial}.
-@c broken link
-@c @item @uref{https://gnunet.org/git/gnunet.git/plain/doc/gnunet-c-tutorial.pdf, GNUnet C tutorial}
+@item @uref{https://tutorial.gnunet.org/, GNUnet C tutorial}
@item GNUnet Java tutorial
@end itemize
@item The version control system (git) keeps our code and enables
distributed development.
-It is publicly accessible at @uref{https://gnunet.org/git/}.
+It is publicly accessible at @uref{https://git.gnunet.org/}.
Only developers with write access can commit code, everyone else is
encouraged to submit patches to the GNUnet-developers mailinglist:
@uref{https://lists.gnu.org/mailman/listinfo/gnunet-developers, https://lists.gnu.org/mailman/listinfo/gnunet-developers}
We use it to track feature requests, open bug reports and their
resolutions.
It can be accessed at
-@uref{https://gnunet.org/bugs/, https://gnunet.org/bugs/}.
+@uref{https://bugs.gnunet.org/, https://bugs.gnunet.org/}.
Anyone can report bugs.
@item Our site installation of the
Continuous Integration (CI) system @code{Buildbot} is used
to check GNUnet builds automatically on a range of platforms.
The web interface of this CI is exposed at
-@uref{https://gnunet.org/buildbot/, https://gnunet.org/buildbot/}.
+@uref{https://old.gnunet.org/buildbot/, https://old.gnunet.org/buildbot/}.
Builds are triggered automatically 30 minutes after the last commit to
our repository was made.
@item @command{gnunet-statistics-gtk} (statistics over time),
@item @command{gnunet-peerinfo-gtk}
(information about current connections and known peers),
-@item @command{gnunet-chat-gtk} (chat GUI) and
+@item @command{gnunet-namestore-gtk} (GNS record editor),
+@item @command{gnunet-conversation-gtk} (voice chat GUI) and
@item @command{gnunet-setup} (setup tool for "everything")
@end itemize
to swap one block for a different one that has the same shape. GNUnet's
architecture is based on LEGOs:
-@c @image{images/service_lego_block,5in,,picture of a LEGO block stack - 3 APIs as connectors upon Network Protocol on top of a Service}
+@image{images/service_lego_block,5in,,picture of a LEGO block stack - 3 APIs upon IPC/network protocol provided by a service}
This chapter documents the GNUnet LEGO system, also known as GNUnet's
system architecture.
Unlike services, daemons do not implement their own network protocol and
they have no API:
+@image{images/daemon_lego_block,5in,,A daemon in GNUnet is a component that does not offer an API for others to build upon}
+
The GNUnet system provides a range of services, daemons and user
interfaces, which are then combined into a layered GNUnet instance (also
known as a peer).
+@image{images/service_stack,5in,,A GNUnet peer consists of many layers of services}
+
Note that while it is generally possible to swap one service for another
compatible service, there is often only one implementation. However,
during development we often have a "new" version of a service in parallel
@code{error} case or the significantly simpler continuation. For example:
@example
-if (0 != stat ("filename," &sbuf)) @{
+if (0 != stat ("filename,"
+ &sbuf))
+@{
error();
- @}
- else @{
- /* handle normal case here */
- @}
+@}
+else
+@{
+ /* handle normal case here */
+@}
@end example
@noindent
should be omitted:
@example
-if (0 != stat ("filename," &sbuf)) @{
+if (0 != stat ("filename",
+ &sbuf))
+@{
error();
return;
- @}
+@}
/* handle normal case here */
@end example
code clarity. For example, one can write:
@example
-if (NULL == (value = lookup_function())) @{
+if (NULL == (value = lookup_function()))
+@{
error();
return;
- @}
+@}
@end example
@item Use @code{break} and @code{continue} wherever possible to avoid
@example
next = head;
-while (NULL != (pos = next)) @{
+while (NULL != (pos = next))
+@{
next = pos->next;
if (! should_free (pos))
continue;
- GNUNET_CONTAINER_DLL_remove (head, tail, pos);
+ GNUNET_CONTAINER_DLL_remove (head,
+ tail,
+ pos);
GNUNET_free (pos);
- @}
+@}
@end example
instead of
next = pos->next;
if (! should_free (pos))
continue;
- GNUNET_CONTAINER_DLL_remove (head, tail, pos);
+ GNUNET_CONTAINER_DLL_remove (head,
+ tail,
+ pos);
GNUNET_free (pos);
@}
@end example
@code{cls} argument to the precise expected type. For example:
@example
-int callback (void *cls, char *args) @{
+int
+callback (void *cls,
+ char *args)
+@{
struct Foo *foo = cls;
int other_variables;
@}
@end example
+@item As shown in the example above, after the return type of a
+function there should be a break. Each parameter should
+be on a new line.
@item It is good practice to write complex @code{if} expressions instead
of using deeply nested @code{if} statements. However, except for addition
and multiplication, all operators should use parens. This is fine:
@example
-if ( (1 == foo) || ((0 == bar) && (x != y)) )
+if ( (1 == foo) ||
+ ( (0 == bar) &&
+ (x != y) ) )
return x;
@end example
First of all you have to obtain gnunet-ext from git:
@example
-git clone https://gnunet.org/git/gnunet-ext.git
+git clone https://git.gnunet.org/gnunet-ext.git
@end example
The next step is to bootstrap and configure it. For configure you have to
* Hosts file format::
* Topology file format::
* Testbed Barriers::
-* Automatic large-scale deployment in the PlanetLab testbed::
* TESTBED Caveats::
@end menu
connected with high probability to well connected peers.
(See Emergence of Scaling in Random Networks. Science 286,
509-512, 1999
-(@uref{https://gnunet.org/git/bibliography.git/plain/docs/emergence_of_scaling_in_random_networks__barabasi_albert_science_286__1999.pdf, pdf}))
+(@uref{https://git.gnunet.org/bibliography.git/plain/docs/emergence_of_scaling_in_random_networks__barabasi_albert_science_286__1999.pdf, pdf}))
@item @code{GNUNET_TESTBED_TOPOLOGY_FROM_FILE}: The topology information
is loaded from a file. The path to the file has to be given.
for ensuring that the barrier is reached by all the controllers and the
downward propagation is for triggering that the barrier is crossed.
-@cindex PlanetLab testbed
-@node Automatic large-scale deployment in the PlanetLab testbed
-@subsection Automatic large-scale deployment in the PlanetLab testbed
-
-PlanetLab is a testbed for computer networking and distributed systems
-research. It was established in 2002 and as of June 2010 was composed of
-1090 nodes at 507 sites worldwide.
-
-To automate the GNUnet we created a set of automation tools to simplify
-the large-scale deployment. We provide you a set of scripts you can use
-to deploy GNUnet on a set of nodes and manage your installation.
-
-Please also check @uref{https://gnunet.org/installation-fedora8-svn} and
-@uref{https://gnunet.org/installation-fedora12-svn} to find detailed
-instructions how to install GNUnet on a PlanetLab node.
-
-
-@c ***********************************************************************
-@menu
-* PlanetLab Automation for Fedora8 nodes::
-* Install buildslave on PlanetLab nodes running fedora core 8::
-* Setup a new PlanetLab testbed using GPLMT::
-* Why do i get an ssh error when using the regex profiler?::
-@end menu
-
-@node PlanetLab Automation for Fedora8 nodes
-@subsubsection PlanetLab Automation for Fedora8 nodes
-
-@c ***********************************************************************
-@node Install buildslave on PlanetLab nodes running fedora core 8
-@subsubsection Install buildslave on PlanetLab nodes running fedora core 8
-@c ** Actually this is a subsubsubsection, but must be fixed differently
-@c ** as subsubsection is the lowest.
-
-Since most of the PlanetLab nodes are running the very old Fedora core 8
-image, installing the buildslave software is quite some pain. For our
-PlanetLab testbed we figured out how to install the buildslave software
-best.
-
-@c This is a very terrible way to suggest installing software.
-@c FIXME: Is there an official, safer way instead of blind-piping a
-@c script?
-@c FIXME: Use newer pypi URLs below.
-Install Distribute for Python:
-
-@example
-curl http://python-distribute.org/distribute_setup.py | sudo python
-@end example
-
-Install Distribute for zope.interface <= 3.8.0 (4.0 and 4.0.1 will not
-work):
-
-@example
-export PYPI=@value{PYPI-URL}
-wget $PYPI/z/zope.interface/zope.interface-3.8.0.tar.gz
-tar zvfz zope.interface-3.8.0.tar.gz
-cd zope.interface-3.8.0
-sudo python setup.py install
-@end example
-
-Install the buildslave software (0.8.6 was the latest version):
-
-@example
-export GCODE="http://buildbot.googlecode.com/files"
-wget $GCODE/buildbot-slave-0.8.6p1.tar.gz
-tar xvfz buildbot-slave-0.8.6p1.tar.gz
-cd buildslave-0.8.6p1
-sudo python setup.py install
-@end example
-
-The setup will download the matching twisted package and install it.
-It will also try to install the latest version of zope.interface which
-will fail to install. Buildslave will work anyway since version 3.8.0
-was installed before!
-
-@c ***********************************************************************
-@node Setup a new PlanetLab testbed using GPLMT
-@subsubsection Setup a new PlanetLab testbed using GPLMT
-
-@itemize @bullet
-@item Get a new slice and assign nodes
-Ask your PlanetLab PI to give you a new slice and assign the nodes you
-need
-@item Install a buildmaster
-You can stick to the buildbot documentation:@
-@uref{http://buildbot.net/buildbot/docs/current/manual/installation.html}
-@item Install the buildslave software on all nodes
-To install the buildslave on all nodes assigned to your slice you can use
-the tasklist @code{install_buildslave_fc8.xml} provided with GPLMT:
-
-@example
-./gplmt.py -c contrib/tumple_gnunet.conf -t \
-contrib/tasklists/install_buildslave_fc8.xml -a -p <planetlab password>
-@end example
-
-@item Create the buildmaster configuration and the slave setup commands
-
-The master and the and the slaves have need to have credentials and the
-master has to have all nodes configured. This can be done with the
-@file{create_buildbot_configuration.py} script in the @file{scripts}
-directory.
-
-This scripts takes a list of nodes retrieved directly from PlanetLab or
-read from a file and a configuration template and creates:
-
-@itemize @bullet
-@item a tasklist which can be executed with gplmt to setup the slaves
-@item a master.cfg file containing a PlanetLab nodes
-@end itemize
-
-A configuration template is included in the <contrib>, most important is
-that the script replaces the following tags in the template:
-
-%GPLMT_BUILDER_DEFINITION :@ GPLMT_BUILDER_SUMMARY@ GPLMT_SLAVES@
-%GPLMT_SCHEDULER_BUILDERS
-
-Create configuration for all nodes assigned to a slice:
-
-@example
-./create_buildbot_configuration.py -u <planetlab username> \
--p <planetlab password> -s <slice> -m <buildmaster+port> \
--t <template>
-@end example
-
-Create configuration for some nodes in a file:
-
-@example
-./create_buildbot_configuration.p -f <node_file> \
--m <buildmaster+port> -t <template>
-@end example
-
-@item Copy the @file{master.cfg} to the buildmaster and start it
-Use @code{buildbot start <basedir>} to start the server
-@item Setup the buildslaves
-@end itemize
-
-@c ***********************************************************************
-@node Why do i get an ssh error when using the regex profiler?
-@subsubsection Why do i get an ssh error when using the regex profiler?
-
-Why do i get an ssh error "Permission denied (publickey,password)." when
-using the regex profiler although passwordless ssh to localhost works
-using publickey and ssh-agent?
-
-You have to generate a public/private-key pair with no password:@
-@code{ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_localhost}@
-and then add the following to your ~/.ssh/config file:
-
-@code{Host 127.0.0.1@ IdentityFile ~/.ssh/id_localhost}
-
-now make sure your hostsfile looks like
-
-@example
-[USERNAME]@@127.0.0.1:22@
-[USERNAME]@@127.0.0.1:22
-@end example
-
-You can test your setup by running @code{ssh 127.0.0.1} in a
-terminal and then in the opened session run it again.
-If you were not asked for a password on either login,
-then you should be good to go.
-
@cindex TESTBED Caveats
@node TESTBED Caveats
@subsection TESTBED Caveats
@subsubsection CORE must be started
A uncomplicated issue is bug #3993
-(@uref{https://gnunet.org/bugs/view.php?id=3993, https://gnunet.org/bugs/view.php?id=3993}):
+(@uref{https://bugs.gnunet.org/view.php?id=3993, https://bugs.gnunet.org/view.php?id=3993}):
Your configuration MUST somehow ensure that for each peer the
@code{CORE} service is started when the peer is setup, otherwise
@code{TESTBED} may fail to connect peers when the topology is initialized,
@file{gnunet_common.h} defines several @strong{log levels}:
@table @asis
-@item ERROR for errors (really problematic situations, often leading to
-crashes)
-@item WARNING for warnings (troubling situations that might have
-negative consequences, although not fatal)
+@item ERROR for errors
+(really problematic situations, often leading to crashes)
+@item WARNING for warnings
+(troubling situations that might have negative consequences, although
+not fatal)
@item INFO for various information.
Used somewhat rarely, as GNUnet statistics is used to hold and display
most of the information that users might find interesting.
Since now activating DEBUG automatically makes it VERBOSE and activates
@strong{all} debug messages by default, you probably want to use the
-https://gnunet.org/logging functionality to filter only relevant messages.
+@uref{https://docs.gnunet.org/#Logging, https://docs.gnunet.org/#Logging}
+functionality to filter only relevant messages.
A suitable configuration could be:
@example
@c ***********************************************************************
@node Client - Establish connection
@subsubsection Client - Establish connection
-@c %**end of header
+
At first, on the client side, the underlying API is employed to create a
@c ***********************************************************************
@node Client - Initialize request message
@subsubsection Client - Initialize request message
-@c %**end of header
+
When the connection is ready, we initialize the message. In this step,
all the fields of the message should be properly initialized, namely the
@c ***********************************************************************
@node Client - Send request and receive response
@subsubsection Client - Send request and receive response
-@c %**end of header
+
@b{FIXME: This is very outdated, see the tutorial for the current API!}
@c ***********************************************************************
@node Server - Add new handles for specified messages
@subsubsection Server - Add new handles for specified messages
-@c %**end of header
+
in the function above the argument @code{run} is used to initiate
transport service,and defined like this:
@c ***********************************************************************
@node Server - Process request message
@subsubsection Server - Process request message
-@c %**end of header
+
After the initialization of transport service, the request message would
be processed. Before handling the main message data, the validity of this
@c ***********************************************************************
@node Server - Response to client
@subsubsection Server - Response to client
-@c %**end of header
+
Once the processing of current request is done, the server should give the
response to the client. A new @code{struct AddressLookupMessage} would be
@c ***********************************************************************
@node Server - Notification of clients
@subsubsection Server - Notification of clients
-@c %**end of header
+
Often a service needs to (repeatedly) transmit notifications to a client
or a group of clients. In these cases, the client typically has once
@node Conversion between Network Byte Order (Big Endian) and Host Byte Order
@subsubsection Conversion between Network Byte Order (Big Endian) and Host Byte Order
@c %** subsub? it's a referenced page on the ipc document.
-@c %**end of header
+
Here we can simply comprehend big endian and little endian as Network Byte
Order and Host Byte Order respectively. What is the difference between
@cindex Cryptography API
@node Cryptography API
@subsection Cryptography API
-@c %**end of header
+
The gnunetutil APIs provides the cryptographic primitives used in GNUnet.
GNUnet uses 2048 bit RSA keys for the session key exchange and for signing
@cindex Message Queue API
@node Message Queue API
@subsection Message Queue API
-@c %**end of header
+
@strong{ Introduction }@
Often, applications need to queue messages that
@cindex Service API
@node Service API
@subsection Service API
-@c %**end of header
+
Most GNUnet code lives in the form of services. Services are processes
that offer an API for other components of the system to build on. Those
@c ***********************************************************************
@node Optimizing Memory Consumption of GNUnet's (Multi-) Hash Maps
@subsection Optimizing Memory Consumption of GNUnet's (Multi-) Hash Maps
-@c %**end of header
+
A commonly used data structure in GNUnet is a (multi-)hash map. It is most
often used to map a peer identity to some data structure, but also to map
@node Analysis
@subsubsection Analysis
-@c %**end of header
+
The main reason for the "excessive" memory consumption by the hash map is
that GNUnet uses 512-bit cryptographic hash codes --- and the
@c ***********************************************************************
@node Solution
@subsubsection Solution
-@c %**end of header
+
The solution that has now been implemented is to @strong{optionally}
allow the hash map to not make a (deep) copy of the hash but instead have
@c ***********************************************************************
@node Migration
@subsubsection Migration
-@c %**end of header
+
To use the new feature, first check that the values contain the respective
key (and never modify it). Then, all calls to
@c ***********************************************************************
@node Conclusion
@subsubsection Conclusion
-@c %**end of header
+
The new optimization can is often applicable and can result in a
reduction in memory consumption of up to 30% in practice. However, it
@c ***********************************************************************
@node Availability
@subsubsection Availability
-@c %**end of header
-The new multi hash map code was committed in SVN 24319 (will be in GNUnet
-0.9.4). Various subsystems (transport, core, dht, file-sharing) were
+
+The new multi hash map code was committed in SVN 24319 (which made its
+way into GNUnet version 0.9.4).
+Various subsystems (transport, core, dht, file-sharing) were
previously audited and modified to take advantage of the new capability.
In particular, memory consumption of the file-sharing service is expected
to drop by 20-30% due to this change.
@cindex CONTAINER_MDLL API
@node CONTAINER_MDLL API
@subsection CONTAINER_MDLL API
-@c %**end of header
+
This text documents the GNUNET_CONTAINER_MDLL API. The
GNUNET_CONTAINER_MDLL API is similar to the GNUNET_CONTAINER_DLL API in
@cindex ARM
@node Automatic Restart Manager (ARM)
@section Automatic Restart Manager (ARM)
-@c %**end of header
+
GNUnet's Automated Restart Manager (ARM) is the GNUnet service responsible
for system initialization and service babysitting. ARM starts and halts
@c ***********************************************************************
@node Basic functionality
@subsection Basic functionality
-@c %**end of header
+
@itemize @bullet
@item ARM source code can be found under "src/arm".@ Service processes are
@c ***********************************************************************
@node Key configuration options
@subsection Key configuration options
-@c %**end of header
+
Configurations for ARM and services should be available in a .conf file
(As an example, see test_arm_api_data.conf). When running ARM, the
@c ***********************************************************************
@node ARM - Availability
@subsection ARM - Availability
-@c %**end of header
+
As mentioned before, one of the features provided by ARM is starting
services on demand. Consider the example of one service "client" that
@cindex TRANSPORT Subsystem
@node TRANSPORT Subsystem
@section TRANSPORT Subsystem
-@c %**end of header
+
This chapter documents how the GNUnet transport subsystem works. The
GNUnet transport subsystem consists of three main components: the
@node Address validation protocol
@subsection Address validation protocol
-@c %**end of header
+
This section documents how the GNUnet transport service validates
connections with other peers. It is a high-level description of the
@cindex NAT library
@node NAT library
@section NAT library
-@c %**end of header
+
The goal of the GNUnet NAT library is to provide a general-purpose API for
NAT traversal @strong{without} third-party support. So protocols that
@node Distance-Vector plugin
@section Distance-Vector plugin
-@c %**end of header
+
The Distance Vector (DV) transport is a transport mechanism that allows
peers to act as relays for each other, thereby connecting peers that would
@cindex SMTP plugin
@node SMTP plugin
@section SMTP plugin
-@c %**end of header
+
+@c TODO: Update!
This section describes the new SMTP transport plugin for GNUnet as it
exists in the 0.7.x and 0.8.x branch. SMTP support is currently not
abstraction (as it existed in 0.7.x and 0.8.x) in more detail and gives
some benchmarking results. The performance results presented are quite
old and maybe outdated at this point.
+For the readers in the year 2019, you will notice by the mention of
+version 0.7, 0.8, and 0.9 that this section has to be taken with your
+usual grain of salt and be updated eventually.
@itemize @bullet
@item Why use SMTP for a peer-to-peer transport?
@node Why use SMTP for a peer-to-peer transport?
@subsection Why use SMTP for a peer-to-peer transport?
-@c %**end of header
+
There are many reasons why one would not want to use SMTP:
@node How does it work?
@subsection How does it work?
-@c %**end of header
+
When a GNUnet peer needs to send a message to another GNUnet peer that has
advertised (only) an SMTP transport address, GNUnet base64-encodes the
@node How do I configure my peer?
@subsection How do I configure my peer?
-@c %**end of header
+
First, you need to configure @code{procmail} to filter your inbound E-mail
for GNUnet traffic. The GNUnet messages must be delivered into a pipe, for
@node How do I test if it works?
@subsection How do I test if it works?
-@c %**end of header
+
Any transport can be subjected to some rudimentary tests using the
@code{gnunet-transport-check} tool. The tool sends a message to the local
@node How fast is it?
@subsection How fast is it?
-@c %**end of header
+
We have measured the performance of the UDP, TCP and SMTP transport layer
directly and when used from an application using the GNUnet core.
@cindex Bluetooth plugin
@node Bluetooth plugin
@section Bluetooth plugin
-@c %**end of header
+
This page describes the new Bluetooth transport plugin for GNUnet. The
plugin is still in the testing stage so don't expect it to work
@node What do I need to use the Bluetooth plugin transport?
@subsection What do I need to use the Bluetooth plugin transport?
-@c %**end of header
+
If you are a GNU/Linux user and you want to use the Bluetooth
transport plugin you should install the
@c FIXME: Change to unique title
@node How does it work2?
@subsection How does it work2?
-@c %**end of header
+
The Bluetooth transport plugin uses virtually the same code as the WLAN
plugin and only the helper binary is different. The helper takes a single
@node What possible errors should I be aware of?
@subsection What possible errors should I be aware of?
-@c %**end of header
+
@emph{This section is dedicated for GNU/Linux users}
@c FIXME: A more unique name
@node How do I configure my peer2?
@subsection How do I configure my peer2?
-@c %**end of header
+
On GNU/Linux, you just have to be sure that the interface name
corresponds to the one that you want to use.
@node How can I test it?
@subsection How can I test it?
-@c %**end of header
+
If you have two Bluetooth devices on the same machine and you are using
GNU/Linux you must:
@node The implementation of the Bluetooth transport plugin
@subsection The implementation of the Bluetooth transport plugin
-@c %**end of header
+
This page describes the implementation of the Bluetooth transport plugin.
@node Linux functionality
@subsubsection Linux functionality
-@c %**end of header
+
In order to implement the plugin functionality on GNU/Linux I
used the BlueZ stack.
@node Details about the broadcast implementation
@subsubsection Details about the broadcast implementation
-@c %**end of header
+
First I want to point out that the broadcast functionality for the CONTROL
messages is not implemented in a conventional way. Since the inquiry scan
@node Windows functionality
@subsubsection Windows functionality
-@c %**end of header
+
For Windows I decided to use the Microsoft Bluetooth stack which has the
advantage of coming standard from Windows XP SP2. The main disadvantage is
@node Pending features
@subsubsection Pending features
-@c %**end of header
+
@itemize @bullet
@item Implement the broadcast functionality on Windows @emph{(currently
@node WLAN plugin
@section WLAN plugin
-@c %**end of header
+
This section documents how the wlan transport plugin works. Parts which
are not implemented yet or could be better implemented are described at
@cindex ATS Subsystem
@node ATS Subsystem
@section ATS Subsystem
-@c %**end of header
+
ATS stands for "automatic transport selection", and the function of ATS in
GNUnet is to decide on which address (and thus transport plugin) should
@cindex CORE Subsystem
@node CORE Subsystem
@section CORE Subsystem
-@c %**end of header
+
The CORE subsystem in GNUnet is responsible for securing link-layer
communications between nodes in the GNUnet overlay network. CORE builds
@cindex core subsystem limitations
@node Limitations
@subsection Limitations
-@c %**end of header
+
CORE does not perform
@uref{http://en.wikipedia.org/wiki/Routing, routing}; using CORE it is
@cindex when is a peer connected
@node When is a peer "connected"?
@subsection When is a peer "connected"?
-@c %**end of header
+
In addition to the security features mentioned above, CORE also provides
one additional key feature to applications using it, and that is a
@cindex libgnunetcore
@node libgnunetcore
@subsection libgnunetcore
-@c %**end of header
+
The CORE API (defined in @file{gnunet_core_service.h}) is the basic
messaging API used by P2P applications built using GNUnet. It provides
@cindex core clinet-service protocol
@node The CORE Client-Service Protocol
@subsection The CORE Client-Service Protocol
-@c %**end of header
+
This section describes the protocol between an application using the CORE
service (the client) and the CORE service process itself.
@node Setup2
@subsubsection Setup2
-@c %**end of header
+
When a client connects to the CORE service, it first sends a
@code{InitMessage} which specifies options for the connection and a set of
@node Notifications
@subsubsection Notifications
-@c %**end of header
+
The CORE will send @code{ConnectNotifyMessage}s and
@code{DisconnectNotifyMessage}s whenever peers connect or disconnect from
@node Sending
@subsubsection Sending
-@c %**end of header
+
When a client wants to transmit a message, it first requests a
transmission slot by sending a @code{SendMessageRequest} which specifies
@cindex CORE Peer-to-Peer Protocol
@node The CORE Peer-to-Peer Protocol
@subsection The CORE Peer-to-Peer Protocol
-@c %**end of header
+
@menu
@cindex EphemeralKeyMessage creation
@node Creating the EphemeralKeyMessage
@subsubsection Creating the EphemeralKeyMessage
-@c %**end of header
+
When the CORE service starts, each peer creates a fresh ephemeral (ECC)
public-private key pair and signs the corresponding
@node Establishing a connection
@subsubsection Establishing a connection
-@c %**end of header
+
Peers begin their interaction by sending a @code{EphemeralKeyMessage} to
the other peer once the TRANSPORT service notifies the CORE service about
@node Encryption and Decryption
@subsubsection Encryption and Decryption
-@c %**end of header
+
All functions related to the key exchange and encryption/decryption of
messages can be found in @file{gnunet-service-core_kx.c} (except for the
@node Type maps
@subsubsection Type maps
-@c %**end of header
+
Once an encrypted connection has been established, peers begin to exchange
type maps. Type maps are used to allow the CORE service to determine which
retransmit the type map (with exponential back-off).
@cindex CADET Subsystem
+@cindex CADET
+@cindex cadet
@node CADET Subsystem
@section CADET Subsystem
created with as reliable, CADET will retransmit the lost message and
deliver it in order to the destination application.
+@pindex GNUNET_CADET_connect
To communicate with other peers using CADET, it is necessary to first
connect to the service using @code{GNUNET_CADET_connect}.
This function takes several parameters in form of callbacks, to allow the
The function returns a handle which has to be used for any further
interaction with the service.
-To connect to a remote peer a client has to call the
+@pindex GNUNET_CADET_channel_create
+To connect to a remote peer, a client has to call the
@code{GNUNET_CADET_channel_create} function. The most important parameters
given are the remote peer's identity (it public key) and a port, which
specifies which application on the remote peer to connect to, similar to
Similar to @code{GNUNET_CADET_connect},@code{GNUNET_CADET_create_channel}
returns a handle to interact with the created channel.
+@pindex GNUNET_CADET_notify_transmit_ready
For every message the client wants to send to the remote application,
@code{GNUNET_CADET_notify_transmit_ready} must be called, indicating the
channel on which the message should be sent and the size of the message
means that the channel is online. The callback can give 0 bytes to CADET
if no message is to be sent, this is OK.
+@pindex GNUNET_CADET_notify_transmit_cancel
If a transmission was requested but before the callback fires it is no
longer needed, it can be canceled with
@code{GNUNET_CADET_notify_transmit_ready_cancel}, which uses the handle
client must not call @code{GNUNET_CADET_notify_transmit_ready} again until
the callback is called or the request is canceled.
+@pindex GNUNET_CADET_channel_destroy
When a channel is no longer needed, a client can call
@code{GNUNET_CADET_channel_destroy} to get rid of it.
Note that CADET will try to transmit all pending traffic before notifying
executes the callbacks given to it at the time of
@code{GNUNET_CADET_connect}.
+@pindex GNUNET_CADET_disconnect
Finally, when an application no longer wants to use CADET, it should call
@code{GNUNET_CADET_disconnect}, but first all channels and pending
transmissions must be closed (otherwise CADET will complain).
@node Target value
@subsubsection Target value
-@c %**end of header
+
The target value itself is generated by hashing the current time, rounded
down to an agreed value. If the rounding amount is 1h (default) and the
@node Timing
@subsubsection Timing
-@c %**end of header
+
The NSE subsystem has some timing control to avoid everybody broadcasting
its ID all at one. Once each peer has the target random value, it
@node Controlled Flooding
@subsubsection Controlled Flooding
-@c %**end of header
+
When a peer receives a value, first it verifies that it is closer than the
closest value it had so far, otherwise it answers the incoming message
@node Calculating the estimate
@subsubsection Calculating the estimate
-@c %**end of header
+
Once the closest ID has been spread across the network each peer gets the
exact distance between this ID and the target value of the round and
@node libgnunetnse
@subsection libgnunetnse
-@c %**end of header
+
The NSE subsystem has the simplest API of all services, with only two
calls: @code{GNUNET_NSE_connect} and @code{GNUNET_NSE_disconnect}.
@node Results
@subsubsection Results
-@c %**end of header
+
The callback provides two values: the average and the
@uref{http://en.wikipedia.org/wiki/Standard_deviation, standard deviation}
@node libgnunetnse - Examples
@subsubsection libgnunetnse -Examples
-@c %**end of header
+
Let's close with a couple examples.
@node The NSE Client-Service Protocol
@subsection The NSE Client-Service Protocol
-@c %**end of header
+
As with the API, the client-service protocol is very simple, only has 2
different messages, defined in @code{src/nse/nse.h}:
@node The NSE Peer-to-Peer Protocol
@subsection The NSE Peer-to-Peer Protocol
-@c %**end of header
+@pindex GNUNET_MESSAGE_TYPE_NSE_P2P_FLOOD
The NSE subsystem only has one message in the P2P protocol, the
@code{GNUNET_MESSAGE_TYPE_NSE_P2P_FLOOD} message.
@node HOSTLIST Subsystem
@section HOSTLIST Subsystem
-@c %**end of header
+
Peers in the GNUnet overlay network need address information so that they
can connect with other peers. GNUnet uses so called HELLO messages to
@node HELLOs
@subsection HELLOs
-@c %**end of header
+
The basic information peers require to connect to other peers are
contained in so called HELLO messages you can think of as a business card.
@node Overview for the HOSTLIST subsystem
@subsection Overview for the HOSTLIST subsystem
-@c %**end of header
+
The HOSTLIST subsystem provides a way to distribute and obtain contact
information to connect to other peers using a simple HTTP GET request.
@node Features
@subsubsection Features
-@c %**end of header
+
The HOSTLIST daemon can:
@node HOSTLIST - Limitations
@subsubsection HOSTLIST - Limitations
-@c %**end of header
+
The HOSTLIST daemon does not:
@node Interacting with the HOSTLIST daemon
@subsection Interacting with the HOSTLIST daemon
-@c %**end of header
+
The HOSTLIST subsystem is currently implemented as a daemon, so there is
no need for the user to interact with it and therefore there is no
@node Hostlist security address validation
@subsection Hostlist security address validation
-@c %**end of header
+
Since information obtained from other parties cannot be trusted without
validation, we have to distinguish between @emph{validated} and
@node The HOSTLIST daemon
@subsection The HOSTLIST daemon
-@c %**end of header
+
The hostlist daemon is the main component of the HOSTLIST subsystem. It is
started by the ARM service and (if configured) starts the HOSTLIST client
and server components.
+@pindex GNUNET_MESSAGE_TYPE_HOSTLIST_ADVERTISEMENT
If the daemon provides a hostlist itself it can advertise it's own
hostlist to other peers. To do so it sends a
@code{GNUNET_MESSAGE_TYPE_HOSTLIST_ADVERTISEMENT} message to other peers
@node The HOSTLIST server
@subsection The HOSTLIST server
-@c %**end of header
+
The server provides a way for other peers to obtain HELLOs. Basically it
is a small web server other peers can connect to and download a list of
@node The HTTP Server
@subsubsection The HTTP Server
-@c %**end of header
+
During startup, the server starts a web server listening on the port
specified with the HTTPPORT value (default 8080). In addition it connects
@node Advertising the URL
@subsubsection Advertising the URL
-@c %**end of header
+
The server also advertises the URL to download the hostlist to other peers
if hostlist advertisement is enabled.
@node The HOSTLIST client
@subsection The HOSTLIST client
-@c %**end of header
+
The client provides the functionality to download the list of HELLOs from
a set of URLs.
@node Bootstrapping
@subsubsection Bootstrapping
-@c %**end of header
+
For bootstrapping, it schedules a task to download the hostlist from the
set of known URLs.
@node Learning
@subsubsection Learning
-@c %**end of header
+
The client also manages hostlist advertisements from other peers. The
HOSTLIST daemon forwards @code{GNUNET_MESSAGE_TYPE_HOSTLIST_ADVERTISEMENT}
@node Usage
@subsection Usage
-@c %**end of header
+
To start HOSTLIST by default, it has to be added to the DEFAULTSERVICES
section for the ARM services. This is done in the default configuration.
@node IDENTITY Subsystem
@section IDENTITY Subsystem
-@c %**end of header
+
Identities of "users" in GNUnet are called egos.
Egos can be used as pseudonyms ("fake names") or be tied to an
@cindex libgnunetidentity
@node libgnunetidentity
@subsection libgnunetidentity
-@c %**end of header
+
@menu
@node Connecting to the service
@subsubsection Connecting to the service
-@c %**end of header
+
First, typical clients connect to the identity service using
@code{GNUNET_IDENTITY_connect}. This function takes a callback as a
@node Operations on Egos
@subsubsection Operations on Egos
-@c %**end of header
+
Given an ego handle, the main operations are to get its associated private
key using @code{GNUNET_IDENTITY_ego_get_private_key} or its associated
@node The anonymous Ego
@subsubsection The anonymous Ego
-@c %**end of header
+
A special way to obtain an ego handle is to call
@code{GNUNET_IDENTITY_ego_get_anonymous}, which returns an ego for the
@node The IDENTITY Client-Service Protocol
@subsection The IDENTITY Client-Service Protocol
-@c %**end of header
+
A client connecting to the identity service first sends a message with
type
@node Editing Zone Information
@subsubsection Editing Zone Information
-@c %**end of header
+
NAMESTORE provides functions to lookup records stored under a label in a
zone and to store records under a label in a zone.
@node Iterating Zone Information
@subsubsection Iterating Zone Information
-@c %**end of header
+
A client can iterate over all information in a zone or all zones managed
by NAMESTORE.
@node Monitoring Zone Information
@subsubsection Monitoring Zone Information
-@c %**end of header
+
Clients can also monitor zones to be notified about changes. Here the
clients uses the @code{GNUNET_NAMESTORE_zone_monitor_start} function and
@node PEERINFO Subsystem
@section PEERINFO Subsystem
-@c %**end of header
+
The PEERINFO subsystem is used to store verified (validated) information
about known peers in a persistent way. It obtains these addresses for
@node PEERINFO - Features
@subsection PEERINFO - Features
-@c %**end of header
+
@itemize @bullet
@item Persistent storage
@node DeveloperPeer Information
@subsection DeveloperPeer Information
-@c %**end of header
+
The PEERINFO subsystem stores these information in the form of HELLO
messages you can think of as business cards.
@node Startup
@subsection Startup
-@c %**end of header
+
During startup the PEERINFO services loads persistent HELLOs from disk.
First PEERINFO parses the directory configured in the HOSTS value of the
@node Managing Information
@subsection Managing Information
-@c %**end of header
+
The PEERINFO services stores information about known PEERS and a single
HELLO message for every peer.
@node Obtaining Information
@subsection Obtaining Information
-@c %**end of header
+
When a client requests information from PEERINFO, PEERINFO performs a
lookup for the respective peer or all peers if desired and transmits this
@node The PEERINFO Client-Service Protocol
@subsection The PEERINFO Client-Service Protocol
-@c %**end of header
+
To connect and disconnect to and from the PEERINFO Service PEERINFO
utilizes the util client/server infrastructure, so no special messages
@node libgnunetpeerinfo
@subsection libgnunetpeerinfo
-@c %**end of header
+
The PEERINFO API consists mainly of three different functionalities:
@node Connecting to the PEERINFO Service
@subsubsection Connecting to the PEERINFO Service
-@c %**end of header
+
To connect to the PEERINFO service the function
@code{GNUNET_PEERINFO_connect} is used, taking a configuration handle as
@node Adding Information to the PEERINFO Service
@subsubsection Adding Information to the PEERINFO Service
-@c %**end of header
+
@code{GNUNET_PEERINFO_add_peer} adds a new peer to the PEERINFO subsystem
storage. This function takes the PEERINFO handle as an argument, the HELLO
@node Obtaining Information from the PEERINFO Service
@subsubsection Obtaining Information from the PEERINFO Service
-@c %**end of header
+
To iterate over information in PEERINFO you use
@code{GNUNET_PEERINFO_iterate}.
@node PEERSTORE Subsystem
@section PEERSTORE Subsystem
-@c %**end of header
+
GNUnet's PEERSTORE subsystem offers persistent per-peer storage for other
GNUnet subsystems. GNUnet subsystems can use PEERSTORE to persistently
@node Functionality
@subsection Functionality
-@c %**end of header
+
Subsystems can store any type of value under a (subsystem, peerid, key)
combination. A "replace" flag set during store operations forces the
@node Architecture
@subsection Architecture
-@c %**end of header
+
PEERSTORE implements the following components:
@node libgnunetpeerstore
@subsection libgnunetpeerstore
-@c %**end of header
+
libgnunetpeerstore is the library containing the PEERSTORE API. Subsystems
wishing to communicate with the PEERSTORE service use this API to open a
@node SET Subsystem
@section SET Subsystem
-@c %**end of header
+
The SET service implements efficient set operations between two peers
over a mesh tunnel.
@node Local Sets
@subsection Local Sets
-@c %**end of header
+
Sets created by a local client can be modified and reused for multiple
operations. As each set operation requires potentially expensive special
@node Set Modifications
@subsection Set Modifications
-@c %**end of header
+
Even when set operations are active, one can add to and remove elements
from a set.
@node Set Operations
@subsection Set Operations
-@c %**end of header
+
Set operations can be started in two ways: Either by accepting an
operation request from a remote peer, or by requesting a set operation
@node Result Elements
@subsection Result Elements
-@c %**end of header
+
The SET service has three @emph{result modes} that determine how an
operation's result set is delivered to the client:
@node libgnunetset
@subsection libgnunetset
-@c %**end of header
+
@menu
* Sets::
@node Sets
@subsubsection Sets
-@c %**end of header
+
New sets are created with @code{GNUNET_SET_create}. Both the local peer's
configuration (as each set has its own client connection) and the
@node Listeners
@subsubsection Listeners
-@c %**end of header
+
Listeners are created with @code{GNUNET_SET_listen}. Each time time a
remote peer suggests a set operation with an application id and operation
@node Operations
@subsubsection Operations
-@c %**end of header
+
Operations to be initiated by the local peer are created with
@code{GNUNET_SET_prepare}. Note that the operation will not be started
@node Supplying a Set
@subsubsection Supplying a Set
-@c %**end of header
+
To create symmetry between the two ways of starting a set operation
(accepting and initiating it), the operation handles returned by
@node The Result Callback
@subsubsection The Result Callback
-@c %**end of header
+
Clients must specify both a result mode and a result callback with
@code{GNUNET_SET_accept} and @code{GNUNET_SET_prepare}. The result
@node The SET Client-Service Protocol
@subsection The SET Client-Service Protocol
-@c %**end of header
+
@menu
* Creating Sets::
@node Creating Sets
@subsubsection Creating Sets
-@c %**end of header
+
For each set of a client, there exists a client connection to the service.
Sets are created by sending the @code{GNUNET_SERVICE_SET_CREATE} message
@node Listeners2
@subsubsection Listeners2
-@c %**end of header
+
Each listener also requires a seperate client connection. By sending the
@code{GNUNET_SERVICE_SET_LISTEN} message, the client notifies the service
@node Initiating Operations
@subsubsection Initiating Operations
-@c %**end of header
+
Operations with remote peers are initiated by sending a
@code{GNUNET_SERVICE_SET_EVALUATE} message to the service. The@ client
@node Modifying Sets
@subsubsection Modifying Sets
-@c %**end of header
+
Sets are modified with the @code{GNUNET_SERVICE_SET_ADD} and
@code{GNUNET_SERVICE_SET_REMOVE} messages.
@node Results and Operation Status
@subsubsection Results and Operation Status
-@c %**end of header
+
The service notifies the client of result elements and success/failure of
a set operation with the @code{GNUNET_SERVICE_SET_RESULT} message.
@node Iterating Sets
@subsubsection Iterating Sets
-@c %**end of header
+
All elements of a set can be requested by sending
@code{GNUNET_SERVICE_SET_ITER_REQUEST}. The server responds with
@node The SET Intersection Peer-to-Peer Protocol
@subsection The SET Intersection Peer-to-Peer Protocol
-@c %**end of header
+
The intersection protocol operates over CADET and starts with a
GNUNET_MESSAGE_TYPE_SET_P2P_OPERATION_REQUEST being sent by the peer
@node The Bloom filter exchange
@subsubsection The Bloom filter exchange
-@c %**end of header
+
In this phase, each peer transmits a Bloom filter over the remaining
keys of the local set to the other peer using a
@node Salt
@subsubsection Salt
-@c %**end of header
+
Bloomfilter operations are probabilistic: With some non-zero probability
the test may incorrectly say an element is in the set, even though it is
@node The SET Union Peer-to-Peer Protocol
@subsection The SET Union Peer-to-Peer Protocol
-@c %**end of header
+
The SET union protocol is based on Eppstein's efficient set reconciliation
without prior context. You should read this paper first if you want to
@node STATISTICS Subsystem
@section STATISTICS Subsystem
-@c %**end of header
+
In GNUnet, the STATISTICS subsystem offers a central place for all
subsystems to publish unsigned 64-bit integer run-time statistics.
@node libgnunetstatistics
@subsection libgnunetstatistics
-@c %**end of header
+
@strong{libgnunetstatistics} is the library containing the API for the
STATISTICS subsystem. Any process requiring to use STATISTICS should use
@node Statistics retrieval
@subsubsection Statistics retrieval
-@c %**end of header
+
Once a connection to the statistics service is obtained, information
about any other system which uses statistics can be retrieved with the
@node Setting statistics and updating them
@subsubsection Setting statistics and updating them
-@c %**end of header
+
So far we have seen how to retrieve statistics, here we will learn how we
can set statistics and update them so that other subsystems can retrieve
@node Watches
@subsubsection Watches
-@c %**end of header
+
As interesting feature of STATISTICS lies in serving notifications
whenever a statistic of our interest is modified.
@node The STATISTICS Client-Service Protocol
@subsection The STATISTICS Client-Service Protocol
-@c %**end of header
+
@menu
@node Statistics retrieval2
@subsubsection Statistics retrieval2
-@c %**end of header
+
To retrieve statistics, the client transmits a message of type
@code{GNUNET_MESSAGE_TYPE_STATISTICS_GET} containing the given subsystem
@node Setting and updating statistics
@subsubsection Setting and updating statistics
-@c %**end of header
+
The subsystem name, parameter name, its value and the persistence flag are
communicated to the service through the message
@node Watching for updates
@subsubsection Watching for updates
-@c %**end of header
+
The function registers the watch at the service by sending a message of
type @code{GNUNET_MESSAGE_TYPE_STATISTICS_WATCH}. The service then sends
@node Distributed Hash Table (DHT)
@section Distributed Hash Table (DHT)
-@c %**end of header
+
GNUnet includes a generic distributed hash table that can be used by
developers building P2P applications in the framework.
@node Block library and plugins
@subsection Block library and plugins
-@c %**end of header
+
@menu
* What is a Block?::
@node What is a Block?
@subsubsection What is a Block?
-@c %**end of header
+
Blocks are small (< 63k) pieces of data stored under a key (struct
GNUNET_HashCode). Blocks have a type (enum GNUNET_BlockType) which defines
@node The API of libgnunetblock
@subsubsection The API of libgnunetblock
-@c %**end of header
+
The block library requires for each (family of) block type(s) a block
plugin (implementing @file{gnunet_block_plugin.h}) that provides basic
@node Queries
@subsubsection Queries
-@c %**end of header
+
The query format for any block in GNUnet consists of four main components.
First, the type of the desired block must be specified. Second, the query
@node Sample Code
@subsubsection Sample Code
-@c %**end of header
+
The source code in @strong{plugin_block_test.c} is a good starting point
for new block plugins --- it does the minimal work by implementing a
@node Conclusion2
@subsubsection Conclusion2
-@c %**end of header
+
In conclusion, GNUnet subsystems that want to use the DHT need to define a
block format and write a plugin to match queries and replies. For testing,
@node libgnunetdht
@subsection libgnunetdht
-@c %**end of header
+
The DHT API itself is pretty simple and offers the usual GET and PUT
functions that work as expected. The specified block type refers to the
@node GET
@subsubsection GET
-@c %**end of header
+
When using GET, the main consideration for developers (other than the
block library) should be that after issuing a GET, the DHT will
@node PUT
@subsubsection PUT
-@c %**end of header
+
@c inconsistent use of ``must'' above it's written ``MUST''
In contrast to GET operations, developers @strong{must} manually re-run
@node MONITOR
@subsubsection MONITOR
-@c %**end of header
+
The DHT API also allows applications to monitor messages crossing the
local DHT service.
@node DHT Routing Options
@subsubsection DHT Routing Options
-@c %**end of header
+
There are two important options for GET and PUT requests:
@node The DHT Client-Service Protocol
@subsection The DHT Client-Service Protocol
-@c %**end of header
+
@menu
* PUTting data into the DHT::
@node PUTting data into the DHT
@subsubsection PUTting data into the DHT
-@c %**end of header
+
To store (PUT) data into the DHT, the client sends a
@code{struct GNUNET_DHT_ClientPutMessage} to the service.
@node GETting data from the DHT
@subsubsection GETting data from the DHT
-@c %**end of header
+
To retrieve (GET) data from the DHT, the client sends a
@code{struct GNUNET_DHT_ClientGetMessage} to the service. The message
@node Monitoring the DHT
@subsubsection Monitoring the DHT
-@c %**end of header
+
To begin monitoring, the client sends a
@code{struct GNUNET_DHT_MonitorStartStop} message to the DHT service.
@node The DHT Peer-to-Peer Protocol
@subsection The DHT Peer-to-Peer Protocol
-@c %**end of header
+
@menu
@node Routing GETs or PUTs
@subsubsection Routing GETs or PUTs
-@c %**end of header
+
When routing GETs or PUTs, the DHT service selects a suitable subset of
neighbours for forwarding. The exact number of neighbours can be zero or
@node PUTting data into the DHT2
@subsubsection PUTting data into the DHT2
-@c %**end of header
+
To PUT data into the DHT, the service sends a @code{struct PeerPutMessage}
of type @code{GNUNET_MESSAGE_TYPE_DHT_P2P_PUT} to the respective
@node GETting data from the DHT2
@subsubsection GETting data from the DHT2
-@c %**end of header
+
A peer can search the DHT by sending @code{struct PeerGetMessage}s of type
@code{GNUNET_MESSAGE_TYPE_DHT_P2P_GET} to other peers. In addition to the
@node GNU Name System (GNS)
@section GNU Name System (GNS)
-@c %**end of header
+
The GNU Name System (GNS) is a decentralized database that enables users
to securely resolve names to values.
@node libgnunetgns
@subsection libgnunetgns
-@c %**end of header
+
The GNS API itself is extremely simple. Clients first connect to the
GNS service using @code{GNUNET_GNS_connect}.
@node Looking up records
@subsubsection Looking up records
-@c %**end of header
+
@code{GNUNET_GNS_lookup} takes a number of arguments:
@node Accessing the records
@subsubsection Accessing the records
-@c %**end of header
+
The @code{libgnunetgnsrecord} library provides an API to manipulate the
GNS record array that is given to proc. In particular, it offers
@node Creating records
@subsubsection Creating records
-@c %**end of header
+
Creating GNS records is typically done by building the respective record
information (possibly with the help of @code{libgnunetgnsrecord} and
@node Future work
@subsubsection Future work
-@c %**end of header
+
In the future, we want to expand @code{libgnunetgns} to allow
applications to observe shortening operations performed during GNS
@node libgnunetgnsrecord
@subsection libgnunetgnsrecord
-@c %**end of header
+
The @code{libgnunetgnsrecord} library is used to manipulate GNS
records (in plaintext or in their encrypted format).
@node Value handling
@subsubsection Value handling
-@c %**end of header
+
@code{GNUNET_GNSRECORD_value_to_string} can be used to convert
the (binary) representation of a GNS record value to a human readable,
@node Type handling
@subsubsection Type handling
-@c %**end of header
+
@code{GNUNET_GNSRECORD_typename_to_number} can be used to obtain the
numeric value associated with a given typename. For example, given the
@node GNS plugins
@subsection GNS plugins
-@c %**end of header
+
Adding a new GNS record type typically involves writing (or extending) a
GNSRECORD plugin. The plugin needs to implement the
@node The GNS Client-Service Protocol
@subsection The GNS Client-Service Protocol
-@c %**end of header
+
The GNS client-service protocol consists of two simple messages, the
@code{LOOKUP} message and the @code{LOOKUP_RESULT}. Each @code{LOOKUP}
@node Hijacking the DNS-Traffic using gnunet-service-dns
@subsection Hijacking the DNS-Traffic using gnunet-service-dns
-@c %**end of header
+
This section documents how the gnunet-service-dns (and the
gnunet-helper-dns) intercepts DNS queries from the local system.
@node Network Setup Details
@subsubsection Network Setup Details
-@c %**end of header
+
The DNS interceptor adds the following rules to the Linux kernel:
@example
@node Serving DNS lookups via GNS on W32
@subsection Serving DNS lookups via GNS on W32
-@c %**end of header
+
This section documents how the libw32nsp (and
gnunet-gns-helper-service-w32) do DNS resolutions of DNS queries on the
@node Importing DNS Zones into GNS
@subsection Importing DNS Zones into GNS
-@c %**end of header
+This section discusses the challenges and problems faced when writing the
+Ascension tool. It also takes a look at possible improvements in the
+future.
+
+Consider the following diagram that shows the workflow of Ascension:
-This section will mainly comprise of the challenges and problems faced when
-writing the ascension tool.
+@image{images/ascension_ssd,6in,,Ascensions workflow}
-When considering to migrate existing into GNS there are a few things to
-consider.
+Further the interaction between components of GNUnet are shown in the diagram
+below:
+@center @image{images/ascension_interaction,,6in,Ascensions workflow}
@menu
* Conversions between DNS and GNS::
* Performance::
@end menu
+@cindex DNS Conversion
@node Conversions between DNS and GNS
@subsubsection Conversions between DNS and GNS
-The differences between the two name systems lies in the details
-and is not visible from the start. For instance an SRV record is converted to a
-gnunet only BOX record.
+The differences between the two name systems lies in the details and is not
+always transparent. For instance an SRV record is converted to a BOX record
+which is unique to GNS.
-This is done by building a BOX record from an existing SRV record:
+This is done by converting to a BOX record from an existing SRV record:
@example
+# SRV
# _service._proto.name. TTL class SRV priority weight port target
-_sip._tcp.example.com. 14000 IN SRV 0 0 5060 www.example.com.
-@end example
-
-Which can be transformed to a GNS BOX record by converting it like this:
-
-@example
+_sip._tcp.example.com. 14000 IN SRV 0 0 5060 www.example.com.
+# BOX
# TTL BOX flags port protocol recordtype priority weight port target
14000 BOX n 5060 6 33 0 0 5060 www.example.com
@end example
-Other records that have such a transformation is the MX record type, as well as
-the SOA record type.
+Other records that need to undergo such transformation is the MX record type,
+as well as the SOA record type.
+
+Transformation of a SOA record into GNS works as described in the
+following example. Very important to note are the rname and mname keys.
-Transformation of a SOA record into GNS works as described in the following
-example. Very important to note are the rname and mname keys.
@example
# BIND syntax for a clean SOA record
@ IN SOA master.example.com. hostmaster.example.com. (
604800 ; expire
600 ) ; ttl
# Recordline for adding the record
-gnunet-namestore -z example.com -a -n @ -t SOA -V rname=master.example.com \
- mname=hostmaster.example.com 2017030300,3600,1800,604800,600 -e 7200s
+$ gnunet-namestore -z example.com -a -n @ -t SOA -V \
+ rname=master.example.com mname=hostmaster.example.com \
+ 2017030300,3600,1800,604800,600 -e 7200s
@end example
The transformation of MX records is done in a simple way.
@example
# mail.example.com. 3600 IN MX 10 mail.example.com.
-gnunet-namestore -z example.com -n mail -R 3600 MX n 10,mail
+$ gnunet-namestore -z example.com -n mail -R 3600 MX n 10,mail
@end example
-Finally, one of the biggest struggling points was the NS records that are found
-in top level domain zones. The inteded behaviour for those is to add GNS2DNS
-records for the zone so that gnunet-gns can resolve the for those domain on it's
-own. Also a very important aspect of this is, that gnunet needs to be able to
-resolve the nameservers from it's own database. This requires migration of the
-DNS GLUE records as well.
+Finally, one of the biggest struggling points were the NS records that are
+found in top level domain zones. The intended behaviour for those is to add
+GNS2DNS records for those so that gnunet-gns can resolve records for those
+domains on its own. Those require the values from DNS GLUE records, provided
+they are within the same zone.
+
+The following two examples show one record with a GLUE record and the other one
+does not have a GLUE record. This takes place in the 'com' TLD.
-This proved to be quite a challenge to implement, as in GNS every dot is a
-strict zone cut.
+@example
+# ns1.example.com 86400 IN A 127.0.0.1
+# example.com 86400 IN NS ns1.example.com.
+$ gnunet-namestore -z com -n example -R 86400 GNS2DNS n \
+ example.com@@127.0.0.1
+
+# example.com 86400 IN NS ns1.example.org.
+$ gnunet-namestore -z com -n example -R 86400 GNS2DNS n \
+ example.com@@ns1.example.org
+@end example
-The issue was fixed by creating a hierarchical zone structure in GNS and linking
+As you can see, one of the GNS2DNS records has an IP address listed and the
+other one a DNS name. For the first one there is a GLUE record to do the
+translation directly and the second one will issue another DNS query to figure
+out the IP of ns1.example.org.
+
+A solution was found by creating a hierarchical zone structure in GNS and linking
the zones using PKEY records to one another. This allows the resolution of the
-nameservers to work within GNS.
+name servers to work within GNS while not taking control over unwanted zones.
+
+Currently the following record types are supported:
+@itemize @bullet
+@item A
+@item AAAA
+@item CNAME
+@item MX
+@item NS
+@item SRV
+@item TXT
+@end itemize
+
+This is not due to technical limitations but rather a practical ones. The
+problem occurs with DNSSEC enabled DNS zones. As records within those zones are
+signed periodically, and every new signature is an update to the zone, there are
+many revisions of zones. This results in a problem with bigger zones as there
+are lots of records that have been signed again but no major changes. Also
+trying to add records that are unknown that require a different format take time
+as they cause a CLI call of the namestore. Furthermore certain record types
+need transformation into a GNS compatible format which, depending on the record
+type, takes more time.
+
+Further a blacklist was added to drop for instance DNSSEC related records. Also
+if a record type is neither in the white list nor the blacklist it is considered
+as a loss of data and a message is shown to the user. This helps with
+transparency and also with contributing, as the not supported record types can
+then be added accordingly.
@node DNS Zone Size
@subsubsection DNS Zone Size
-
Another very big problem exists with very large zones. When migrating a small
zone the delay between adding of records and their expiry is negligible. However
-when working with a TLD zone that has more that 1 million records this delay
-becomes a problem.
+when working with big zones that easily have more than a few million records
+this delay becomes a problem.
Records will start to expire well before the zone has finished migrating. This
-causes unwanted anomalies when trying to resolve records.
+is usually not a problem but can cause a high CPU load when a peer is restarted
+and the records have expired.
A good solution has not been found yet. One of the idea that floated around was
that the records should be added with the s (shadow) flag to keep the records
to detect if a record has been removed from the zone and would require deletion
of said record(s).
+Another problem that still persists is how to refresh records. Expired records
+are still displayed when calling gnunet-namestore but do not resolve with
+gnunet-gns. Zonemaster will sign the expired records again and make sure that
+the records are still valid. With a recent change this was fixed as gnunet-gns
+to improve the suffix lookup which allows for a fast lookup even with thousands
+of local egos.
+
+Currently the pace of adding records in general is around 10 records per second.
+Crypto is the upper limit for adding of records. The performance of your machine
+can be tested with the perf_crypto_* tools. There is still a big discrepancy
+between the pace of Ascension and the theoretical limit.
+
+A performance metric for measuring improvements has not yet been implemented in
+Ascension.
+
@node Performance
@subsubsection Performance
-The performance when migrating a zone using the ascension tool is limited by a
-handful of factors. First of all ascension is written in python3 and calls the
-CLI tools of gnunet. Furthermore all the records that are added to the same
+The performance when migrating a zone using the Ascension tool is limited by a
+handful of factors. First of all ascension is written in Python3 and calls the
+CLI tools of GNUnet. This is comparable to a fork and exec call which costs a
+few CPU cycles. Furthermore all the records that are added to the same
label are signed using the zones private key. This signing operation is very
resource heavy and was optimized during development by adding the '-R'
-(Recordline) option to gnunet-namestore. This allows to add multiple records
-at once using the CLI.
-
-The result of this was a much faster migration of TLD zones, as most records
-with the same label have two nameservers.
+(Recordline) option to gnunet-namestore which allows to specify multiple records
+using the CLI tool. Assuming that in a TLD zone every domain has at least two
+name servers this halves the amount of signatures needed.
Another improvement that could be made is with the addition of multiple threads
-when opening the gnunet CLI tools. This could be implemented by simply creating
-more workers in the program but performance improvements were not tested.
+or using asynchronous subprocesses when opening the GNUnet CLI tools. This could
+be implemented by simply creating more workers in the program but performance
+improvements were not tested.
-During the entire development of the ascension tool sqlite was used as a
-database backend. Other backends need to be tested in the future.
+Ascension was tested using different hardware and database backends. Performance
+differences between SQLite and postgresql are marginal and almost non existent.
+What did make a huge impact on record adding performance was the storage medium.
+On a traditional mechanical hard drive adding of records were slow compared to a
+solid state disk.
In conclusion there are many bottlenecks still around in the program, namely the
-signing process and the single threaded implementation. In the future a solution
-that uses the c api would be cleaner and better.
-
+single threaded implementation and inefficient, sequential calls of
+gnunet-namestore. In the future a solution that uses the C API would be cleaner
+and better.
@cindex GNS Namecache
@node GNS Namecache
@section GNS Namecache
-@c %**end of header
-
The NAMECACHE subsystem is responsible for caching (encrypted) resolution
results of the GNU Name System (GNS). GNS makes zone information available
to other users via the DHT. However, as accessing the DHT for every
@node libgnunetnamecache
@subsection libgnunetnamecache
-@c %**end of header
+
The NAMECACHE API consists of five simple functions. First, there is
@code{GNUNET_NAMECACHE_connect} to connect to the NAMECACHE service.
@node The NAMECACHE Client-Service Protocol
@subsection The NAMECACHE Client-Service Protocol
-@c %**end of header
+
All messages in the NAMECACHE IPC protocol start with the
@code{struct GNUNET_NAMECACHE_Header} which adds a request
@node Lookup
@subsubsection Lookup
-@c %**end of header
+
The @code{struct LookupBlockMessage} is used to lookup a block stored in
the cache.
@node Store
@subsubsection Store
-@c %**end of header
+
The @code{struct BlockCacheMessage} is used to cache a block in the
NAMECACHE.
@node The NAMECACHE Plugin API
@subsection The NAMECACHE Plugin API
-@c %**end of header
+
The NAMECACHE plugin API consists of two functions, @code{cache_block} to
store a block in the database, and @code{lookup_block} to lookup a block
@node Lookup2
@subsubsection Lookup2
-@c %**end of header
+
The @code{lookup_block} function is expected to return at most one block
to the iterator, and return @code{GNUNET_NO} if there were no non-expired
@node Store2
@subsubsection Store2
-@c %**end of header
+
The @code{cache_block} function is expected to try to store the block in
the database, and return @code{GNUNET_SYSERR} if this was not possible
@cindex REVOCATION Subsystem
@node REVOCATION Subsystem
@section REVOCATION Subsystem
-@c %**end of header
+
The REVOCATION subsystem is responsible for key revocation of Egos.
If a user learns that theis private key has been compromised or has lost
@node Dissemination
@subsection Dissemination
-@c %**end of header
+
When a revocation is performed, the revocation is first of all
disseminated by flooding the overlay network.
@node Revocation Message Design Requirements
@subsection Revocation Message Design Requirements
-@c %**end of header
+
However, flooding is also quite costly, creating O(|E|) messages on a
network with |E| edges.
@node libgnunetrevocation
@subsection libgnunetrevocation
-@c %**end of header
+
The REVOCATION API consists of two parts, to query and to issue
revocations.
@node Querying for revoked keys
@subsubsection Querying for revoked keys
-@c %**end of header
+
@code{GNUNET_REVOCATION_query} is used to check if a given ECDSA public
key has been revoked.
@node Preparing revocations
@subsubsection Preparing revocations
-@c %**end of header
+
It is often desirable to create a revocation record ahead-of-time and
store it in an off-line location to be used later in an emergency.
@node The REVOCATION Peer-to-Peer Protocol
@subsection The REVOCATION Peer-to-Peer Protocol
-@c %**end of header
+
Revocation uses two disjoint ways to spread revocation information among
peers.
@node File-sharing (FS) Subsystem
@section File-sharing (FS) Subsystem
-@c %**end of header
+
This chapter describes the details of how the file-sharing service works.
As with all services, it is split into an API (libgnunetfs), the service
@node Encoding for Censorship-Resistant Sharing (ECRS)
@subsection Encoding for Censorship-Resistant Sharing (ECRS)
-@c %**end of header
+
When GNUnet shares files, it uses a content encoding that is called ECRS,
the Encoding for Censorship-Resistant Sharing.
@node Namespace Advertisements
@subsubsection Namespace Advertisements
-@c %**end of header
+
@c %**FIXME: all zeroses -> ?
An @code{SBlock} with identifier all zeros is a signed
@node KSBlocks
@subsubsection KSBlocks
-@c %**end of header
+
GNUnet implements @code{KSBlocks} which are @code{KBlocks} that, instead
of encrypting a CHK and metadata, encrypt an @code{SBlock} instead.
Collections are also advertised using @code{KSBlock}s.
-@c https://gnunet.org/sites/default/files/ecrs.pdf
+@c https://old.gnunet.org/sites/default/files/ecrs.pdf
@node File-sharing persistence directory structure
@subsection File-sharing persistence directory structure
-@c %**end of header
+
This section documents how the file-sharing library implements
persistence of file-sharing operations and specifically the resulting
@node REGEX Subsystem
@section REGEX Subsystem
-@c %**end of header
+
Using the REGEX subsystem, you can discover peers that offer a particular
service using regular expressions.
@node How to run the regex profiler
@subsection How to run the regex profiler
-@c %**end of header
+
The gnunet-regex-profiler can be used to profile the usage of mesh/regex
for a given set of regular expressions and strings.
Mesh/regex allows you to announce your peer ID under a certain regex and
search for peers matching a particular regex using a string.
-See @uref{https://gnunet.org/szengel2012ms, szengel2012ms} for a full
+See @uref{https://old.gnunet.org/szengel2012ms, szengel2012ms} for a full
introduction.
First of all, the regex profiler uses GNUnet testbed, thus all the
@node REST Subsystem
@section REST Subsystem
-@c %**end of header
+
Using the REST subsystem, you can expose REST-based APIs or services.
The REST service is designed as a pluggable architecture.
This is WIP. Endpoints should be documented appropriately.
Preferably using annotations.
+
+@cindex RPS Subsystem
+@node RPS Subsystem
+@section RPS Subsystem
+
+In literature, Random Peer Sampling (RPS) refers to the problem of
+reliably drawing random samples from an unstructured p2p network.
+
+Doing so in a reliable manner is not only hard because of inherent
+problems but also because of possible malicious peers that could try to
+bias the selection.
+
+It is useful for all kind of gossip protocols that require the selection
+of random peers in the whole network like gathering statistics,
+spreading and aggregating information in the network, load balancing and
+overlay topology management.
+
+The approach chosen in the rps implementation in GNUnet follows the
+Brahms@uref{https://bib.gnunet.org/full/date.html\#2009_5f0} design.
+
+The current state is "work in progress". There are a lot of things that
+need to be done, primarily finishing the experimental evaluation and a
+re-design of the API.
+
+The abstract idea is to subscribe to connect to/start the rps service
+and request random peers that will be returned when they represent a
+random selection from the whole network with high probability.
+
+An additional feature to the original Brahms-design is the selection of
+sub-groups: The GNUnet implementation of rps enables clients to ask for
+random peers from a group that is defined by a common shared secret.
+(The secret could of course also be public, depending on the use-case.)
+
+Another addition to the original protocol was made: The sampler
+mechanism that was introduced in Brahms was slightly adapted and used to
+actually sample the peers and returned to the client.
+This is necessary as the original design only keeps peers connected to
+random other peers in the network. In order to return random peers to
+client requests independently random, they cannot be drawn from the
+connected peers.
+The adapted sampler makes sure that each request for random peers is
+independent from the others.
+
+@node Brahms
+@subsection Brahms
+The high-level concept of Brahms is two-fold: Combining push-pull gossip
+with locally fixing a assumed bias using cryptographic min-wise
+permutations.
+The central data structure is the view - a peer's current local sample.
+This view is used to select peers to push to and pull from.
+This simple mechanism can be biased easily. For this reason Brahms
+'fixes' the bias by using the so-called sampler. A data structure that
+takes a list of elements as input and outputs a random one of them
+independently of the frequency in the input set. Both an element that
+was put into the sampler a single time and an element that was put into
+it a million times have the same probability of being the output.
+This is achieved this is achieved with exploiting min-wise independent
+permutations. In rps we use HMACs: On the initialisation of a sampler
+element, a key is chosen at random. On each input the HMAC with the
+random key is computed. The sampler element keeps the element with the
+minimal HMAC.
+
+In order to fix the bias in the view, a fraction of the elements in the
+view are sampled through the sampler from the random stream of peer IDs.
+
+According to the theoretical analysis of Bortnikov et al. this suffices
+to keep the network connected and having random peers in the view.
+
projects / oweals / gnunet.git / blobdiff