3 @setfilename gnunet-c-tutorial.info
4 @documentencoding UTF-8
5 @settitle GNUnet C Tutorial
9 @c including 'version.texi' makes makeinfo throw errors.
10 @include version2.texi
13 Copyright @copyright{} 2001-2017 GNUnet e.V.
15 Permission is granted to copy, distribute and/or modify this document
16 under the terms of the GNU Free Documentation License, Version 1.3 or
17 any later version published by the Free Software Foundation; with no
18 Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A
19 copy of the license is included in the section entitled ``GNU Free
20 Documentation License''.
22 A copy of the license is also available from the Free Software
23 Foundation Web site at @url{http://www.gnu.org/licenses/fdl.html}.
25 Alternately, this document is also available under the General
26 Public License, version 3 or later, as published by the Free Software
27 Foundation. A copy of the license is included in the section entitled
28 ``GNU General Public License''.
30 A copy of the license is also available from the Free Software
31 Foundation Web site at @url{http://www.gnu.org/licenses/gpl.html}.
36 * GNUnet-C-Tutorial: (gnunet-c-tutorial). C Tutorial for GNunet
41 @title GNUnet C Tutorial
42 @subtitle A Tutorial for GNUnet @value{VERSION} (C version)
43 @author The GNUnet Developers
46 @vskip 0pt plus 1filll
55 @c 2. Either reference main documentation or
56 @c 3. Merge this into main documentation
61 This tutorials explains how to install GNUnet on a
62 GNU/Linux system and gives an introduction on how
63 GNUnet can be used to develop a Peer-to-Peer application.
64 Detailed installation instructions for
65 various operating systems and a detailed list of all
66 dependencies can be found on our website at
67 @uref{https://gnunet.org/installation} and in our
68 Reference Documentation (GNUnet Handbook).
70 Please read this tutorial carefully since every single step is
71 important and do not hesitate to contact the GNUnet team if you have
72 any questions or problems! Check here how to contact the GNUnet
73 team: @uref{https://gnunet.org/contact_information}
77 * Installing GNUnet:: Installing GNUnet
78 * Introduction to GNUnet Architecture:: Introduction to GNUnet Architecture
79 * First Steps with GNUnet:: First Steps with GNUnet
80 * Developing Applications:: Developing Applications
83 --- The Detailed Node Listing ---
87 * Obtaining a stable version::
88 * Installing Build Tool Chain and Dependencies::
89 * Obtaining the latest version from Git::
90 * Compiling and Installing GNUnet::
91 * Common Issues - Check your GNUnet installation::
93 Introduction to GNUnet Architecture
95 First Steps with GNUnet
97 * Configure your peer::
100 * Starting Two Peers by Hand::
101 * Starting Peers Using the Testbed Service::
103 Developing Applications
106 * Adapting the Template::
107 * Writing a Client Application::
108 * Writing a Service::
109 * Interacting directly with other Peers using the CORE Service::
110 * Storing peer-specific data using the PEERSTORE service::
112 * Debugging with gnunet-arm::
117 @node Installing GNUnet
118 @chapter Installing GNUnet
120 First of all you have to install a current version of GNUnet.
121 You can download a tarball of a stable version from GNU FTP mirrors
122 or obtain the latest development version from our Git repository.
124 Most of the time you should prefer to download the stable version
125 since with the latest development version things can be broken,
126 functionality can be changed or tests can fail. You should only use
127 the development version if you know that you require a certain
128 feature or a certain issue has been fixed since the last release.
131 * Obtaining a stable version::
132 * Installing Build Tool Chain and Dependencies::
133 * Obtaining the latest version from Git::
134 * Compiling and Installing GNUnet::
135 * Common Issues - Check your GNUnet installation::
138 @node Obtaining a stable version
139 @section Obtaining a stable version
141 Download the tarball from
142 @indicateurl{https://ftp.gnu.org/gnu/gnunet/gnunet-@value{VERSION}.tar.gz}.
144 Make sure to download the associated @file{.sig} file and to verify the
145 authenticity of the tarball against it, like this:
148 $ wget https://ftp.gnu.org/gnu/gnunet/gnunet-@value{VERSION}.tar.gz.sig
149 $ gpg --verify-files gnunet-@value{VERSION}.tar.gz.sig
153 If this command fails because you do not have the required public key,
154 then you need to run this command to import it:
157 $ gpg --keyserver keys.gnupg.net --recv-keys 48426C7E
161 and rerun the @code{gpg --verify-files} command.
164 @b{The pub key to sign the 0.10.1 release has been
165 revoked}. You will get an error message stating that
166 @b{there is no known public key or that it has been revoked}.
167 The next release of GNUnet will have a valid signature
168 again. We are sorry for the inconvenience this causes.
169 Another possible source you could use is our
170 "gnunet" git repository which has mandatory signed commits
173 Now you can extract the tarball and rename the resulting
174 directory to @file{gnunet} which we will be using in the
175 remainder of this document.
178 $ tar xvzf gnunet-@value{VERSION}.tar.gz
179 $ mv gnunet-@value{VERSION} gnunet
184 However, please note that stable versions can be very outdated.
185 As a developer you are @b{strongly} encouraged to use the version
186 from @uref{https://gnunet.org/git/, git}.
188 @node Installing Build Tool Chain and Dependencies
189 @section Installing Build Tool Chain and Dependencies
191 To successfully compile GNUnet, you need the tools to build GNUnet and
192 the required dependencies. Please take a look at the
193 GNUnet Reference Documentation
194 (@pxref{Dependencies, The GNUnet Reference Documentation,, gnunet, The GNUnet Reference Documentation})
195 for a list of required dependencies
197 (@pxref{Generic installation instructions, The GNUnet Reference Documentation,, gnunet, The GNUnet Reference Documentation})
198 read its Installation chapter for specific instructions for
199 your operating system.
200 Please check the notes at the end of the configure process about
201 required dependencies.
203 For GNUnet bootstrapping support and the HTTP(S) plugin you should
204 install @uref{https://gnunet.org/gnurl, libgnurl}.
205 For the filesharing service you should install at least one of the
206 datastore backends. MySQL, SQlite and PostgreSQL are supported.
208 @node Obtaining the latest version from Git
209 @section Obtaining the latest version from Git
211 The latest development version can obtained from our Git repository.
212 To obtain the code you need to have @code{Git} installed, which is
213 required for obtaining the repository via:
216 $ git clone https://gnunet.org/git/gnunet
220 After cloning the repository you have to execute the @file{bootstrap}
221 script in the new directory:
229 The remainder of this tutorial will assume that you have the
230 Git branch ``master'' checked out.
232 @node Compiling and Installing GNUnet
233 @section Compiling and Installing GNUnet
235 Note: This section is a duplication of the more in depth
236 @pxref{GNUnet Installation Handbook, The GNUnet Reference Documentation,, gnunet, The GNUnet Reference Documentation}.
238 First, you need to install libgnupgerror @geq{} 1.27 and
239 libgcrypt @geq{} 1.7.6:
242 $ export GNUPGFTP="https://www.gnupg.org/ftp/gcrypt"
243 $ wget $GNUPGFTP/libgpg-error/libgpg-error-1.27.tar.bz2
244 $ tar xf libgpg-error-1.27.tar.bz2
245 $ cd libgpg-error-1.27
252 $ export GNUPGFTP="https://www.gnupg.org/ftp/gcrypt"
253 $ wget $GNUPGFTP/libgcrypt/libgcrypt-1.7.6.tar.bz2
254 $ tar xf libgcrypt-1.7.6.tar.bz2
266 @subsection Installation
267 Assuming all dependencies are installed, the following commands will
268 compile and install GNUnet in your home directory. You can specify the
269 directory where GNUnet will be installed by changing the
270 @code{--prefix} value when calling @command{./configure}. If
271 you do not specifiy a prefix, GNUnet is installed in the directory
272 @file{/usr/local}. When developing new applications you may want
273 to enable verbose logging by adding @code{--enable-logging=verbose}:
276 $ ./configure --prefix=$PREFIX --enable-logging
282 After installing GNUnet you have to add your GNUnet installation
283 to your path environmental variable. In addition you have to
284 create the @file{.config} directory in your home directory
285 (unless it already exists) where GNUnet stores its data and an
286 empty GNUnet configuration file:
289 $ export PATH=$PATH:$PREFIX/bin
290 $ echo export PATH=$PREFIX/bin:\\$PATH >> ~/.bashrc
292 $ touch ~/.config/gnunet.conf
295 @node Common Issues - Check your GNUnet installation
296 @section Common Issues - Check your GNUnet installation
298 You should check your installation to ensure that installing GNUnet
299 was successful up to this point. You should be able to access GNUnet's
300 binaries and run GNUnet's self check.
307 should return $PREFIX/bin/gnunet-arm. It should be located in your
308 GNUnet installation and the output should not be empty.
309 If you see an output like:
316 check your PATH variable to ensure GNUnet's @file{bin} directory is
319 GNUnet provides tests for all of its subcomponents. Run
326 to execute tests for all components. @command{make check} traverses all
327 subdirectories in @file{src}. For every subdirectory you should
328 get a message like this:
331 make[2]: Entering directory `/home/$USER/gnunet/contrib'
332 PASS: test_gnunet_prefix
338 @node Introduction to GNUnet Architecture
339 @chapter Introduction to GNUnet Architecture
341 GNUnet is organized in layers and services. Each service is composed of a
342 main service implementation and a client library for other programs to use
343 the service's functionality, described by an API.
344 @c This approach is shown in
345 @c FIXME: enable this once the commented block below works:
346 @c figure~\ref fig:service.
347 Some services provide an additional command line tool to enable the user
348 to interact with the service.
350 Very often it is other GNUnet services that will use these APIs to build
351 the higher layers of GNUnet on top of the lower ones. Each layer expands
352 or extends the functionality of the service below (for instance, to build
353 a mesh on top of a DHT).
354 @c FXIME: See comment above.
355 @c See figure ~\ref fig:interaction for an illustration of this approach.
357 @c ** @image filename[, width[, height[, alttext[, extension]]]]
358 @c FIXME: Texlive (?) 20112 makes the assumption that this means
359 @c 'images/OBJECTNAME.txt' but later versions of it (2017) use this
360 @c syntax as described below.
361 @c TODO: Checkout the makedoc script Guile uses.
364 @c @image{images/gnunet-tutorial-service,,5in,Service with API and network protocol,.png}
365 @c @image{images/gnunet-tutorial-system,,5in,The layered system architecture of GNUnet,.png}
367 @c \begin{figure}[!h]
369 @c % \begin{subfigure}
370 @c \begin{subfigure}[b]{0.3\textwidth}
372 @c \includegraphics[width=\textwidth]{figs/Service.pdf}
373 @c \caption{Service with API and network protocol}
374 @c \label{fig:service}
377 @c \begin{subfigure}[b]{0.3\textwidth}
379 @c \includegraphics[width=\textwidth]{figs/System.pdf}
380 @c \caption{Service interaction}
381 @c \label{fig:interaction}
384 @c \caption{GNUnet's layered system architecture}
387 The main service implementation runs as a standalone process in the
388 operating system and the client code runs as part of the client program,
389 so crashes of a client do not affect the service process or other clients.
390 The service and the clients communicate via a message protocol to be
391 defined and implemented by the programmer.
393 @node First Steps with GNUnet
394 @chapter First Steps with GNUnet
397 * Configure your peer::
400 * Starting Two Peers by Hand::
401 * Starting Peers Using the Testbed Service::
404 @node Configure your peer
405 @section Configure your peer
407 First of all we need to configure your peer. Each peer is started with
408 a configuration containing settings for GNUnet itself and its services.
409 This configuration is based on the default configuration shipped with
410 GNUnet and can be modified. The default configuration is located in the
411 @file{$PREFIX/share/gnunet/config.d} directory. When starting a peer, you
412 can specify a customized configuration using the the @command{-c} command
413 line switch when starting the ARM service and all other services. When
414 using a modified configuration the default values are loaded and only
415 values specified in the configuration file will replace the default
418 Since we want to start additional peers later, we need some modifications
419 from the default configuration. We need to create a separate service
420 home and a file containing our modifications for this peer:
428 Now add the following lines to @file{peer1.conf} to use this directory.
429 For simplified usage we want to prevent the peer to connect to the GNUnet
430 network since this could lead to confusing output. This modifications
431 will replace the default settings:
435 # Use this directory to store GNUnet data
436 GNUNET_HOME = ~/gnunet1/
438 # prevent bootstrapping
443 @section Start a peer
444 Each GNUnet instance (called peer) has an identity (peer ID) based on a
445 cryptographic public private key pair. The peer ID is the printable hash
448 GNUnet services are controlled by a master service, the so called
449 @dfn{Automatic Restart Manager} (ARM). ARM starts, stops and even
450 restarts services automatically or on demand when a client connects.
451 You interact with the ARM service using the @command{gnunet-arm} tool.
452 GNUnet can then be started with @command{gnunet-arm -s} and stopped with
453 @command{gnunet-arm -e}. An additional service not automatically started
454 can be started using @command{gnunet-arm -i <service name>} and stopped
455 using @command{gnunet-arm -k <servicename>}.
457 Once you have started your peer, you can use many other GNUnet commands
458 to interact with it. For example, you can run:
465 to obtain the public key of your peer.
467 You should see an output containing the peer ID similar to:
470 I am peer `0PA02UVRKQTS2C .. JL5Q78F6H0B1ACPV1CJI59MEQUMQCC5G'.
474 @section Monitor a peer
476 In this section, we will monitor the behaviour of our peer's DHT
477 service with respect to a specific key. First we will start
478 GNUnet and then start the DHT service and use the DHT monitor tool
479 to monitor the PUT and GET commands we issue ussing the
480 @command{gnunet-dht-put} and @command{gnunet-dht-get} commands.
481 Using the ``monitor'' line given below, you can observe the behavior
482 of your own peer's DHT with respect to the specified KEY:
485 # start gnunet with all default services:
486 $ gnunet-arm -c ~/peer1.conf -s
488 $ gnunet-arm -c ~/peer1.conf -i dht
489 $ cd ~/gnunet/src/dht;
490 $ ./gnunet-dht-monitor -c ~/peer1.conf -k KEY
494 Now open a separate terminal and change again to
495 the @file{gnunet/src/dht} directory:
498 $ cd ~/gnunet/src/dht
499 # put VALUE under KEY in the DHT:
500 $ ./gnunet-dht-put -c ~/peer1.conf -k KEY -d VALUE
501 # get key KEY from the DHT:
502 $ ./gnunet/src/dht/gnunet-dht-get -c ~/peer1.conf -k KEY
503 # print statistics about current GNUnet state:
504 $ gnunet-statistics -c ~/peer1.conf
505 # print statistics about DHT service:
506 $ gnunet-statistics -c ~/peer1.conf -s dht
509 @node Starting Two Peers by Hand
510 @section Starting Two Peers by Hand
512 This section describes how to start two peers on the same machine by hand.
513 The process is rather painful, but the description is somewhat
514 instructive. In practice, you might prefer the automated method
515 (@pxref{Starting Peers Using the Testbed Service}).
518 * Setup a second peer::
519 * Start the second peer and connect the peers::
520 * How to connect manually::
523 @node Setup a second peer
524 @subsection Setup a second peer
525 We will now start a second peer on your machine.
526 For the second peer, you will need to manually create a modified
527 configuration file to avoid conflicts with ports and directories.
528 A peers configuration file is by default located
529 in @file{~/.gnunet/gnunet.conf}. This file is typically very short
530 or even empty as only the differences to the defaults need to be
531 specified. The defaults are located in many files in the
532 @file{$PREFIX/share/gnunet/config.d} directory.
534 To configure the second peer, use the files
535 @file{$PREFIX/share/gnunet/config.d} as a template for your main
539 $ cat $PREFIX/share/gnunet/config.d/*.conf > peer2.conf
543 Now you have to edit @file{peer2.conf} and change:
546 @item @code{GNUNET\_TEST\_HOME} under @code{PATHS}
547 @item Every (uncommented) value for ``@code{PORT}'' (add 10000) in any
548 section (the option may be commented out if @code{PORT} is
549 prefixed by "\#", in this case, UNIX domain sockets are used
550 and the PORT option does not need to be touched)
551 @item Every value for ``@code{UNIXPATH}'' in any section
552 (e.g. by adding a "-p2" suffix)
555 to a fresh, unique value. Make sure that the PORT numbers stay
556 below 65536. From now on, whenever you interact with the second peer,
557 you need to specify @command{-c peer2.conf} as an additional
558 command line argument.
560 Now, generate the 2nd peer's private key:
563 $ gnunet-peerinfo -s -c peer2.conf
567 This may take a while, generate entropy using your keyboard or mouse
568 as needed. Also, make sure the output is different from the
569 gnunet-peerinfo output for the first peer (otherwise you made an
570 error in the configuration).
572 @node Start the second peer and connect the peers
573 @subsection Start the second peer and connect the peers
575 Then, you can start a second peer using:
578 $ gnunet-arm -c peer2.conf -s
579 $ gnunet-arm -c peer2.conf -i dht
580 $ ~/gnunet/src/dht/gnunet-dht-put -c peer2.conf -k KEY -d VALUE
581 $ ~/gnunet/src/dht/gnunet-dht-get -c peer2.conf -k KEY
584 If you want the two peers to connect, you have multiple options:
587 @item UDP neighbour discovery (automatic)
588 @item Setup a bootstrap server
589 @item Connect manually
592 To setup peer 1 as bootstrapping server change the configuration of
593 the first one to be a hostlist server by adding the following lines to
594 @file{peer1.conf} to enable bootstrapping server:
602 Then change @file{peer2.conf} and replace the ``@code{SERVERS}''
603 line in the ``@code{[hostlist]}'' section with
604 ``@code{http://localhost:8080/}''. Restart both peers using:
608 $ gnunet-arm -c peer1.conf -e
610 $ gnunet-arm -c peer1.conf -s
612 $ gnunet-arm -c peer2.conf -s
616 Note that if you start your peers without changing these settings, they
617 will use the ``global'' hostlist servers of the GNUnet P2P network and
618 likely connect to those peers. At that point, debugging might become
619 tricky as you're going to be connected to many more peers and would
620 likely observe traffic and behaviors that are not explicitly controlled
623 @node How to connect manually
624 @subsection How to connect manually
626 If you want to use the @code{peerinfo} tool to connect your
630 @item Set @code{FORCESTART = NO} in section @code{hostlist}
631 (to not connect to the global GNUnet)
632 @item Start both peers running @command{gnunet-arm -c peer1.conf -s}
633 and @command{gnunet-arm -c peer2.conf -s}
634 @item Get @code{HELLO} message of the first peer running
635 @command{gnunet-peerinfo -c peer1.conf -g}
636 @item Give the output to the second peer by running
637 @command{gnunet-peerinfo -c peer2.conf -p '<output>'}
640 Check that they are connected using @command{gnunet-core -c peer1.conf},
641 which should give you the other peer's peer identity:
644 $ gnunet-core -c peer1.conf
645 Peer `9TVUCS8P5A7ILLBGO6 [...shortened...] 1KNBJ4NGCHP3JPVULDG'
648 @node Starting Peers Using the Testbed Service
649 @section Starting Peers Using the Testbed Service
650 @c \label{sec:testbed}
652 GNUnet's testbed service is used for testing scenarios where
653 a number of peers are to be started. The testbed can manage peers
654 on a single host or on multiple hosts in a distributed fashion.
655 On a single affordable computer, it should be possible to run
656 around tens of peers without drastically increasing the load on the
659 The testbed service can be access through its API
660 @file{include/gnunet\_testbed\_service.h}. The API provides many
661 routines for managing a group of peers. It also provides a helper
662 function @code{GNUNET\_TESTBED\_test\_run()} to quickly setup a
663 minimalistic testing environment on a single host.
665 This function takes a configuration file which will be used as a
666 template configuration for the peers. The testbed takes care of
667 modifying relevant options in the peers' configuration such as
668 @code{SERVICEHOME}, @code{PORT}, @code{UNIXPATH} to unique values
669 so that peers run without running into conflicts. It also checks
670 and assigns the ports in configurations only if they are free.
672 Additionally, the testbed service also reads its options from the
673 same configuration file. Various available options and details
674 about them can be found in the testbed default configuration file
675 @file{src/testbed/testbed.conf}.
677 With the testbed API, a sample test case can be structured as follows:
680 @verbatiminclude testbed_test.c
684 The source code for the above listing can be found at
685 @uref{https://gnunet.org/git/gnunet.git/tree/doc/
686 documentation/testbed_test.c}
687 or in the @file{doc/documentation/} folder of your repository check-out.
688 After installing GNUnet, the above source code can be compiled as:
691 $ export CPPFLAGS="-I/path/to/gnunet/headers"
692 $ export LDFLAGS="-L/path/to/gnunet/libraries"
693 $ gcc $CPPFLAGS $LDFLAGS -o testbed-test testbed_test.c \
694 -lgnunettestbed -lgnunetdht -lgnunetutil
695 # Generate (empty) configuration
696 $ touch template.conf
697 # run it (press CTRL-C to stop)
702 The @code{CPPFLAGS} and @code{LDFLAGS} are necessary if GNUnet
703 is installed into a different directory other than @file{/usr/local}.
705 All of testbed API's peer management functions treat management
706 actions as operations and return operation handles. It is expected
707 that the operations begin immediately, but they may get delayed (to
708 balance out load on the system). The program using the API then has
709 to take care of marking the operation as ``done'' so that its
710 associated resources can be freed immediately and other waiting
711 operations can be executed. Operations will be canceled if they are
712 marked as ``done'' before their completion.
714 An operation is treated as completed when it succeeds or fails.
715 Completion of an operation is either conveyed as events through
716 @dfn{controller event callback} or through respective
717 @dfn{operation completion callbacks}.
718 In functions which support completion notification
719 through both controller event callback and operation
720 completion callback, first the controller event callback will be
721 called. If the operation is not marked as done in that callback
722 or if the callback is given as NULL when creating the operation,
723 the operation completion callback will be called. The API
724 documentation shows which event are to be expected in the
725 controller event notifications. It also documents any exceptional
728 Once the peers are started, test cases often need to connect
729 some of the peers' services. Normally, opening a connect to
730 a peer's service requires the peer's configuration. While using
731 testbed, the testbed automatically generates per-peer configuration.
732 Accessing those configurations directly through file system is
733 discouraged as their locations are dynamically created and will be
734 different among various runs of testbed. To make access to these
735 configurations easy, testbed API provides the function
736 @code{GNUNET\_TESTBED\_service\_connect()}. This function fetches
737 the configuration of a given peer and calls the @dfn{Connect Adapter}.
738 In the example code, it is the @code{dht\_ca}. A connect adapter is
739 expected to open the connection to the needed service by using the
740 provided configuration and return the created service connection handle.
741 Successful connection to the needed service is signaled through
742 @code{service\_connect\_comp\_cb}.
744 A dual to connect adapter is the @dfn{Disconnect Adapter}. This callback
745 is called after the connect adapter has been called when the operation
746 from @code{GNUNET\_TESTBED\_service\_connect()} is marked as ``done''.
747 It has to disconnect from the service with the provided service
748 handle (@code{op\_result}).
750 Exercise: Find out how many peers you can run on your system.
752 Exercise: Find out how to create a 2D torus topology by changing the
753 options in the configuration file.
754 @xref{Supported Topologies, The GNUnet Reference Documentation ,, gnunet, The GNUnet Reference Documentation},
755 then use the DHT API to store and retrieve values in the network.
757 @node Developing Applications
758 @chapter Developing Applications
762 * Adapting the Template::
763 * Writing a Client Application::
764 * Writing a Service::
765 * Interacting directly with other Peers using the CORE Service::
766 * Storing peer-specific data using the PEERSTORE service::
768 * Debugging with gnunet-arm::
773 To develop a new peer-to-peer application or to extend GNUnet we provide
774 a template build system for writing GNUnet extensions in C. It can be
778 $ git clone https://gnunet.org/git/gnunet-ext
781 $ ./configure --prefix=$PREFIX --with-gnunet=$PREFIX
788 The GNUnet ext template includes examples and a working buildsystem
789 for a new GNUnet service. A common GNUnet service consists of the
790 following parts which will be discussed in detail in the remainder
791 of this document. The functionality of a GNUnet service is implemented in:
794 @item the GNUnet service (gnunet-ext/src/ext/gnunet-service-ext.c)
795 @item the client API (gnunet-ext/src/ext/ext_api.c)
796 @item the client application using the service API
797 (gnunet-ext/src/ext/gnunet-ext.c)
800 The interfaces for these entities are defined in:
803 @item client API interface (gnunet-ext/src/ext/ext.h)
804 @item the service interface (gnunet-ext/src/include/gnunet_service_SERVICE.h)
805 @item the P2P protocol (gnunet-ext/src/include/gnunet_protocols_ext.h)
809 In addition the ext systems provides:
812 @item a test testing the API (gnunet-ext/src/ext/test_ext_api.c)
813 @item a configuration template for the service
814 (gnunet-ext/src/ext/ext.conf.in)
817 @node Adapting the Template
818 @section Adapting the Template
820 The first step for writing any extension with a new service is to
821 ensure that the @file{ext.conf.in} file contains entries for the
822 @code{UNIXPATH}, @code{PORT} and @code{BINARY} for the service in a
823 section named after the service.
825 If you want to adapt the template rename the @file{ext.conf.in} to
826 match your services name, you have to modify the @code{AC\_OUTPUT}
827 section in @file{configure.ac} in the @file{gnunet-ext} root.
829 @node Writing a Client Application
830 @section Writing a Client Application
832 When writing any client application (for example, a command-line
833 tool), the basic structure is to start with the
834 @code{GNUNET\_PROGRAM\_run} function. This function will parse
835 command-line options, setup the scheduler and then invoke the
836 @code{run} function (with the remaining non-option arguments)
837 and a handle to the parsed configuration (and the configuration
838 file name that was used, which is typically not needed):
841 @verbatiminclude tutorial-examples/001.c
845 * Handling command-line options::
846 * Writing a Client Library::
847 * Writing a user interface::
850 @node Handling command-line options
851 @subsection Handling command-line options
853 Options can then be added easily by adding global variables and
854 expanding the @code{options} array. For example, the following would
855 add a string-option and a binary flag (defaulting to @code{NULL} and
856 @code{GNUNET\_NO} respectively):
859 @verbatiminclude tutorial-examples/002.c
862 Issues such as displaying some helpful text describing options using
863 the @code{--help} argument and error handling are taken care of when
864 using this approach. Other @code{GNUNET\_GETOPT\_}-functions can be used
865 to obtain integer value options, increment counters, etc. You can
866 even write custom option parsers for special circumstances not covered
867 by the available handlers. To check if an argument was specified by the
868 user you initialize the variable with a specific value (e.g. NULL for
869 a string and GNUNET\_SYSERR for a integer) and check after parsing
870 happened if the values were modified.
872 Inside the @code{run} method, the program would perform the
873 application-specific logic, which typically involves initializing and
874 using some client library to interact with the service. The client
875 library is supposed to implement the IPC whereas the service provides
876 more persistent P2P functions.
878 Exercise: Add a few command-line options and print them inside
879 of @code{run}. What happens if the user gives invalid arguments?
881 @node Writing a Client Library
882 @subsection Writing a Client Library
884 The first and most important step in writing a client library is to
885 decide on an API for the library. Typical API calls include
886 connecting to the service, performing application-specific requests
887 and cleaning up. Many examples for such service APIs can be found
888 in the @file{gnunet/src/include/gnunet\_*\_service.h} files.
890 Then, a client-service protocol needs to be designed. This typically
891 involves defining various message formats in a header that will be
892 included by both the service and the client library (but is otherwise
893 not shared and hence located within the service's directory and not
894 installed by @command{make install}). Each message must start with a
895 @code{struct GNUNET\_MessageHeader} and must be shorter than 64k. By
896 convention, all fields in IPC (and P2P) messages must be in big-endian
897 format (and thus should be read using @code{ntohl} and similar
898 functions and written using @code{htonl} and similar functions).
899 Unique message types must be defined for each message struct in the
900 @file{gnunet\_protocols.h} header (or an extension-specific include
904 * Connecting to the Service::
906 * Receiving Replies from the Service::
909 @node Connecting to the Service
910 @subsubsection Connecting to the Service
912 Before a client library can implement the application-specific protocol
913 with the service, a connection must be created:
916 @verbatiminclude tutorial-examples/003.c
920 As a result a @code{GNUNET\_MQ\_Handle} is returned
921 which can to used henceforth to transmit messages to the service.
922 The complete MQ API can be found in @file{gnunet\_mq\_lib.h}.
923 The @code{hanlders} array in the example above is incomplete.
924 Here is where you will define which messages you expect to
925 receive from the service, and which functions handle them.
926 The @code{error\_cb} is a function that is to be called whenever
927 there are errors communicating with the service.
929 @node Sending messages
930 @subsubsection Sending messages
932 In GNUnet, messages are always sent beginning with a
933 @code{struct GNUNET\_MessageHeader} in big endian format.
934 This header defines the size and the type of the
935 message, the payload follows after this header.
938 @verbatiminclude tutorial-examples/004.c
942 Existing message types are defined in @file{gnunet\_protocols.h}.
943 A common way to create a message is with an envelope:
946 @verbatiminclude tutorial-examples/005.c
950 Exercise: Define a message struct that includes a 32-bit
951 unsigned integer in addition to the standard GNUnet MessageHeader.
952 Add a C struct and define a fresh protocol number for your message.
953 Protocol numbers in gnunet-ext are defined
954 in @file{gnunet-ext/src/include/gnunet_protocols_ext.h}
956 Exercise: Find out how you can determine the number of messages
959 Exercise: Find out how you can determine when a message you
960 have queued was actually transmitted.
962 Exercise: Define a helper function to transmit a 32-bit
963 unsigned integer (as payload) to a service using some given client
966 @node Receiving Replies from the Service
967 @subsubsection Receiving Replies from the Service
969 Clients can receive messages from the service using the handlers
970 specified in the @code{handlers} array we specified when connecting
971 to the service. Entries in the the array are usually created using
972 one of two macros, depending on whether the message is fixed size
973 or variable size. Variable size messages are managed using two
974 callbacks, one to check that the message is well-formed, the other
975 to actually process the message. Fixed size messages are fully
976 checked by the MQ-logic, and thus only need to provide the handler
977 to process the message. Note that the prefixes @code{check\_}
978 and @code{handle\_} are mandatory.
981 @verbatiminclude tutorial-examples/006.c
985 Exercise: Expand your helper function to receive a response message
986 (for example, containing just the @code{struct GNUnet MessageHeader}
987 without any payload). Upon receiving the service's response, you
988 should call a callback provided to your helper function's API.
990 Exercise: Figure out where you can pass values to the
991 closures (@code{cls}).
993 @node Writing a user interface
994 @subsection Writing a user interface
996 Given a client library, all it takes to access a service now is to
997 combine calls to the client library with parsing command-line
1000 Exercise: Call your client API from your @code{run()} method in your
1001 client application to send a request to the service. For example,
1002 send a 32-bit integer value based on a number given at the
1003 command-line to the service.
1005 @node Writing a Service
1006 @section Writing a Service
1008 Before you can test the client you've written so far, you'll
1009 need to also implement the corresponding service.
1013 * Starting a Service::
1016 @node Code Placement
1017 @subsection Code Placement
1019 New services are placed in their own subdirectory under
1020 @file{gnunet/src}. This subdirectory should contain the API
1021 implementation file @file{SERVICE\_api.c}, the description of
1022 the client-service protocol @file{SERVICE.h} and P2P protocol
1023 @file{SERVICE\_protocol.h}, the implementation of the service itself
1024 @file{gnunet-service-SERVICE.h} and several files for tests,
1025 including test code and configuration files.
1027 @node Starting a Service
1028 @subsection Starting a Service
1030 The key API definition for creating a service is the
1031 @code{GNUNET\_SERVICE\_MAIN} macro:
1034 @verbatiminclude tutorial-examples/007.c
1038 In addition to the service name and flags, the macro takes three
1039 functions, typically called @code{run}, @code{client\_connect\_cb} and
1040 @code{client\_disconnect\_cb} as well as an array of message handlers
1041 that will be called for incoming messages from clients.
1043 A minimal version of the three central service funtions would look
1047 @verbatiminclude tutorial-examples/008.c
1051 Exercise: Write a stub service that processes no messages at all
1052 in your code. Create a default configuration for it, integrate it
1053 with the build system and start the service from
1054 @command{gnunet-service-arm} using @command{gnunet-arm -i NAME}.
1056 Exercise: Figure out how to set the closure (@code{cls}) for handlers
1059 Exercise: Figure out how to send messages from the service back to the
1062 Each handler function in the service @b{must} eventually (possibly in some
1063 asynchronous continuation) call
1064 @code{GNUNET\_SERVICE\_client\_continue()}. Only after this call
1065 additional messages from the same client may
1066 be processed. This way, the service can throttle processing messages
1067 from the same client.
1069 Exercise: Change the service to ``handle'' the message from your
1070 client (for now, by printing a message). What happens if you
1071 forget to call @code{GNUNET\_SERVICE\_client\_continue()}?
1073 @node Interacting directly with other Peers using the CORE Service
1074 @section Interacting directly with other Peers using the CORE Service
1076 FIXME: This section still needs to be updated to the lastest API!
1078 One of the most important services in GNUnet is the @code{CORE} service
1079 managing connections between peers and handling encryption between peers.
1081 One of the first things any service that extends the P2P protocol
1082 typically does is connect to the @code{CORE} service using:
1085 @verbatiminclude tutorial-examples/009.c
1089 * New P2P connections::
1090 * Receiving P2P Messages::
1091 * Sending P2P Messages::
1092 * End of P2P connections::
1095 @node New P2P connections
1096 @subsection New P2P connections
1098 Before any traffic with a different peer can be exchanged, the peer must
1099 be known to the service. This is notified by the @code{CORE}
1100 @code{connects} callback, which communicates the identity of the new
1101 peer to the service:
1104 @verbatiminclude tutorial-examples/010.c
1108 Note that whatever you return from @code{connects} is given as the
1109 @code{cls} argument to the message handlers for messages from
1110 the respective peer.
1112 Exercise: Create a service that connects to the @code{CORE}. Then
1113 start (and connect) two peers and print a message once your connect
1114 callback is invoked.
1116 @node Receiving P2P Messages
1117 @subsection Receiving P2P Messages
1119 To receive messages from @code{CORE}, you pass the desired
1120 @code{handlers} to the @code{GNUNET\_CORE\_connect()} function,
1121 just as we showed for services.
1123 It is your responsibility to process messages fast enough or
1124 to implement flow control. If an application does not process
1125 CORE messages fast enough, CORE will randomly drop messages
1126 to not keep a very long queue in memory.
1128 Exercise: Start one peer with a new service that has a message
1129 handler and start a second peer that only has your ``old'' service
1130 without message handlers. Which ``connect'' handlers are invoked when
1131 the two peers are connected? Why?
1133 @node Sending P2P Messages
1134 @subsection Sending P2P Messages
1136 You can transmit messages to other peers using the @code{mq} you were
1137 given during the @code{connect} callback. Note that the @code{mq}
1138 automatically is released upon @code{disconnect} and that you must
1139 not use it afterwards.
1141 It is your responsibility to not over-fill the message queue, GNUnet
1142 will send the messages roughly in the order given as soon as possible.
1144 Exercise: Write a service that upon connect sends messages as
1145 fast as possible to the other peer (the other peer should run a
1146 service that ``processes'' those messages). How fast is the
1147 transmission? Count using the STATISTICS service on both ends. Are
1148 messages lost? How can you transmit messages faster? What happens if
1149 you stop the peer that is receiving your messages?
1151 @node End of P2P connections
1152 @subsection End of P2P connections
1154 If a message handler returns @code{GNUNET\_SYSERR}, the remote
1155 peer shuts down or there is an unrecoverable network
1156 disconnection, CORE notifies the service that the peer disconnected.
1157 After this notification no more messages will be received from the
1158 peer and the service is no longer allowed to send messages to the peer.
1159 The disconnect callback looks like the following:
1162 @verbatiminclude tutorial-examples/011.c
1166 Exercise: Fix your service to handle peer disconnects.
1168 @node Storing peer-specific data using the PEERSTORE service
1169 @section Storing peer-specific data using the PEERSTORE service
1171 GNUnet's PEERSTORE service offers a persistorage for arbitrary
1172 peer-specific data. Other GNUnet services can use the PEERSTORE
1173 to store, retrieve and monitor data records. Each data record
1174 stored with PEERSTORE contains the following fields:
1177 @item subsystem: Name of the subsystem responsible for the record.
1178 @item peerid: Identity of the peer this record is related to.
1179 @item key: a key string identifying the record.
1180 @item value: binary record value.
1181 @item expiry: record expiry date.
1184 The first step is to start a connection to the PEERSTORE service:
1186 @verbatiminclude tutorial-examples/012.c
1189 The service handle @code{peerstore_handle} will be needed for
1190 all subsequent PEERSTORE operations.
1194 * Retrieving records::
1195 * Monitoring records::
1196 * Disconnecting from PEERSTORE::
1199 @node Storing records
1200 @subsection Storing records
1202 To store a new record, use the following function:
1205 @verbatiminclude tutorial-examples/013.c
1209 The @code{options} parameter can either be
1210 @code{GNUNET_PEERSTORE_STOREOPTION_MULTIPLE} which means that multiple
1211 values can be stored under the same key combination
1212 (subsystem, peerid, key), or @code{GNUNET_PEERSTORE_STOREOPTION_REPLACE}
1213 which means that PEERSTORE will replace any existing values under the
1214 given key combination (subsystem, peerid, key) with the new given value.
1216 The continuation function @code{cont} will be called after the store
1217 request is successfully sent to the PEERSTORE service. This does not
1218 guarantee that the record is successfully stored, only that it was
1219 received by the service.
1221 The @code{GNUNET_PEERSTORE_store} function returns a handle to the store
1222 operation. This handle can be used to cancel the store operation only
1223 before the continuation function is called:
1226 @verbatiminclude tutorial-examples/013.1.c
1229 @node Retrieving records
1230 @subsection Retrieving records
1232 To retrieve stored records, use the following function:
1235 @verbatiminclude tutorial-examples/014.c
1239 The values of @code{peer} and @code{key} can be @code{NULL}. This
1240 allows the iteration over values stored under any of the following
1245 @item (subsystem, peerid)
1246 @item (subsystem, key)
1247 @item (subsystem, peerid, key)
1250 The @code{callback} function will be called once with each retrieved
1251 record and once more with a @code{NULL} record to signal the end of
1254 The @code{GNUNET_PEERSTORE_iterate} function returns a handle to the
1255 iterate operation. This handle can be used to cancel the iterate
1256 operation only before the callback function is called with a
1259 @node Monitoring records
1260 @subsection Monitoring records
1262 PEERSTORE offers the functionality of monitoring for new records
1263 stored under a specific key combination (subsystem, peerid, key).
1264 To start the monitoring, use the following function:
1267 @verbatiminclude tutorial-examples/015.c
1271 Whenever a new record is stored under the given key combination,
1272 the @code{callback} function will be called with this new
1273 record. This will continue until the connection to the PEERSTORE
1274 service is broken or the watch operation is canceled:
1277 @verbatiminclude tutorial-examples/016.c
1280 @node Disconnecting from PEERSTORE
1281 @subsection Disconnecting from PEERSTORE
1283 When the connection to the PEERSTORE service is no longer needed,
1284 disconnect using the following function:
1287 @verbatiminclude tutorial-examples/017.c
1291 If the @code{sync_first} flag is set to @code{GNUNET_YES},
1292 the API will delay the disconnection until all store requests
1293 are received by the PEERSTORE service. Otherwise, it will
1294 disconnect immediately.
1297 @section Using the DHT
1299 The DHT allows to store data so other peers in the P2P network can
1300 access it and retrieve data stored by any peers in the network.
1301 This section will explain how to use the DHT. Of course, the first
1302 thing to do is to connect to the DHT service:
1305 @verbatiminclude tutorial-examples/018.c
1309 The second parameter indicates how many requests in parallel to expect.
1310 It is not a hard limit, but a good approximation will make the DHT more
1314 * Storing data in the DHT::
1315 * Obtaining data from the DHT::
1316 * Implementing a block plugin::
1317 * Monitoring the DHT::
1320 @node Storing data in the DHT
1321 @subsection Storing data in the DHT
1322 Since the DHT is a dynamic environment (peers join and leave frequently)
1323 the data that we put in the DHT does not stay there indefinitely. It is
1324 important to ``refresh'' the data periodically by simply storing it
1325 again, in order to make sure other peers can access it.
1327 The put API call offers a callback to signal that the PUT request has been
1328 sent. This does not guarantee that the data is accessible to others peers,
1329 or even that is has been stored, only that the service has requested to
1330 a neighboring peer the retransmission of the PUT request towards its final
1331 destination. Currently there is no feedback about whether or not the data
1332 has been sucessfully stored or where it has been stored. In order to
1333 improve the availablilty of the data and to compensate for possible
1334 errors, peers leaving and other unfavorable events, just make several
1338 @verbatiminclude tutorial-examples/019.c
1342 Exercise: Store a value in the DHT periodically to make sure it
1343 is available over time. You might consider using the function
1344 @code{GNUNET\_SCHEDULER\_add\_delayed} and call
1345 @code{GNUNET\_DHT\_put} from inside a helper function.
1347 @node Obtaining data from the DHT
1348 @subsection Obtaining data from the DHT
1350 As we saw in the previous example, the DHT works in an asynchronous mode.
1351 Each request to the DHT is executed ``in the background'' and the API
1352 calls return immediately. In order to receive results from the DHT, the
1353 API provides a callback. Once started, the request runs in the service,
1354 the service will try to get as many results as possible (filtering out
1355 duplicates) until the timeout expires or we explicitly stop the request.
1356 It is possible to give a ``forever'' timeout with
1357 @code{GNUNET\_TIME\_UNIT\_FOREVER\_REL}.
1359 If we give a route option @code{GNUNET\_DHT\_RO\_RECORD\_ROUTE}
1360 the callback will get a list of all the peers the data has travelled,
1361 both on the PUT path and on the GET path.
1364 @verbatiminclude tutorial-examples/020.c
1368 Exercise: Store a value in the DHT and after a while retrieve it.
1369 Show the IDs of all the peers the requests have gone through.
1370 In order to convert a peer ID to a string, use the function
1371 @code{GNUNET\_i2s}. Pay attention to the route option parameters
1374 @node Implementing a block plugin
1375 @subsection Implementing a block plugin
1377 In order to store data in the DHT, it is necessary to provide a block
1378 plugin. The DHT uses the block plugin to ensure that only well-formed
1379 requests and replies are transmitted over the network.
1381 The block plugin should be put in a file @file{plugin\_block\_SERVICE.c}
1382 in the service's respective directory. The
1383 mandatory functions that need to be implemented for a block plugin are
1384 described in the following sections.
1387 * Validating requests and replies::
1388 * Deriving a key from a reply::
1389 * Initialization of the plugin::
1390 * Shutdown of the plugin::
1391 * Integration of the plugin with the build system::
1394 @node Validating requests and replies
1395 @subsubsection Validating requests and replies
1397 The evaluate function should validate a reply or a request. It returns
1398 a @code{GNUNET\_BLOCK\_EvaluationResult}, which is an enumeration. All
1399 possible answers are in @file{gnunet\_block\_lib.h}. The function will
1400 be called with a @code{reply\_block} argument of @code{NULL} for
1401 requests. Note that depending on how @code{evaluate} is called, only
1402 some of the possible return values are valid. The specific meaning of
1403 the @code{xquery} argument is application-specific. Applications that
1404 do not use an extended query should check that the @code{xquery\_size}
1405 is zero. The block group is typically used to filter duplicate
1409 @verbatiminclude tutorial-examples/021.c
1413 Note that it is mandatory to detect duplicate replies in this function
1414 and return the respective status code. Duplicate detection is
1415 typically done using the Bloom filter block group provided by
1416 @file{libgnunetblockgroup.so}. Failure to do so may cause replies to
1417 circle in the network.
1419 @node Deriving a key from a reply
1420 @subsubsection Deriving a key from a reply
1422 The DHT can operate more efficiently if it is possible to derive a key
1423 from the value of the corresponding block. The @code{get\_key}
1424 function is used to obtain the key of a block --- for example, by
1425 means of hashing. If deriving the key is not possible, the function
1426 should simply return @code{GNUNET\_SYSERR} (the DHT will still work
1427 just fine with such blocks).
1430 @verbatiminclude tutorial-examples/022.c
1433 @node Initialization of the plugin
1434 @subsubsection Initialization of the plugin
1436 The plugin is realized as a shared C library. The library must export
1437 an initialization function which should initialize the plugin. The
1438 initialization function specifies what block types the plugin cares
1439 about and returns a struct with the functions that are to be used for
1440 validation and obtaining keys (the ones just defined above).
1443 @verbatiminclude tutorial-examples/023.c
1446 @node Shutdown of the plugin
1447 @subsubsection Shutdown of the plugin
1449 Following GNUnet's general plugin API concept, the plugin must
1450 export a second function for cleaning up. It usually does very
1454 @verbatiminclude tutorial-examples/024.c
1457 @node Integration of the plugin with the build system
1458 @subsubsection Integration of the plugin with the build system
1460 In order to compile the plugin, the @file{Makefile.am} file for the
1461 service SERVICE should contain a rule similar to this:
1462 @c Actually this is a Makefile not C. But the whole structure of examples
1463 @c must be improved.
1466 @verbatiminclude tutorial-examples/025.c
1470 Exercise: Write a block plugin that accepts all queries
1471 and all replies but prints information about queries and replies
1472 when the respective validation hooks are called.
1474 @node Monitoring the DHT
1475 @subsection Monitoring the DHT
1477 It is possible to monitor the functioning of the local
1478 DHT service. When monitoring the DHT, the service will
1479 alert the monitoring program of any events, both started
1480 locally or received for routing from another peer.
1481 The are three different types of events possible: a
1482 GET request, a PUT request or a response (a reply to a GET).
1484 Since the different events have different associated data,
1485 the API gets 3 different callbacks (one for each message type)
1486 and optional type and key parameters, to allow for filtering of
1487 messages. When an event happens, the appropiate callback is
1488 called with all the information about the event.
1491 @verbatiminclude tutorial-examples/026.c
1494 @node Debugging with gnunet-arm
1495 @section Debugging with gnunet-arm
1497 Even if services are managed by @command{gnunet-arm}, you can
1498 start them with @command{gdb} or @command{valgrind}. For
1499 example, you could add the following lines to your
1500 configuration file to start the DHT service in a @command{gdb}
1501 session in a fresh @command{xterm}:
1505 PREFIX=xterm -e gdb --args
1509 Alternatively, you can stop a service that was started via
1510 ARM and run it manually:
1514 $ gdb --args gnunet-service-dht -L DEBUG
1515 $ valgrind gnunet-service-dht -L DEBUG
1519 Assuming other services are well-written, they will automatically
1520 re-integrate the restarted service with the peer.
1522 GNUnet provides a powerful logging mechanism providing log
1523 levels @code{ERROR}, @code{WARNING}, @code{INFO} and @code{DEBUG}.
1524 The current log level is configured using the @code{$GNUNET_FORCE_LOG}
1525 environmental variable. The @code{DEBUG} level is only available if
1526 @command{--enable-logging=verbose} was used when running
1527 @command{configure}. More details about logging can be found under
1528 @uref{https://gnunet.org/logging}.
1530 You should also probably enable the creation of core files, by setting
1531 @code{ulimit}, and echo'ing @code{1} into
1532 @file{/proc/sys/kernel/core\_uses\_pid}. Then you can investigate the
1533 core dumps with @command{gdb}, which is often the fastest method to
1536 Exercise: Add a memory leak to your service and obtain a trace
1537 pointing to the leak using @command{valgrind} while running the service
1538 from @command{gnunet-service-arm}.