3 @setfilename gnunet-c-tutorial.info
4 @documentencoding UTF-8
5 @settitle GNUnet C Tutorial
9 @c including 'version.texi' makes makeinfo throw errors.
10 @include version2.texi
13 Copyright @copyright{} 2001-2017 GNUnet e.V.
15 Permission is granted to copy, distribute and/or modify this document
16 under the terms of the GNU Free Documentation License, Version 1.3 or
17 any later version published by the Free Software Foundation; with no
18 Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A
19 copy of the license is included in the section entitled ``GNU Free
20 Documentation License''.
22 A copy of the license is also available from the Free Software
23 Foundation Web site at @url{http://www.gnu.org/licenses/fdl.html}.
25 Alternately, this document is also available under the General
26 Public License, version 3 or later, as published by the Free Software
27 Foundation. A copy of the license is included in the section entitled
28 ``GNU General Public License''.
30 A copy of the license is also available from the Free Software
31 Foundation Web site at @url{http://www.gnu.org/licenses/gpl.html}.
36 * GNUnet-C-Tutorial: (gnunet-c-tutorial). C Tutorial for GNunet
41 @title GNUnet C Tutorial
42 @subtitle A Tutorial for GNUnet @value{VERSION} (C version)
43 @author The GNUnet Developers
46 @vskip 0pt plus 1filll
55 @c 2. Either reference main documentation or
56 @c 3. Merge this into main documentation
61 This tutorials explains how to install GNUnet on a
62 GNU/Linux system and gives an introduction on how
63 GNUnet can be used to develop a Peer-to-Peer application.
64 Detailed installation instructions for
65 various operating systems and a detailed list of all
66 dependencies can be found on our website at
67 @uref{https://gnunet.org/installation} and in our
68 Reference Documentation (GNUnet Handbook).
70 Please read this tutorial carefully since every single step is
71 important and do not hesitate to contact the GNUnet team if you have
72 any questions or problems! Check here how to contact the GNUnet
73 team: @uref{https://gnunet.org/contact_information}
77 * Installing GNUnet:: Installing GNUnet
78 * Introduction to GNUnet Architecture:: Introduction to GNUnet Architecture
79 * First Steps with GNUnet:: First Steps with GNUnet
80 * Developing Applications:: Developing Applications
83 --- The Detailed Node Listing ---
87 * Obtaining a stable version::
88 * Installing Build Tool Chain and Dependencies::
89 * Obtaining the latest version from Git::
90 * Compiling and Installing GNUnet::
91 * Common Issues - Check your GNUnet installation::
93 Introduction to GNUnet Architecture
95 First Steps with GNUnet
97 * Configure your peer::
100 * Starting Two Peers by Hand::
101 * Starting Peers Using the Testbed Service::
103 Developing Applications
106 * Adapting the Template::
107 * Writing a Client Application::
108 * Writing a Service::
109 * Interacting directly with other Peers using the CORE Service::
110 * Storing peer-specific data using the PEERSTORE service::
112 * Debugging with gnunet-arm::
117 @node Installing GNUnet
118 @chapter Installing GNUnet
120 First of all you have to install a current version of GNUnet.
121 You can download a tarball of a stable version from GNU FTP mirrors
122 or obtain the latest development version from our Git repository.
124 Most of the time you should prefer to download the stable version
125 since with the latest development version things can be broken,
126 functionality can be changed or tests can fail. You should only use
127 the development version if you know that you require a certain
128 feature or a certain issue has been fixed since the last release.
131 * Obtaining a stable version::
132 * Installing Build Tool Chain and Dependencies::
133 * Obtaining the latest version from Git::
134 * Compiling and Installing GNUnet::
135 * Common Issues - Check your GNUnet installation::
138 @node Obtaining a stable version
139 @section Obtaining a stable version
141 Download the tarball from
142 @indicateurl{https://ftp.gnu.org/gnu/gnunet/gnunet-@value{VERSION}.tar.gz}.
144 Make sure to download the associated @file{.sig} file and to verify the
145 authenticity of the tarball against it, like this:
148 $ wget https://ftp.gnu.org/gnu/gnunet/gnunet-@value{VERSION}.tar.gz.sig
149 $ gpg --verify-files gnunet-@value{VERSION}.tar.gz.sig
153 If this command fails because you do not have the required public key,
154 then you need to run this command to import it:
157 $ gpg --keyserver keys.gnupg.net --recv-keys 48426C7E
161 and rerun the @code{gpg --verify-files} command.
164 @b{The pub key to sign the 0.10.1 release has been
165 revoked}. You will get an error message stating that
166 @b{there is no known public key or that it has been revoked}.
167 The next release of GNUnet will have a valid signature
168 again. We are sorry for the inconvenience this causes.
169 Another possible source you could use is our
170 "gnunet" git repository which has mandatory signed commits
173 Now you can extract the tarball and rename the resulting
174 directory to @file{gnunet} which we will be using in the
175 remainder of this document.
178 $ tar xvzf gnunet-@value{VERSION}.tar.gz
179 $ mv gnunet-@value{VERSION} gnunet
184 However, please note that stable versions can be very outdated.
185 As a developer you are @b{strongly} encouraged to use the version
186 from @uref{https://gnunet.org/git/, git}.
188 @node Installing Build Tool Chain and Dependencies
189 @section Installing Build Tool Chain and Dependencies
191 To successfully compile GNUnet, you need the tools to build GNUnet and
192 the required dependencies. Please take a look at the
193 GNUnet Reference Documentation
194 (@pxref{Dependencies, The GNUnet Reference Documentation,, gnunet, The GNUnet Reference Documentation})
195 for a list of required dependencies
197 (@pxref{Generic installation instructions, The GNUnet Reference Documentation,, gnunet, The GNUnet Reference Documentation})
198 read its Installation chapter for specific instructions for
199 your operating system.
200 Please check the notes at the end of the configure process about
201 required dependencies.
203 For GNUnet bootstrapping support and the HTTP(S) plugin you should
204 install @uref{https://gnunet.org/gnurl, libgnurl}.
205 For the filesharing service you should install at least one of the
206 datastore backends. MySQL, SQlite and PostgreSQL are supported.
208 @node Obtaining the latest version from Git
209 @section Obtaining the latest version from Git
211 The latest development version can obtained from our Git repository.
212 To obtain the code you need to have @code{Git} installed, which is
213 required for obtaining the repository via:
216 $ git clone https://gnunet.org/git/gnunet
220 After cloning the repository you have to execute the @file{bootstrap}
221 script in the new directory:
229 The remainder of this tutorial will assume that you have the
230 Git branch ``master'' checked out.
232 @node Compiling and Installing GNUnet
233 @section Compiling and Installing GNUnet
235 Note: This section is a duplication of the more in depth
236 @pxref{GNUnet Installation Handbook, The GNUnet Reference Documentation,, gnunet, The GNUnet Reference Documentation}.
238 First, you need to install libgnupgerror @geq{} 1.27 and
239 libgcrypt @geq{} 1.7.6:
242 $ export GNUPGFTP="https://www.gnupg.org/ftp/gcrypt"
243 $ wget $GNUPGFTP/libgpg-error/libgpg-error-1.27.tar.bz2
244 $ tar xf libgpg-error-1.27.tar.bz2
245 $ cd libgpg-error-1.27
253 $ export GNUPGFTP="https://www.gnupg.org/ftp/gcrypt"
254 $ wget $GNUPGFTP/libgcrypt/libgcrypt-1.7.6.tar.bz2
255 $ tar xf libgcrypt-1.7.6.tar.bz2
268 @subsection Installation
269 Assuming all dependencies are installed, the following commands will
270 compile and install GNUnet in your home directory. You can specify the
271 directory where GNUnet will be installed by changing the
272 @code{--prefix} value when calling @command{./configure}. If
273 you do not specifiy a prefix, GNUnet is installed in the directory
274 @file{/usr/local}. When developing new applications you may want
275 to enable verbose logging by adding @code{--enable-logging=verbose}:
278 $ ./configure --prefix=$PREFIX --enable-logging
284 After installing GNUnet you have to add your GNUnet installation
285 to your path environmental variable. In addition you have to
286 create the @file{.config} directory in your home directory
287 (unless it already exists) where GNUnet stores its data and an
288 empty GNUnet configuration file:
291 $ export PATH=$PATH:$PREFIX/bin
292 $ echo export PATH=$PREFIX/bin:\\$PATH >> ~/.bashrc
294 $ touch ~/.config/gnunet.conf
297 @node Common Issues - Check your GNUnet installation
298 @section Common Issues - Check your GNUnet installation
300 You should check your installation to ensure that installing GNUnet
301 was successful up to this point. You should be able to access GNUnet's
302 binaries and run GNUnet's self check.
309 should return $PREFIX/bin/gnunet-arm. It should be located in your
310 GNUnet installation and the output should not be empty.
311 If you see an output like:
318 check your PATH variable to ensure GNUnet's @file{bin} directory is
321 GNUnet provides tests for all of its subcomponents. Run
328 to execute tests for all components. @command{make check} traverses all
329 subdirectories in @file{src}. For every subdirectory you should
330 get a message like this:
333 make[2]: Entering directory `/home/$USER/gnunet/contrib'
334 PASS: test_gnunet_prefix
340 @node Introduction to GNUnet Architecture
341 @chapter Introduction to GNUnet Architecture
343 GNUnet is organized in layers and services. Each service is composed of a
344 main service implementation and a client library for other programs to use
345 the service's functionality, described by an API.
346 @c This approach is shown in
347 @c FIXME: enable this once the commented block below works:
348 @c figure~\ref fig:service.
349 Some services provide an additional command line tool to enable the user
350 to interact with the service.
352 Very often it is other GNUnet services that will use these APIs to build
353 the higher layers of GNUnet on top of the lower ones. Each layer expands
354 or extends the functionality of the service below (for instance, to build
355 a mesh on top of a DHT).
356 @c FXIME: See comment above.
357 @c See figure ~\ref fig:interaction for an illustration of this approach.
359 @c ** @image filename[, width[, height[, alttext[, extension]]]]
360 @c FIXME: Texlive (?) 20112 makes the assumption that this means
361 @c 'images/OBJECTNAME.txt' but later versions of it (2017) use this
362 @c syntax as described below.
363 @c TODO: Checkout the makedoc script Guile uses.
366 @c @image{images/gnunet-tutorial-service,,5in,Service with API and network protocol,.png}
367 @c @image{images/gnunet-tutorial-system,,5in,The layered system architecture of GNUnet,.png}
369 @c \begin{figure}[!h]
371 @c % \begin{subfigure}
372 @c \begin{subfigure}[b]{0.3\textwidth}
374 @c \includegraphics[width=\textwidth]{figs/Service.pdf}
375 @c \caption{Service with API and network protocol}
376 @c \label{fig:service}
379 @c \begin{subfigure}[b]{0.3\textwidth}
381 @c \includegraphics[width=\textwidth]{figs/System.pdf}
382 @c \caption{Service interaction}
383 @c \label{fig:interaction}
386 @c \caption{GNUnet's layered system architecture}
389 The main service implementation runs as a standalone process in the
390 operating system and the client code runs as part of the client program,
391 so crashes of a client do not affect the service process or other clients.
392 The service and the clients communicate via a message protocol to be
393 defined and implemented by the programmer.
395 @node First Steps with GNUnet
396 @chapter First Steps with GNUnet
399 * Configure your peer::
402 * Starting Two Peers by Hand::
403 * Starting Peers Using the Testbed Service::
406 @node Configure your peer
407 @section Configure your peer
409 First of all we need to configure your peer. Each peer is started with
410 a configuration containing settings for GNUnet itself and its services.
411 This configuration is based on the default configuration shipped with
412 GNUnet and can be modified. The default configuration is located in the
413 @file{$PREFIX/share/gnunet/config.d} directory. When starting a peer, you
414 can specify a customized configuration using the the @command{-c} command
415 line switch when starting the ARM service and all other services. When
416 using a modified configuration the default values are loaded and only
417 values specified in the configuration file will replace the default
420 Since we want to start additional peers later, we need some modifications
421 from the default configuration. We need to create a separate service
422 home and a file containing our modifications for this peer:
430 Now add the following lines to @file{peer1.conf} to use this directory.
431 For simplified usage we want to prevent the peer to connect to the GNUnet
432 network since this could lead to confusing output. This modifications
433 will replace the default settings:
437 # Use this directory to store GNUnet data
438 GNUNET_HOME = ~/gnunet1/
440 # prevent bootstrapping
445 @section Start a peer
446 Each GNUnet instance (called peer) has an identity (peer ID) based on a
447 cryptographic public private key pair. The peer ID is the printable hash
450 GNUnet services are controlled by a master service, the so called
451 @dfn{Automatic Restart Manager} (ARM). ARM starts, stops and even
452 restarts services automatically or on demand when a client connects.
453 You interact with the ARM service using the @command{gnunet-arm} tool.
454 GNUnet can then be started with @command{gnunet-arm -s} and stopped with
455 @command{gnunet-arm -e}. An additional service not automatically started
456 can be started using @command{gnunet-arm -i <service name>} and stopped
457 using @command{gnunet-arm -k <servicename>}.
459 Once you have started your peer, you can use many other GNUnet commands
460 to interact with it. For example, you can run:
467 to obtain the public key of your peer.
469 You should see an output containing the peer ID similar to:
472 I am peer `0PA02UVRKQTS2C .. JL5Q78F6H0B1ACPV1CJI59MEQUMQCC5G'.
476 @section Monitor a peer
478 In this section, we will monitor the behaviour of our peer's DHT
479 service with respect to a specific key. First we will start
480 GNUnet and then start the DHT service and use the DHT monitor tool
481 to monitor the PUT and GET commands we issue ussing the
482 @command{gnunet-dht-put} and @command{gnunet-dht-get} commands.
483 Using the ``monitor'' line given below, you can observe the behavior
484 of your own peer's DHT with respect to the specified KEY:
487 # start gnunet with all default services:
488 $ gnunet-arm -c ~/peer1.conf -s
490 $ gnunet-arm -c ~/peer1.conf -i dht
491 $ cd ~/gnunet/src/dht;
492 $ ./gnunet-dht-monitor -c ~/peer1.conf -k KEY
496 Now open a separate terminal and change again to
497 the @file{gnunet/src/dht} directory:
500 $ cd ~/gnunet/src/dht
501 # put VALUE under KEY in the DHT:
502 $ ./gnunet-dht-put -c ~/peer1.conf -k KEY -d VALUE
503 # get key KEY from the DHT:
504 $ ./gnunet/src/dht/gnunet-dht-get -c ~/peer1.conf -k KEY
505 # print statistics about current GNUnet state:
506 $ gnunet-statistics -c ~/peer1.conf
507 # print statistics about DHT service:
508 $ gnunet-statistics -c ~/peer1.conf -s dht
511 @node Starting Two Peers by Hand
512 @section Starting Two Peers by Hand
514 This section describes how to start two peers on the same machine by hand.
515 The process is rather painful, but the description is somewhat
516 instructive. In practice, you might prefer the automated method
517 (@pxref{Starting Peers Using the Testbed Service}).
520 * Setup a second peer::
521 * Start the second peer and connect the peers::
522 * How to connect manually::
525 @node Setup a second peer
526 @subsection Setup a second peer
527 We will now start a second peer on your machine.
528 For the second peer, you will need to manually create a modified
529 configuration file to avoid conflicts with ports and directories.
530 A peers configuration file is by default located
531 in @file{~/.gnunet/gnunet.conf}. This file is typically very short
532 or even empty as only the differences to the defaults need to be
533 specified. The defaults are located in many files in the
534 @file{$PREFIX/share/gnunet/config.d} directory.
536 To configure the second peer, use the files
537 @file{$PREFIX/share/gnunet/config.d} as a template for your main
541 $ cat $PREFIX/share/gnunet/config.d/*.conf > peer2.conf
545 Now you have to edit @file{peer2.conf} and change:
548 @item @code{GNUNET\_TEST\_HOME} under @code{PATHS}
549 @item Every (uncommented) value for ``@code{PORT}'' (add 10000) in any
550 section (the option may be commented out if @code{PORT} is
551 prefixed by "\#", in this case, UNIX domain sockets are used
552 and the PORT option does not need to be touched)
553 @item Every value for ``@code{UNIXPATH}'' in any section
554 (e.g. by adding a "-p2" suffix)
557 to a fresh, unique value. Make sure that the PORT numbers stay
558 below 65536. From now on, whenever you interact with the second peer,
559 you need to specify @command{-c peer2.conf} as an additional
560 command line argument.
562 Now, generate the 2nd peer's private key:
565 $ gnunet-peerinfo -s -c peer2.conf
569 This may take a while, generate entropy using your keyboard or mouse
570 as needed. Also, make sure the output is different from the
571 gnunet-peerinfo output for the first peer (otherwise you made an
572 error in the configuration).
574 @node Start the second peer and connect the peers
575 @subsection Start the second peer and connect the peers
577 Then, you can start a second peer using:
580 $ gnunet-arm -c peer2.conf -s
581 $ gnunet-arm -c peer2.conf -i dht
582 $ ~/gnunet/src/dht/gnunet-dht-put -c peer2.conf -k KEY -d VALUE
583 $ ~/gnunet/src/dht/gnunet-dht-get -c peer2.conf -k KEY
586 If you want the two peers to connect, you have multiple options:
589 @item UDP neighbour discovery (automatic)
590 @item Setup a bootstrap server
591 @item Connect manually
594 To setup peer 1 as bootstrapping server change the configuration of
595 the first one to be a hostlist server by adding the following lines to
596 @file{peer1.conf} to enable bootstrapping server:
604 Then change @file{peer2.conf} and replace the ``@code{SERVERS}''
605 line in the ``@code{[hostlist]}'' section with
606 ``@code{http://localhost:8080/}''. Restart both peers using:
610 $ gnunet-arm -c peer1.conf -e
612 $ gnunet-arm -c peer1.conf -s
614 $ gnunet-arm -c peer2.conf -s
618 Note that if you start your peers without changing these settings, they
619 will use the ``global'' hostlist servers of the GNUnet P2P network and
620 likely connect to those peers. At that point, debugging might become
621 tricky as you're going to be connected to many more peers and would
622 likely observe traffic and behaviors that are not explicitly controlled
625 @node How to connect manually
626 @subsection How to connect manually
628 If you want to use the @code{peerinfo} tool to connect your
632 @item Set @code{FORCESTART = NO} in section @code{hostlist}
633 (to not connect to the global GNUnet)
634 @item Start both peers running @command{gnunet-arm -c peer1.conf -s}
635 and @command{gnunet-arm -c peer2.conf -s}
636 @item Get @code{HELLO} message of the first peer running
637 @command{gnunet-peerinfo -c peer1.conf -g}
638 @item Give the output to the second peer by running
639 @command{gnunet-peerinfo -c peer2.conf -p '<output>'}
642 Check that they are connected using @command{gnunet-core -c peer1.conf},
643 which should give you the other peer's peer identity:
646 $ gnunet-core -c peer1.conf
647 Peer `9TVUCS8P5A7ILLBGO6 [...shortened...] 1KNBJ4NGCHP3JPVULDG'
650 @node Starting Peers Using the Testbed Service
651 @section Starting Peers Using the Testbed Service
652 @c \label{sec:testbed}
654 GNUnet's testbed service is used for testing scenarios where
655 a number of peers are to be started. The testbed can manage peers
656 on a single host or on multiple hosts in a distributed fashion.
657 On a single affordable computer, it should be possible to run
658 around tens of peers without drastically increasing the load on the
661 The testbed service can be access through its API
662 @file{include/gnunet\_testbed\_service.h}. The API provides many
663 routines for managing a group of peers. It also provides a helper
664 function @code{GNUNET\_TESTBED\_test\_run()} to quickly setup a
665 minimalistic testing environment on a single host.
667 This function takes a configuration file which will be used as a
668 template configuration for the peers. The testbed takes care of
669 modifying relevant options in the peers' configuration such as
670 @code{SERVICEHOME}, @code{PORT}, @code{UNIXPATH} to unique values
671 so that peers run without running into conflicts. It also checks
672 and assigns the ports in configurations only if they are free.
674 Additionally, the testbed service also reads its options from the
675 same configuration file. Various available options and details
676 about them can be found in the testbed default configuration file
677 @file{src/testbed/testbed.conf}.
679 With the testbed API, a sample test case can be structured as follows:
682 @verbatiminclude testbed_test.c
686 The source code for the above listing can be found at
687 @uref{https://gnunet.org/git/gnunet.git/tree/doc/
688 documentation/testbed_test.c}
689 or in the @file{doc/documentation/} folder of your repository check-out.
690 After installing GNUnet, the above source code can be compiled as:
693 $ export CPPFLAGS="-I/path/to/gnunet/headers"
694 $ export LDFLAGS="-L/path/to/gnunet/libraries"
695 $ gcc $CPPFLAGS $LDFLAGS -o testbed-test testbed_test.c \
696 -lgnunettestbed -lgnunetdht -lgnunetutil
697 # Generate (empty) configuration
698 $ touch template.conf
699 # run it (press CTRL-C to stop)
704 The @code{CPPFLAGS} and @code{LDFLAGS} are necessary if GNUnet
705 is installed into a different directory other than @file{/usr/local}.
707 All of testbed API's peer management functions treat management
708 actions as operations and return operation handles. It is expected
709 that the operations begin immediately, but they may get delayed (to
710 balance out load on the system). The program using the API then has
711 to take care of marking the operation as ``done'' so that its
712 associated resources can be freed immediately and other waiting
713 operations can be executed. Operations will be canceled if they are
714 marked as ``done'' before their completion.
716 An operation is treated as completed when it succeeds or fails.
717 Completion of an operation is either conveyed as events through
718 @dfn{controller event callback} or through respective
719 @dfn{operation completion callbacks}.
720 In functions which support completion notification
721 through both controller event callback and operation
722 completion callback, first the controller event callback will be
723 called. If the operation is not marked as done in that callback
724 or if the callback is given as NULL when creating the operation,
725 the operation completion callback will be called. The API
726 documentation shows which event are to be expected in the
727 controller event notifications. It also documents any exceptional
730 Once the peers are started, test cases often need to connect
731 some of the peers' services. Normally, opening a connect to
732 a peer's service requires the peer's configuration. While using
733 testbed, the testbed automatically generates per-peer configuration.
734 Accessing those configurations directly through file system is
735 discouraged as their locations are dynamically created and will be
736 different among various runs of testbed. To make access to these
737 configurations easy, testbed API provides the function
738 @code{GNUNET\_TESTBED\_service\_connect()}. This function fetches
739 the configuration of a given peer and calls the @dfn{Connect Adapter}.
740 In the example code, it is the @code{dht\_ca}. A connect adapter is
741 expected to open the connection to the needed service by using the
742 provided configuration and return the created service connection handle.
743 Successful connection to the needed service is signaled through
744 @code{service\_connect\_comp\_cb}.
746 A dual to connect adapter is the @dfn{Disconnect Adapter}. This callback
747 is called after the connect adapter has been called when the operation
748 from @code{GNUNET\_TESTBED\_service\_connect()} is marked as ``done''.
749 It has to disconnect from the service with the provided service
750 handle (@code{op\_result}).
752 Exercise: Find out how many peers you can run on your system.
754 Exercise: Find out how to create a 2D torus topology by changing the
755 options in the configuration file.
756 @xref{Supported Topologies, The GNUnet Reference Documentation ,, gnunet, The GNUnet Reference Documentation},
757 then use the DHT API to store and retrieve values in the network.
759 @node Developing Applications
760 @chapter Developing Applications
764 * Adapting the Template::
765 * Writing a Client Application::
766 * Writing a Service::
767 * Interacting directly with other Peers using the CORE Service::
768 * Storing peer-specific data using the PEERSTORE service::
770 * Debugging with gnunet-arm::
775 To develop a new peer-to-peer application or to extend GNUnet we provide
776 a template build system for writing GNUnet extensions in C. It can be
780 $ git clone https://gnunet.org/git/gnunet-ext
783 $ ./configure --prefix=$PREFIX --with-gnunet=$PREFIX
790 The GNUnet ext template includes examples and a working buildsystem
791 for a new GNUnet service. A common GNUnet service consists of the
792 following parts which will be discussed in detail in the remainder
793 of this document. The functionality of a GNUnet service is implemented in:
796 @item the GNUnet service (gnunet-ext/src/ext/gnunet-service-ext.c)
797 @item the client API (gnunet-ext/src/ext/ext_api.c)
798 @item the client application using the service API
799 (gnunet-ext/src/ext/gnunet-ext.c)
802 The interfaces for these entities are defined in:
805 @item client API interface (gnunet-ext/src/ext/ext.h)
806 @item the service interface (gnunet-ext/src/include/gnunet_service_SERVICE.h)
807 @item the P2P protocol (gnunet-ext/src/include/gnunet_protocols_ext.h)
811 In addition the ext systems provides:
814 @item a test testing the API (gnunet-ext/src/ext/test_ext_api.c)
815 @item a configuration template for the service
816 (gnunet-ext/src/ext/ext.conf.in)
819 @node Adapting the Template
820 @section Adapting the Template
822 The first step for writing any extension with a new service is to
823 ensure that the @file{ext.conf.in} file contains entries for the
824 @code{UNIXPATH}, @code{PORT} and @code{BINARY} for the service in a
825 section named after the service.
827 If you want to adapt the template rename the @file{ext.conf.in} to
828 match your services name, you have to modify the @code{AC\_OUTPUT}
829 section in @file{configure.ac} in the @file{gnunet-ext} root.
831 @node Writing a Client Application
832 @section Writing a Client Application
834 When writing any client application (for example, a command-line
835 tool), the basic structure is to start with the
836 @code{GNUNET\_PROGRAM\_run} function. This function will parse
837 command-line options, setup the scheduler and then invoke the
838 @code{run} function (with the remaining non-option arguments)
839 and a handle to the parsed configuration (and the configuration
840 file name that was used, which is typically not needed):
843 @verbatiminclude tutorial-examples/001.c
847 * Handling command-line options::
848 * Writing a Client Library::
849 * Writing a user interface::
852 @node Handling command-line options
853 @subsection Handling command-line options
855 Options can then be added easily by adding global variables and
856 expanding the @code{options} array. For example, the following would
857 add a string-option and a binary flag (defaulting to @code{NULL} and
858 @code{GNUNET\_NO} respectively):
861 @verbatiminclude tutorial-examples/002.c
864 Issues such as displaying some helpful text describing options using
865 the @code{--help} argument and error handling are taken care of when
866 using this approach. Other @code{GNUNET\_GETOPT\_}-functions can be used
867 to obtain integer value options, increment counters, etc. You can
868 even write custom option parsers for special circumstances not covered
869 by the available handlers. To check if an argument was specified by the
870 user you initialize the variable with a specific value (e.g. NULL for
871 a string and GNUNET\_SYSERR for a integer) and check after parsing
872 happened if the values were modified.
874 Inside the @code{run} method, the program would perform the
875 application-specific logic, which typically involves initializing and
876 using some client library to interact with the service. The client
877 library is supposed to implement the IPC whereas the service provides
878 more persistent P2P functions.
880 Exercise: Add a few command-line options and print them inside
881 of @code{run}. What happens if the user gives invalid arguments?
883 @node Writing a Client Library
884 @subsection Writing a Client Library
886 The first and most important step in writing a client library is to
887 decide on an API for the library. Typical API calls include
888 connecting to the service, performing application-specific requests
889 and cleaning up. Many examples for such service APIs can be found
890 in the @file{gnunet/src/include/gnunet\_*\_service.h} files.
892 Then, a client-service protocol needs to be designed. This typically
893 involves defining various message formats in a header that will be
894 included by both the service and the client library (but is otherwise
895 not shared and hence located within the service's directory and not
896 installed by @command{make install}). Each message must start with a
897 @code{struct GNUNET\_MessageHeader} and must be shorter than 64k. By
898 convention, all fields in IPC (and P2P) messages must be in big-endian
899 format (and thus should be read using @code{ntohl} and similar
900 functions and written using @code{htonl} and similar functions).
901 Unique message types must be defined for each message struct in the
902 @file{gnunet\_protocols.h} header (or an extension-specific include
906 * Connecting to the Service::
908 * Receiving Replies from the Service::
911 @node Connecting to the Service
912 @subsubsection Connecting to the Service
914 Before a client library can implement the application-specific protocol
915 with the service, a connection must be created:
918 @verbatiminclude tutorial-examples/003.c
922 As a result a @code{GNUNET\_MQ\_Handle} is returned
923 which can to used henceforth to transmit messages to the service.
924 The complete MQ API can be found in @file{gnunet\_mq\_lib.h}.
925 The @code{hanlders} array in the example above is incomplete.
926 Here is where you will define which messages you expect to
927 receive from the service, and which functions handle them.
928 The @code{error\_cb} is a function that is to be called whenever
929 there are errors communicating with the service.
931 @node Sending messages
932 @subsubsection Sending messages
934 In GNUnet, messages are always sent beginning with a
935 @code{struct GNUNET\_MessageHeader} in big endian format.
936 This header defines the size and the type of the
937 message, the payload follows after this header.
940 @verbatiminclude tutorial-examples/004.c
944 Existing message types are defined in @file{gnunet\_protocols.h}.
945 A common way to create a message is with an envelope:
948 @verbatiminclude tutorial-examples/005.c
952 Exercise: Define a message struct that includes a 32-bit
953 unsigned integer in addition to the standard GNUnet MessageHeader.
954 Add a C struct and define a fresh protocol number for your message.
955 Protocol numbers in gnunet-ext are defined
956 in @file{gnunet-ext/src/include/gnunet_protocols_ext.h}
958 Exercise: Find out how you can determine the number of messages
961 Exercise: Find out how you can determine when a message you
962 have queued was actually transmitted.
964 Exercise: Define a helper function to transmit a 32-bit
965 unsigned integer (as payload) to a service using some given client
968 @node Receiving Replies from the Service
969 @subsubsection Receiving Replies from the Service
971 Clients can receive messages from the service using the handlers
972 specified in the @code{handlers} array we specified when connecting
973 to the service. Entries in the the array are usually created using
974 one of two macros, depending on whether the message is fixed size
975 or variable size. Variable size messages are managed using two
976 callbacks, one to check that the message is well-formed, the other
977 to actually process the message. Fixed size messages are fully
978 checked by the MQ-logic, and thus only need to provide the handler
979 to process the message. Note that the prefixes @code{check\_}
980 and @code{handle\_} are mandatory.
983 @verbatiminclude tutorial-examples/006.c
987 Exercise: Expand your helper function to receive a response message
988 (for example, containing just the @code{struct GNUnet MessageHeader}
989 without any payload). Upon receiving the service's response, you
990 should call a callback provided to your helper function's API.
992 Exercise: Figure out where you can pass values to the
993 closures (@code{cls}).
995 @node Writing a user interface
996 @subsection Writing a user interface
998 Given a client library, all it takes to access a service now is to
999 combine calls to the client library with parsing command-line
1002 Exercise: Call your client API from your @code{run()} method in your
1003 client application to send a request to the service. For example,
1004 send a 32-bit integer value based on a number given at the
1005 command-line to the service.
1007 @node Writing a Service
1008 @section Writing a Service
1010 Before you can test the client you've written so far, you'll
1011 need to also implement the corresponding service.
1015 * Starting a Service::
1018 @node Code Placement
1019 @subsection Code Placement
1021 New services are placed in their own subdirectory under
1022 @file{gnunet/src}. This subdirectory should contain the API
1023 implementation file @file{SERVICE\_api.c}, the description of
1024 the client-service protocol @file{SERVICE.h} and P2P protocol
1025 @file{SERVICE\_protocol.h}, the implementation of the service itself
1026 @file{gnunet-service-SERVICE.h} and several files for tests,
1027 including test code and configuration files.
1029 @node Starting a Service
1030 @subsection Starting a Service
1032 The key API definition for creating a service is the
1033 @code{GNUNET\_SERVICE\_MAIN} macro:
1036 @verbatiminclude tutorial-examples/007.c
1040 In addition to the service name and flags, the macro takes three
1041 functions, typically called @code{run}, @code{client\_connect\_cb} and
1042 @code{client\_disconnect\_cb} as well as an array of message handlers
1043 that will be called for incoming messages from clients.
1045 A minimal version of the three central service funtions would look
1049 @verbatiminclude tutorial-examples/008.c
1053 Exercise: Write a stub service that processes no messages at all
1054 in your code. Create a default configuration for it, integrate it
1055 with the build system and start the service from
1056 @command{gnunet-service-arm} using @command{gnunet-arm -i NAME}.
1058 Exercise: Figure out how to set the closure (@code{cls}) for handlers
1061 Exercise: Figure out how to send messages from the service back to the
1064 Each handler function in the service @b{must} eventually (possibly in some
1065 asynchronous continuation) call
1066 @code{GNUNET\_SERVICE\_client\_continue()}. Only after this call
1067 additional messages from the same client may
1068 be processed. This way, the service can throttle processing messages
1069 from the same client.
1071 Exercise: Change the service to ``handle'' the message from your
1072 client (for now, by printing a message). What happens if you
1073 forget to call @code{GNUNET\_SERVICE\_client\_continue()}?
1075 @node Interacting directly with other Peers using the CORE Service
1076 @section Interacting directly with other Peers using the CORE Service
1078 FIXME: This section still needs to be updated to the lastest API!
1080 One of the most important services in GNUnet is the @code{CORE} service
1081 managing connections between peers and handling encryption between peers.
1083 One of the first things any service that extends the P2P protocol
1084 typically does is connect to the @code{CORE} service using:
1087 @verbatiminclude tutorial-examples/009.c
1091 * New P2P connections::
1092 * Receiving P2P Messages::
1093 * Sending P2P Messages::
1094 * End of P2P connections::
1097 @node New P2P connections
1098 @subsection New P2P connections
1100 Before any traffic with a different peer can be exchanged, the peer must
1101 be known to the service. This is notified by the @code{CORE}
1102 @code{connects} callback, which communicates the identity of the new
1103 peer to the service:
1106 @verbatiminclude tutorial-examples/010.c
1110 Note that whatever you return from @code{connects} is given as the
1111 @code{cls} argument to the message handlers for messages from
1112 the respective peer.
1114 Exercise: Create a service that connects to the @code{CORE}. Then
1115 start (and connect) two peers and print a message once your connect
1116 callback is invoked.
1118 @node Receiving P2P Messages
1119 @subsection Receiving P2P Messages
1121 To receive messages from @code{CORE}, you pass the desired
1122 @code{handlers} to the @code{GNUNET\_CORE\_connect()} function,
1123 just as we showed for services.
1125 It is your responsibility to process messages fast enough or
1126 to implement flow control. If an application does not process
1127 CORE messages fast enough, CORE will randomly drop messages
1128 to not keep a very long queue in memory.
1130 Exercise: Start one peer with a new service that has a message
1131 handler and start a second peer that only has your ``old'' service
1132 without message handlers. Which ``connect'' handlers are invoked when
1133 the two peers are connected? Why?
1135 @node Sending P2P Messages
1136 @subsection Sending P2P Messages
1138 You can transmit messages to other peers using the @code{mq} you were
1139 given during the @code{connect} callback. Note that the @code{mq}
1140 automatically is released upon @code{disconnect} and that you must
1141 not use it afterwards.
1143 It is your responsibility to not over-fill the message queue, GNUnet
1144 will send the messages roughly in the order given as soon as possible.
1146 Exercise: Write a service that upon connect sends messages as
1147 fast as possible to the other peer (the other peer should run a
1148 service that ``processes'' those messages). How fast is the
1149 transmission? Count using the STATISTICS service on both ends. Are
1150 messages lost? How can you transmit messages faster? What happens if
1151 you stop the peer that is receiving your messages?
1153 @node End of P2P connections
1154 @subsection End of P2P connections
1156 If a message handler returns @code{GNUNET\_SYSERR}, the remote
1157 peer shuts down or there is an unrecoverable network
1158 disconnection, CORE notifies the service that the peer disconnected.
1159 After this notification no more messages will be received from the
1160 peer and the service is no longer allowed to send messages to the peer.
1161 The disconnect callback looks like the following:
1164 @verbatiminclude tutorial-examples/011.c
1168 Exercise: Fix your service to handle peer disconnects.
1170 @node Storing peer-specific data using the PEERSTORE service
1171 @section Storing peer-specific data using the PEERSTORE service
1173 GNUnet's PEERSTORE service offers a persistorage for arbitrary
1174 peer-specific data. Other GNUnet services can use the PEERSTORE
1175 to store, retrieve and monitor data records. Each data record
1176 stored with PEERSTORE contains the following fields:
1179 @item subsystem: Name of the subsystem responsible for the record.
1180 @item peerid: Identity of the peer this record is related to.
1181 @item key: a key string identifying the record.
1182 @item value: binary record value.
1183 @item expiry: record expiry date.
1186 The first step is to start a connection to the PEERSTORE service:
1188 @verbatiminclude tutorial-examples/012.c
1191 The service handle @code{peerstore_handle} will be needed for
1192 all subsequent PEERSTORE operations.
1196 * Retrieving records::
1197 * Monitoring records::
1198 * Disconnecting from PEERSTORE::
1201 @node Storing records
1202 @subsection Storing records
1204 To store a new record, use the following function:
1207 @verbatiminclude tutorial-examples/013.c
1211 The @code{options} parameter can either be
1212 @code{GNUNET_PEERSTORE_STOREOPTION_MULTIPLE} which means that multiple
1213 values can be stored under the same key combination
1214 (subsystem, peerid, key), or @code{GNUNET_PEERSTORE_STOREOPTION_REPLACE}
1215 which means that PEERSTORE will replace any existing values under the
1216 given key combination (subsystem, peerid, key) with the new given value.
1218 The continuation function @code{cont} will be called after the store
1219 request is successfully sent to the PEERSTORE service. This does not
1220 guarantee that the record is successfully stored, only that it was
1221 received by the service.
1223 The @code{GNUNET_PEERSTORE_store} function returns a handle to the store
1224 operation. This handle can be used to cancel the store operation only
1225 before the continuation function is called:
1228 @verbatiminclude tutorial-examples/013.1.c
1231 @node Retrieving records
1232 @subsection Retrieving records
1234 To retrieve stored records, use the following function:
1237 @verbatiminclude tutorial-examples/014.c
1241 The values of @code{peer} and @code{key} can be @code{NULL}. This
1242 allows the iteration over values stored under any of the following
1247 @item (subsystem, peerid)
1248 @item (subsystem, key)
1249 @item (subsystem, peerid, key)
1252 The @code{callback} function will be called once with each retrieved
1253 record and once more with a @code{NULL} record to signal the end of
1256 The @code{GNUNET_PEERSTORE_iterate} function returns a handle to the
1257 iterate operation. This handle can be used to cancel the iterate
1258 operation only before the callback function is called with a
1261 @node Monitoring records
1262 @subsection Monitoring records
1264 PEERSTORE offers the functionality of monitoring for new records
1265 stored under a specific key combination (subsystem, peerid, key).
1266 To start the monitoring, use the following function:
1269 @verbatiminclude tutorial-examples/015.c
1273 Whenever a new record is stored under the given key combination,
1274 the @code{callback} function will be called with this new
1275 record. This will continue until the connection to the PEERSTORE
1276 service is broken or the watch operation is canceled:
1279 @verbatiminclude tutorial-examples/016.c
1282 @node Disconnecting from PEERSTORE
1283 @subsection Disconnecting from PEERSTORE
1285 When the connection to the PEERSTORE service is no longer needed,
1286 disconnect using the following function:
1289 @verbatiminclude tutorial-examples/017.c
1293 If the @code{sync_first} flag is set to @code{GNUNET_YES},
1294 the API will delay the disconnection until all store requests
1295 are received by the PEERSTORE service. Otherwise, it will
1296 disconnect immediately.
1299 @section Using the DHT
1301 The DHT allows to store data so other peers in the P2P network can
1302 access it and retrieve data stored by any peers in the network.
1303 This section will explain how to use the DHT. Of course, the first
1304 thing to do is to connect to the DHT service:
1307 @verbatiminclude tutorial-examples/018.c
1311 The second parameter indicates how many requests in parallel to expect.
1312 It is not a hard limit, but a good approximation will make the DHT more
1316 * Storing data in the DHT::
1317 * Obtaining data from the DHT::
1318 * Implementing a block plugin::
1319 * Monitoring the DHT::
1322 @node Storing data in the DHT
1323 @subsection Storing data in the DHT
1324 Since the DHT is a dynamic environment (peers join and leave frequently)
1325 the data that we put in the DHT does not stay there indefinitely. It is
1326 important to ``refresh'' the data periodically by simply storing it
1327 again, in order to make sure other peers can access it.
1329 The put API call offers a callback to signal that the PUT request has been
1330 sent. This does not guarantee that the data is accessible to others peers,
1331 or even that is has been stored, only that the service has requested to
1332 a neighboring peer the retransmission of the PUT request towards its final
1333 destination. Currently there is no feedback about whether or not the data
1334 has been sucessfully stored or where it has been stored. In order to
1335 improve the availablilty of the data and to compensate for possible
1336 errors, peers leaving and other unfavorable events, just make several
1340 @verbatiminclude tutorial-examples/019.c
1344 Exercise: Store a value in the DHT periodically to make sure it
1345 is available over time. You might consider using the function
1346 @code{GNUNET\_SCHEDULER\_add\_delayed} and call
1347 @code{GNUNET\_DHT\_put} from inside a helper function.
1349 @node Obtaining data from the DHT
1350 @subsection Obtaining data from the DHT
1352 As we saw in the previous example, the DHT works in an asynchronous mode.
1353 Each request to the DHT is executed ``in the background'' and the API
1354 calls return immediately. In order to receive results from the DHT, the
1355 API provides a callback. Once started, the request runs in the service,
1356 the service will try to get as many results as possible (filtering out
1357 duplicates) until the timeout expires or we explicitly stop the request.
1358 It is possible to give a ``forever'' timeout with
1359 @code{GNUNET\_TIME\_UNIT\_FOREVER\_REL}.
1361 If we give a route option @code{GNUNET\_DHT\_RO\_RECORD\_ROUTE}
1362 the callback will get a list of all the peers the data has travelled,
1363 both on the PUT path and on the GET path.
1366 @verbatiminclude tutorial-examples/020.c
1370 Exercise: Store a value in the DHT and after a while retrieve it.
1371 Show the IDs of all the peers the requests have gone through.
1372 In order to convert a peer ID to a string, use the function
1373 @code{GNUNET\_i2s}. Pay attention to the route option parameters
1376 @node Implementing a block plugin
1377 @subsection Implementing a block plugin
1379 In order to store data in the DHT, it is necessary to provide a block
1380 plugin. The DHT uses the block plugin to ensure that only well-formed
1381 requests and replies are transmitted over the network.
1383 The block plugin should be put in a file @file{plugin\_block\_SERVICE.c}
1384 in the service's respective directory. The
1385 mandatory functions that need to be implemented for a block plugin are
1386 described in the following sections.
1389 * Validating requests and replies::
1390 * Deriving a key from a reply::
1391 * Initialization of the plugin::
1392 * Shutdown of the plugin::
1393 * Integration of the plugin with the build system::
1396 @node Validating requests and replies
1397 @subsubsection Validating requests and replies
1399 The evaluate function should validate a reply or a request. It returns
1400 a @code{GNUNET\_BLOCK\_EvaluationResult}, which is an enumeration. All
1401 possible answers are in @file{gnunet\_block\_lib.h}. The function will
1402 be called with a @code{reply\_block} argument of @code{NULL} for
1403 requests. Note that depending on how @code{evaluate} is called, only
1404 some of the possible return values are valid. The specific meaning of
1405 the @code{xquery} argument is application-specific. Applications that
1406 do not use an extended query should check that the @code{xquery\_size}
1407 is zero. The block group is typically used to filter duplicate
1411 @verbatiminclude tutorial-examples/021.c
1415 Note that it is mandatory to detect duplicate replies in this function
1416 and return the respective status code. Duplicate detection is
1417 typically done using the Bloom filter block group provided by
1418 @file{libgnunetblockgroup.so}. Failure to do so may cause replies to
1419 circle in the network.
1421 @node Deriving a key from a reply
1422 @subsubsection Deriving a key from a reply
1424 The DHT can operate more efficiently if it is possible to derive a key
1425 from the value of the corresponding block. The @code{get\_key}
1426 function is used to obtain the key of a block --- for example, by
1427 means of hashing. If deriving the key is not possible, the function
1428 should simply return @code{GNUNET\_SYSERR} (the DHT will still work
1429 just fine with such blocks).
1432 @verbatiminclude tutorial-examples/022.c
1435 @node Initialization of the plugin
1436 @subsubsection Initialization of the plugin
1438 The plugin is realized as a shared C library. The library must export
1439 an initialization function which should initialize the plugin. The
1440 initialization function specifies what block types the plugin cares
1441 about and returns a struct with the functions that are to be used for
1442 validation and obtaining keys (the ones just defined above).
1445 @verbatiminclude tutorial-examples/023.c
1448 @node Shutdown of the plugin
1449 @subsubsection Shutdown of the plugin
1451 Following GNUnet's general plugin API concept, the plugin must
1452 export a second function for cleaning up. It usually does very
1456 @verbatiminclude tutorial-examples/024.c
1459 @node Integration of the plugin with the build system
1460 @subsubsection Integration of the plugin with the build system
1462 In order to compile the plugin, the @file{Makefile.am} file for the
1463 service SERVICE should contain a rule similar to this:
1464 @c Actually this is a Makefile not C. But the whole structure of examples
1465 @c must be improved.
1468 @verbatiminclude tutorial-examples/025.c
1472 Exercise: Write a block plugin that accepts all queries
1473 and all replies but prints information about queries and replies
1474 when the respective validation hooks are called.
1476 @node Monitoring the DHT
1477 @subsection Monitoring the DHT
1479 It is possible to monitor the functioning of the local
1480 DHT service. When monitoring the DHT, the service will
1481 alert the monitoring program of any events, both started
1482 locally or received for routing from another peer.
1483 The are three different types of events possible: a
1484 GET request, a PUT request or a response (a reply to a GET).
1486 Since the different events have different associated data,
1487 the API gets 3 different callbacks (one for each message type)
1488 and optional type and key parameters, to allow for filtering of
1489 messages. When an event happens, the appropiate callback is
1490 called with all the information about the event.
1493 @verbatiminclude tutorial-examples/026.c
1496 @node Debugging with gnunet-arm
1497 @section Debugging with gnunet-arm
1499 Even if services are managed by @command{gnunet-arm}, you can
1500 start them with @command{gdb} or @command{valgrind}. For
1501 example, you could add the following lines to your
1502 configuration file to start the DHT service in a @command{gdb}
1503 session in a fresh @command{xterm}:
1507 PREFIX=xterm -e gdb --args
1511 Alternatively, you can stop a service that was started via
1512 ARM and run it manually:
1516 $ gdb --args gnunet-service-dht -L DEBUG
1517 $ valgrind gnunet-service-dht -L DEBUG
1521 Assuming other services are well-written, they will automatically
1522 re-integrate the restarted service with the peer.
1524 GNUnet provides a powerful logging mechanism providing log
1525 levels @code{ERROR}, @code{WARNING}, @code{INFO} and @code{DEBUG}.
1526 The current log level is configured using the @code{$GNUNET_FORCE_LOG}
1527 environmental variable. The @code{DEBUG} level is only available if
1528 @command{--enable-logging=verbose} was used when running
1529 @command{configure}. More details about logging can be found under
1530 @uref{https://gnunet.org/logging}.
1532 You should also probably enable the creation of core files, by setting
1533 @code{ulimit}, and echo'ing @code{1} into
1534 @file{/proc/sys/kernel/core\_uses\_pid}. Then you can investigate the
1535 core dumps with @command{gdb}, which is often the fastest method to
1538 Exercise: Add a memory leak to your service and obtain a trace
1539 pointing to the leak using @command{valgrind} while running the service
1540 from @command{gnunet-service-arm}.