add response headers, add replace api to namestore

[oweals/gnunet.git] / README

                       Welcome to GNUnet

ToC
===

* ToC
* What is GNUnet?
* Dependencies
  o direct dependencies
  o test suite dependencies
  o optional dependencies
  o autotools
* Notes on setuid
* Scope of Operating System support
* How to install
  o binary packages
  o Building GNUnet from source
  o Notes on compiling from Git
* Configuration
* Usage
* Hacking GNUnet
* Running HTTP on port 80 and HTTPS on port 443
* Further Reading
* Stay tuned

What is GNUnet?
===============

GNUnet is peer-to-peer framework providing a network abstractions and
applications focusing on security and privacy.  So far, we have
created applications for anonymous file-sharing, decentralized naming
and identity management, decentralized and confidential telephony and
tunneling IP traffic over GNUnet.  GNUnet is currently developed by a
worldwide group of independent free software developers.  GNUnet is a
GNU package (http://www.gnu.org/).

This is an ALPHA release.  There are known and significant bugs as
well as many missing features in this release.

GNUnet is free software released under the GNU Affero General Public
License (v3 or later). For details see the COPYING file in this
directory.  If you fork this software, you MUST adjust GNUNET_AGPL_URL
in src/include/gnunet_util_lib.h to point to the source code of your
fork!

Additional documentation about GNUnet can be found at
https://gnunet.org/ and in the 'doc/' folder.
Online documentation is provided at
'https://docs.gnunet.org' and 'https://tutorial.gnunet.org'.


Dependencies:
=============

The dependencies for building GNUnet will require around 0.74 GiB
diskspace. GNUnet itself will require 8 - 9.2 MiB depending on
configuration.

These are the direct dependencies for running GNUnet:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- Bash                              (for some scripts)
- gettext
- gnutls             >= 3.2.12      (highly recommended a gnutls
                                     linked against libunbound)
- A curl build against gnutls, or gnurl:
  * libgnurl         >= 7.35.0      (recommended, available from
                                     https://gnunet.org/en/gnurl.html)
  or
  * libcurl          >= 7.35.0      (alternative to libgnurl)
- libgcrypt          >= 1.6
- libunistring       >= 0.9.2
- libidn:
  * libidn2 (prefered)
  or
  * libidn           >= 1.0
- libmicrohttpd      >= 0.9.63      (strongly recommended for
                                     a wide range of features)
- makeinfo           >= 4.8
- make[*3]
- nss                               (certutil binary, for
                                     gnunet-gns-proxy-setup-ca)
- openssl            >= 1.0         (binary, used to generate
                                     X.509 certificate
                                     for gnunet-gns-proxy-setup-ca)
- pkgconf or pkg-config
- A Posix shell                       (for some scripts)
- Texinfo            >= 5.2         [*1]
- libltdl            >= 2.2         (part of GNU libtool)
- 1 or more databases:
  * sqlite           >= 3.8         (default database, required)
  and/or
  * mysql            >= 5.1         (alternative to sqlite)
  and/or
  * postgres         >= 9.5         (alternative to sqlite)
- which                             (contrib/apparmor(?), gnunet-bugreport,
                                     and possibly more)
- zlib
- argon2             >= 20190702    (for proof-of-work calculations in
                                     revocation)
- libsodium          >= 1.0.11      (for elliptic curve cryptography)

These are the dependencies for GNUnet's testsuite:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- Bash                              (for some tests[*4])
- A Posix Shell                     (for some tests)
- python             >= 3.4         (3.4 and higher technically supported,
                                     at least python 3.7 tested to work)
- base tools
  - mostly:
    - bc,
    - curl,
    - sed,
    - awk,
    - which


These are the optional dependencies:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- awk                               (for linting tests)
- Bash                              (for Docker and Vagrant)
- bluez                             (for bluetooth support)
- grof                              (for linting of man pages)
- libextractor       >= 0.6.1       (highly recommended[*5])
- libjansson
- libopus            >= 1.0.1       (for conversation tool)
- libpulse           >= 2.0         (for conversation tool)
- libogg             >= 1.3.0       (for conversation tool)
- libnss                            (certtool binary (for convenient
                                     installation of GNS proxy))
- libzbar            >= 0.10        (for gnunet-qr)
- libpbc             >= 0.5.14      (for Attribute-Based Encryption and
                                     Identity Provider functionality)
- libgabe                           (for Attribute-Based Encryption and
                                     Identity Provider functionality, from
                                     https://github.com/schanzen/libgabe)
- mandoc                            (for linting of man pages, generation of
                                     html output of man pages (not part of
                                     the regular build))
- miniupnpc
- perl5                             (for some utilities)
- TeX Live           >= 2012        (for gnunet-bcd[*])
- texi2mdoc                         (for automatic mdoc generation [*2], not
                                     the texi2mdoc script distributed with
                                     autogen but the texi2mdoc C application)

Recommended autotools for compiling the Git version are:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- autoconf           >= 2.59
- automake           >= 1.11.1
- libtool            >= 2.2


[*] Mandatory for compiling the info output of the documentation,
    a limited subset ('texlive-tiny' in Guix) is enough.

[*1] The default configuration is to build the info output of the
     documentation, and therefore require texinfo.  You can pass
     '--disable-documentation' to the configure script to change this.

[*2] If you still prefer to have documentation, you can pass
     '--enable-texi2mdoc-generation' to build the mdocml ("mandoc")
     documentation (experimental stages in gnunet).
     If this proves to be reliable, we will
     include the mdocml output in the release tarballs.
     Contrary to the name, texi2mdoc does not require Texinfo,
     It is a standalone ISO C utility.

[*3] GNU make introduced the != operator in version 4.0.
     GNU make was released in october 2013, reasonable to
     be widespread by now. If this is not working out for
     you, open a bug so that we can get a more portable
     fix in.

[*4] We are commited  to portable tools and solutions
     where possible. New scripts should be Posix sh
     compatible, current and older scripts are
     in the process of being rewritten to comply
     with this requirement.

[*5] While libextractor ("LE") is optional, it is recommended to
     build gnunet against it. If you install it later,
     you won't benefit from libextractor.
     If you are a distributor, we recommend to split
     LE into basis + plugins rather than making LE
     an option as an afterthought by the user.
     LE itself is very small, but its dependency chain
     on first, second, third etc level can be big.
     There is a small effect on privacy if your LE build
     differs from one which includes all
     plugins (plugins are build as shared objects):
     if users publish a directory with a mixture of file
     types (for example mpeg, jpeg, png, gif) the
     configuration of LE could leak which plugins are
     installed for which filetypes are not providing
     more details.
     However, this leak is just a minor concern.

Notes on setuid
===============

For a correct functionality depending on the host OS, you need
to run the equivalent of these steps after installation.
Replace $(DESTDIR)$(libexecdir) with the appropriate paths,
for example /usr/local/lib/gnunet/libexec/. Note that this
obviously must be run as priviledged user.

chown root:root $(DESTDIR)$(libexecdir)/gnunet-helper-vpn
chmod u+s $(DESTDIR)$(libexecdir)/gnunet-helper-vpn
chown root:root $(DESTDIR)$(libexecdir)/gnunet-helper-transport-wlan
chmod u+s $(DESTDIR)$(libexecdir)/gnunet-helper-transport-wlan
chown root:root $(DESTDIR)$(libexecdir)/gnunet-helper-transport-bluetooth
chmod u+s $(DESTDIR)$(libexecdir)/gnunet-helper-transport-bluetooth
chown root $(DESTDIR)$(libexecdir)/gnunet-helper-dns
chgrp $(GNUNETDNS_GROUP) $(DESTDIR)$(libexecdir)/gnunet-helper-dns
chmod 4750 $(DESTDIR)$(libexecdir)/gnunet-helper-dns
chgrp $(GNUNETDNS_GROUP) $(DESTDIR)$(libexecdir)/gnunet-helper-dns
chown gnunet:$(GNUNETDNS_GROUP) $(DESTDIR)$(libexecdir)/gnunet-helper-dns
chmod 2750 $(DESTDIR)$(libexecdir)/gnunet-helper-dns
chown root:root $(DESTDIR)$(libexecdir)/gnunet-helper-exit
chmod u+s $(DESTDIR)$(libexecdir)/gnunet-helper-exit
chown root:root $(DESTDIR)$(libexecdir)/gnunet-helper-nat-server
chown root:root $(DESTDIR)$(libexecdir)/gnunet-helper-nat-client
chmod u+s $(DESTDIR)$(libexecdir)/gnunet-helper-nat-server
chmod u+s $(DESTDIR)$(libexecdir)/gnunet-helper-nat-client


Scope of Operating System support
=================================

We actively support GNUnet on a broad range of Free Software Operating
Systems.

For proprietary Operating Systems, like for example Microsoft Windows
or Apple OS X, we accept patches if they don't break anything for
other Operating Systems.
If you are implementing support for a proprietary Operating System,
you should be aware that progress in our codebase could break
functionality on your OS and cause unpredicted behavior we can
not test. However, we do not break support on Operating Systems
with malicious intent.
Regressions which do occur on these Operating Systems are 3rd
class issues and we expect users and developers of these
Operating Systems to send proposed patches to fix regressions.

For more information about our stand on some of the motivating
points here, read the 'Philosophy' Chapter of our handbook.


How to install?
===============

binary packages
~~~~~~~~~~~~~~~

We recommend to use binary packages provided by the package manager integrated
within your Operating System. GNUnet is reportedly available for at least:

ALT Linux, Archlinux, Debian, Deepin, Devuan, GNU Guix, Hyperbola,
Kali Linux, LEDE/OpenWRT, Manjaro, Nix, Parabola, Pardus, Parrot,
PureOS, Raspbian, Rosa, Trisquel, and Ubuntu.

If GNUnet is available for your Operating System and it is missing,
send us feedback so that we can add it to this list. Furthermore, if
you are interested in packaging GNUnet for your Operating System,
get in touch with us at gnunet-developers@gnu.org if you require
help with this job.

If you were using an Operating System with the apt package manager,
GNUnet could be installed as simple as:

$ apt-get install gnunet

Generic installation instructions are in the INSTALL file in this
directory.

Building GNUnet from source
~~~~~~~~~~~~~~~~~~~~~~~~~~~

IMPORTANT: You can read further notes about compilation from source in
the handbook under doc/handbook/, which includes notes about specific
requirements for operating systems aswell. If you are a package
mantainer for an Operating System we invite you to add your notes if
you feel it is necessary and can not be covered in your Operating
System's documentation.

Two prominent examples which currently lack cross-compilation
support in GNUnet (and native binaries) are MS Windows and Apple macOS.
For macOS we recommend you to do the build process via Homebrew and a
recent XCode installation. We don't recommend using GNUnet with any
recent MS Windows system as it officially spies on its users (according
to its T&C), defying some of the purposes of GNUnet.

Note that some functions of GNUnet require "root" access.  GNUnet will
install (tiny) SUID binaries for those functions is you run "make
install" as root.  If you do not, GNUnet will still work, but some
functionality will not be available (including certain forms of NAT
traversal).

GNUnet requires the GNU MP library (https://www.gnu.org/software/gmp/)
and libgcrypt (https://www.gnupg.org/).  You can specify the path to
libgcrypt by passing "--with-gcrypt=PATH" to configure.  You will also
need either sqlite (http://www.sqlite.org/), MySQL
(http://www.mysql.org/) or PostGres (http://www.postgres.org/).

If you install from source, you need to install GNU libextractor first
(download from https://www.gnu.org/software/libextractor/).  We also
recommend installing GNU libmicrohttpd (download from
https://www.gnu.org/software/libmicrohttpd/). Furthermore we recommend
libgnurl (from https://gnunet.org/en/gnurl.html).
Then you can start the actual GNUnet compilation process with:


$ export GNUNET_PREFIX=/usr/local/lib # or other directory of your choice
# addgroup gnunetdns
# adduser --system --home "/var/lib/gnunet" --group gnunet --shell /bin/sh
# ./configure --prefix=$GNUNET_PREFIX/.. --with-extractor=$LE_PREFIX
$ make

And finally install GNUnet with:

# make install

Complete the process by either adjusting one of our example service files
in 'contrib/services' or by running:

# sudo -u gnunet gnunet-arm -s


Note that you must read paragraph "Notes on setuid", which documents steps you
have to follow after the installation, as a priviledged user. We require some
binaries to be setuid. The most portable approach across all supported
platforms and targets is to let this be handled manually.
The installation will work if you do not run these steps as root, but some
components may not be installed in the perfect place or with the right
permissions and thus won't work.

This will create the users and groups needed for running GNUnet
securely and then compile and install GNUnet to $GNUNET_PREFIX/../bin/,
$GNUNET_PREFIX/ and $GNUNET_PREFIX/../share/ and start the system
with the default configuration.  It is strongly recommended that you
add a user "gnunet" to run "gnunet-arm".  You can then still run the
end-user applications as another user.

If you create a system user "gnunet", it is recommended that you edit
the configuration file slightly so that data can be stored in the
system user home directory at "/var/lib/gnunet".  Depending on what
the $HOME-directory of your "gnunet" user is, you might need to set
the SERVICEHOME option in section "[PATHS]" to "/var/lib/gnunet" to
do this.  Depending on your personal preferences, you may also want to
use "/etc/gnunet.conf" for the location of the configuration file in
this case (instead of ~gnunet/.config/gnunet.conf").  In this case,
you need to start GNUnet using "gnunet-arm -s -c /etc/gnunet.conf" or
set "XDG_CONFIG_HOME=/etc/".

You can avoid running 'make install' as root if you have extensive sudo rights
(can run "chmod +s" and "chown" via 'sudo').  If you run 'make install' as a
normal user without sudo rights (or the configure option), certain binaries
that require additional privileges will not be installed properly (and
autonomous NAT traversal, WLAN, DNS/GNS and the VPN will then not work).

If you run 'configure' and 'make install' as root, GNUnet's build system will
install "libnss_gns*" libraries to "/lib/" regardless (!) of the
$GNUNET_PREFIX you might have specified, as those libraries must be in
"/lib/".  If you are packaging GNUnet for binary distribution, this may cause
your packaging script to miss those plugins, so you might need to do some
additional manual work to include those libraries in your binary package(s).
Similarly, if you want to use the GNUnet Name System and did NOT run
GNUnet's 'make install' process with priviledged rights, the libraries will be
installed to "$GNUNET_PREFIX" and you will have to move them to "/lib/"
manually.

Notes on compiling from Git
~~~~~~~~~~~~~~~~~~~~~~~~~~~

Finally, if you are compiling the code from git, you have to
run "sh ./bootstrap" before running "./configure".  If you receive an error during
the running of "sh ./bootstrap" that looks like "macro `AM_PATH_GTK'
not found in library", you may need to run aclocal by hand with the -I
option, pointing to your aclocal m4 macros, i.e.

$ aclocal -I /usr/local/share/aclocal


Configuration
=============

Note that additional, per-user configuration files can be created by
each user.  However, this is usually not necessary as there are few
per-user options that normal users would want to modify.  The defaults
that are shipped with the installation are usually just fine.

The gnunet-setup tool is particularly useful to generate the master
configuration for the peer.  gnunet-setup can be used to configure and
test (!) the network settings, choose which applications should be run
and configure databases.  Other options you might want to control
include system limitations (such as disk space consumption, bandwidth,
etc).  The resulting configuration files are human-readable and can
theoretically be created or edited by hand.

gnunet-setup is a separate download and requires somewhat recent
versions of GTK+ and Glade. You can also create the configuration file
by hand, but this is not recommended.  For more general information
about the GNU build process read the INSTALL file.

GNUnet uses two types of configuration files, one that specifies the
system-wide defaults (typically located in
$GNUNET_PREFIX/../share/gnunet/config.d/) and a second one that overrides
default values with user-specific preferences.  The user-specific
configuration file should be located in "~/.config/gnunet.conf" or its
location can be specified by giving the "-c" option to the respective
GNUnet application.

For more information about the configuration (as well as usage) refer
to the 'GNUnet User Handbook' chapter of the documentation, included
in this software distribution.


Usage
=====

For detailed usage notes, instructions and examples, refer to the
included 'GNUnet Handbook'.

First, you must obtain an initial list of GNUnet hosts.  Knowing a
single peer is sufficient since after that GNUnet propagates
information about other peers.  Note that the default configuration
contains URLs from where GNUnet downloads an initial hostlist
whenever it is started.  If you want to create an alternative URL for
others to use, the file can be generated on any machine running
GNUnet by periodically executing

$ cat $SERVICEHOME/data/hosts/* > the_file

and offering 'the_file' via your web server.  Alternatively, you can
run the build-in web server by adding '-p' to the OPTIONS value
in the "hostlist" section of gnunet.conf and opening the respective
HTTPPORT to the public.

If the solution with the hostlist URL is not feasible for your
situation, you can also add hosts manually.  Simply copy the hostkeys
to "$SERVICEHOME/data/hosts/" (where $SERVICEHOME is the directory
specified in the gnunet.conf configuration file).  You can also use
"gnunet-peerinfo -g" to GET a URI for a peer and "gnunet-peerinfo -p
URI" to add a URI from another peer.  Finally, GNUnet peers that use
UDP or WLAN will discover each other automatically (if they are in the
vicinity of each other) using broadcasts (IPv4/WLAN) or multicasts
(IPv6).

The local node is started using "gnunet-arm -s".  We recommend to run
GNUnet 24/7 if you want to maximize your anonymity, as this makes
partitioning attacks harder.

Once your peer is running, you should then be able to access GNUnet
using the shell:

$ gnunet-search KEYWORD

This will display a list of results to the console.  You can abort
the command using "CTRL-C".  Then use

$ gnunet-download -o FILENAME GNUNET_URI

to retrieve a file.  The GNUNET_URI is printed by gnunet-search
together with a description.  To publish files on GNUnet, use the
"gnunet-publish" command.


The GTK user interface is shipped separately.
After installing gnunet-gtk, you can invoke the setup tool and
the file-sharing GUI with:

$ gnunet-setup
$ gnunet-fs-gtk

For further documentation, see our webpage or the 'GNUnet User Handbook',
included in this software distribution.


Hacking GNUnet
==============

Contributions are welcome. Please submit bugs you find to
https://bugs.gnunet.org/ or our bugs mailinglist.
Please make sure to run the script "contrib/scripts/gnunet-bugreport"
and include the output with your bug reports.  More about how to
report bugs can be found in the GNUnet FAQ on the webpage.  Submit
patches via E-Mail to gnunet-developers@gnu.org, formated with
`git format-patch`.

In order to run the unit tests by hand (instead of using "make check"),
you need to set the environment variable "GNUNET_PREFIX" to the
directory where GNUnet's libraries are installed.
Before running any testcases, you must complete the installation.

Quick summary:

$ ./configure --prefix=$SOMEWHERE
$ make
$ make install
$ export $GNUNET_PREFIX=$SOMEWHERE
$ make check

Some of the testcases require python >= 3.4, and the python module
"pexpect" to be installed.
If any testcases fail to pass on your system, run
"contrib/scripts/gnunet-bugreport" (in the repository) or "gnunet-bugreport"
when you already have GNUnet installed and report its output together with
information about the failing testcase(s) to the Mantis bugtracking
system at https://bugs.gnunet.org/.


Running HTTP on port 80 and HTTPS on port 443
=============================================

In order to hide GNUnet's HTTP/HTTPS traffic perfectly, you might
consider running GNUnet's HTTP/HTTPS transport on port 80/443.
However, we do not recommend running GNUnet as root.  Instead, forward
port 80 to say 1080 with this command (as root, in your startup
scripts):

# iptables -t nat -A PREROUTING -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 1080

or for HTTPS

# iptables -t nat -A PREROUTING -p tcp -m tcp --dport 443 -j REDIRECT --to-ports 4433

Then set in the HTTP section of gnunet.conf the "ADVERTISED_PORT" to
"80" and "PORT" to 1080 and similarly in the HTTPS section the
"ADVERTISED_PORT" to "443" and "PORT" to 4433.

You can do the same trick for the TCP and UDP transports if you want
to map them to a priviledged port (from the point of view of the
network).  However, we are not aware of this providing any advantages
at this point.

If you are already running an HTTP or HTTPS server on port 80 (or 443),
you may be able to configure it as a "ReverseProxy".  Here, you tell
GNUnet that the externally visible URI is some sub-page on your website,
and GNUnet can then tunnel its traffic via your existing HTTP server.
This is particularly powerful if your existing server uses HTTPS, as
it makes it harder for an adversary to distinguish normal traffic to
your server from GNUnet traffic.  Finally, even if you just use HTTP,
you might benefit (!) from ISP's traffic shaping as opposed to being
throttled by ISPs that dislike P2P.  Details for configuring the
reverse proxy are documented on our website.


Further Reading
===============

* Documentation

  An HTML version of the GNUnet manual is deployed at

    https://docs.gnunet.org

  which currently displays just GNUnet documentation. In the future
  we will add more reading material.

* Academia / papers

  In almost 20 years various people in our community have written and
  collected a good number of papers which have been implemented in
  GNUnet or projects around GNUnet.
  There are currently 2 ways to get them:

  * Using git (NOTE: 1.1 GiB as of 2019-03-09):
    git clone https://git.gnunet.org/bibliography.git
  * Using the webbrowser:
    https://bib.gnunet.org/


Stay tuned
==========

* https://gnunet.org/
* https://bugs.gnunet.org
* https://git.gnunet.org
* http://www.gnu.org/software/gnunet/
* http://mail.gnu.org/mailman/listinfo/gnunet-developers
* http://mail.gnu.org/mailman/listinfo/help-gnunet
* http://mail.gnu.org/mailman/listinfo/info-gnunet
* http://mail.gnu.org/mailman/listinfo/gnunet-svn