From dfc745e0e977a20de99a88958f9baae6c850cd53 Mon Sep 17 00:00:00 2001 From: Christian Grothoff Date: Sun, 18 Jul 2010 20:18:17 +0000 Subject: [PATCH] update docs --- RATIONALE | 45 +++++++++++---------- README | 100 ++++++++++++++++++++++------------------------ TODO | 41 ++++++++++++------- contrib/report.sh | 21 +++++----- mkinstalldirs | 5 ++- 5 files changed, 111 insertions(+), 101 deletions(-) diff --git a/RATIONALE b/RATIONALE index b37e1a2fe..d68e244c6 100644 --- a/RATIONALE +++ b/RATIONALE @@ -1,12 +1,13 @@ -This document is a summary of why we're moving to GNUnet NG and what -this major redesign tries to address. +This document is a summary of the changes made to GNUnet for version +0.9.x (from 0.8.x) and what this major redesign tries to address. First of all, the redesign does not (intentionally) change anything fundamental about the application-level protocols or how files are encoded and shared. However, it is not protocol-compatible due to other changes that do not relate to the essence of the application -protocols. - +protocols. This choice was made since productive development and +readable code were considered more important than compatibility at +this point. The redesign tries to address the following major problem groups describing isssues that apply more or less to all GNUnet versions @@ -26,10 +27,10 @@ PROBLEM GROUP 1 (scalability): mutexes and almost 1000 lines of lock/unlock operations. It is challenging for even good programmers to program or maintain good multi-threaded code with this complexity. - The excessive locking essentially prevents GNUnet from + The excessive locking essentially prevents GNUnet 0.8 from actually doing much in parallel on multicores. * Despite efforts like Freeway, it was virtually - impossible to contribute code to GNUnet that was not + impossible to contribute code to GNUnet 0.8 that was not writen in C/C++. * Changes to the configuration almost always required restarts of gnunetd; the existence of change-notifications does not @@ -44,11 +45,11 @@ PROBLEM GROUP 1 (scalability): days, result in really nasty and hard-to-find crashes. * structs of function pointers in service APIs were needlessly adding complexity, especially since in - most cases there was no polymorphism + most cases there was no actual polymorphism SOLUTION: * Use multiple, lously-coupled processes and one big select - loop in each (supported by a powerful library to eliminate + loop in each (supported by a powerful util library to eliminate code duplication for each process). * Eliminate all threads, manage the processes with a master-process (gnunet-arm, for automatic restart manager) @@ -65,13 +66,15 @@ SOLUTION: => Process priorities can be used to schedule the CPU better Note that we can not just use one process with a big select loop because we have blocking operations (and the - blocking is outside of our control, thanks MySQL, + blocking is outside of our control, thanks to MySQL, sqlite, gethostbyaddr, etc.). So in order to perform reasonably well, we need some construct for parallel - execution. + execution. RULE: If your service contains blocking functions, it - MUST be a process by itself. + MUST be a process by itself. If your service + is sufficiently complex, you MAY choose to make + it a separate process. * Eliminate structs with function pointers for service APIs; instead, provide a library (still ending in _service.h) API that transmits the requests nicely to the respective @@ -121,6 +124,8 @@ SOLUTION: thing given the potential for bugs. * There is no more TIME API function to do anything with 32-bit seconds +* There is now a bandwidth API to handle + non-trivial bandwidth utilization calculations PROBLEM GROUP 3 (statistics): @@ -237,18 +242,15 @@ PROBLEM GROUP 6 (FS-APIs): * FILENAME metadata is killed by ECRS/FSUI to avoid exposing HOME, but what if the user set it manually? -SOLUTION (draft, not done yet, details missing...): +SOLUTION: * Eliminate threads from FS-APIs - => Open question: how to best write the APIs to - allow integration with diverse event loops - of GUI libraries? -* Store FS-state always also on disk - => Open question: how to do this without - compromising state/scalability? +* Incrementally store FS-state always also on disk using many + small files instead of one big file * Have API to manipulate sharing tree before upload; have auto-construction modify FILENAME but allow user-modifications afterwards + PROBLEM GROUP 7 (User experience): * Searches often do not return a sufficient / significant number of results @@ -281,11 +283,12 @@ SUMMARY: - network traffic monitors (goal: eliminate) - IPC semaphores (goal: d-bus? / eliminate?) - second timers - - DNS lookup (goal: have async service; issue: still need synchronous - resolution in places, current code may not be portable) - => code shrunk from 61 files to 34, 22k LOC to 15k LOC, 470k to 330k (with symbols) * New features in util: - scheduler - service and program boot-strap code + - bandwidth and time APIs + - buffered IO API + - HKDF implementation (crypto) * Major changes in util: - more expressive server (replaces selector) + - DNS lookup replaced by async service diff --git a/README b/README index 5767598d0..905d97d3b 100644 --- a/README +++ b/README @@ -7,13 +7,13 @@ What is GNUnet? GNUnet is peer-to-peer framework focusing on security. The first and primary application for GNUnet is anonymous file-sharing. GNUnet is currently developed by a worldwide group of independent free software -developers. GNUnet is a part of the GNU project (http://www.gnu.org/). +developers. GNUnet is a GNU package (http://www.gnu.org/). This is an ALPHA release. There are known and significant bugs as well as many missing features in this release. -For a longer description of the GNUnet System see our webpages -http://www.gnu.org/software/gnunet/ and https://ng.gnunet.org/. +Additional documentation about GNUnet can be found at +https://gnunet.org/. Dependencies: @@ -45,40 +45,41 @@ How to install? The fastest way is to use a binary package if it is available for your system. For a more detailed description, read the installation -instructions on the webpage at https://ng.gnunet.org/installation. +instructions on the webpage at https://gnunet.org/installation. + +GNUnet requires the GNU MP library (http://www.gnu.org/software/gmp/) +and libgcrypt (http://www.gnupg.org/). You can specify the path to +libgcrypt by passing "--with-gcrypt=PATH" to configure. You will also +need either sqlite (http://www.sqlite.org/) or MySQL +(http://www.mysql.org/). If you install from source, you need to install GNU libextractor first (download from http://www.gnu.org/software/libextractor/). We also -recommend installing GNU libmicrohttpd. Then you can start the actual -GNUnet compilation process with: +recommend installing GNU libmicrohttpd (download from +http://www.gnu.org/software/libmicrohttpd/). Then you can start the +actual GNUnet compilation process with: $ ./configure --prefix=$HOME --with-extractor=$HOME $ make # make install -# sudo -u gnunet gnunet-setup # (note: does not yet exist!) +# sudo -u gnunet mkdir ~/.gnunet/ +# sudo -u gnunet touch ~/.gnunet/gnunet.conf # sudo -u gnunet gnunet-arm -s -This will compile and install GNUnet to ~/bin/, ~/lib/ and ~/man/. -gnunet-setup will create the configuration; this step is interactive. -You can run gnunet-setup as root for a system-wide installation or as -a particular user to create a personal installation. It is -recommended that you add a user "gnunet" to run "gnunet-arm". You can -then still run the end-user applications as another user. If you +This will compile and install GNUnet to $HOME/bin/, $HOME/lib/ and +$HOME/share/ and start the system with the default configuration. It +is recommended that you add a user "gnunet" to run "gnunet-arm". You +can then still run the end-user applications as another user. If you create a user "gnunet", it is recommended that you edit the configuration file slightly so that data can be stored in "/var/lib/gnunet"; you may also want to use "/etc/gnunet.conf" for the -configuration in this case. +location of the configuration file in this case. Note that additional, per-user configuration files (~/.gnunet/gnunet.conf) need to be created by each user (for example, by running gnunet-setup). For more general information about the GNU build process read the INSTALL file. -GNUnet requires the GNU MP library (http://www.gnu.org/software/gmp/) -and libgcrypt. You can specify the path to libgcrypt by passing -"--with-gcrypt=PATH" to configure. You will also need either sqlite -(version 3 or higher) or MySQL (version 5.0 or higher). - If you are compiling the code from subversion, you have to run ". bootstrap" before ./configure. If you receive an error during the running of ". bootstrap" that looks like "macro `AM_PATH_GTK' not @@ -91,24 +92,20 @@ $ aclocal -I /usr/local/share/aclocal Configuration ============= -// FIXME: update this section once we have gnunet-setup! GNUnet uses -two types of configuration files, one for the daemon (called -gnunetd.conf) and one for each user (gnunet.conf). You can create and -edit these configuration files with the gnunet-setup tool. The -defaults that are created the first time you run gnunet-setup are -usually ok, you may want to adjust the limitations (space consumption, -bandwidth, etc.) though. The configuration files are human-readable; -GNUnet's configuration is typically located at -"~/.gnunet/gnunet.conf". +GNUnet uses two types of configuration files, one that specifies the +system-wide defaults (typically located in /usr/share/gnunet/defaults +.conf) and a second one that overrides default values with +user-specific preferences. The user-specific configuration file +should be located in "~/.gnunet/gnunet.conf" or its location can be +specified by giving the "-c" option to the respective GNUnet +application. -You MUST create "~/.gnunet/gnunet.conf" explicitly before starting -GNUnet, and the recommended way to do this is to run gnunet-setup. -You can also create an empty configuration file, in which case -default values will be used for everything. - -If you want to use a different configuration file, pass the name of -the configuration file as an argument with the option "-c" to any -GNUnet application. +The defaults that are shipped with the installation are usually ok, +you may want to adjust the limitations (space consumption, bandwidth, +etc.) though. The configuration files are human-readable. Note that +you MUST create "~/.gnunet/gnunet.conf" explicitly before starting +GNUnet. You can either copy "defaults.conf" or simply create an empty +file. Usage @@ -126,19 +123,19 @@ $ cat $SERVICEHOME/data/hosts/* > the_file and offering 'the_file' via your web server. Alternatively, you can run the build-in web server by adding '-p' to the OPTIONS value -in the "hostlist" section of gnunet.conf. +in the "hostlist" section of gnunet.conf and opening the respective +HTTPPORT to the public. -If the solution with the URL is not feasible for your situation, you -can also add hosts manually. Simply copy the hostkeys to -"$SERVICEHOME/data/hosts/" (where $SERVICEHOME is the directory -specified in the /etc/gnunet.conf configuration file). +If the solution with the hostlist URL is not feasible for your +situation, you can also add hosts manually. Simply copy the hostkeys +to "$SERVICEHOME/data/hosts/" (where $SERVICEHOME is the directory +specified in the gnunet.conf configuration file). Now start the local node using "gnunet-arm -s". GNUnet should run 24/7 if you want to maximize your anonymity. -// FIXME: gnunet-gtk is not yet supported in 0.9.x -The GTK user interface is shipped separately from GNUnet. After -downloading and installing gnunet-gtk, you can invoke the GUI with: +The GTK user interface is shipped separately. After downloading and +installing gnunet-gtk, you can invoke the GUI with: $ gnunet-gtk @@ -202,13 +199,13 @@ Running http on port 80 ======================= In order to hide GNUnet's HTTP traffic perfectly, you might consider -running GNUnet's HTTP transport on port 80. However, you should not -run GNUnet as root. Instead, forward port 80 to say 8080 with this -command (as root, in your startup scripts): +running GNUnet's HTTP transport on port 80. However, we do not +recommend running GNUnet as root. Instead, forward port 80 to say +8080 with this command (as root, in your startup scripts): # iptables -t nat -A PREROUTING -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 8080 -Then set in the HTTP section of gnunetd.conf the "ADVERTISED-PORT" +Then set in the HTTP section of gnunet.conf the "ADVERTISED-PORT" to "80" and "PORT" to 8080. You can do the same trick for the TCP and UDP transports if you want to map them to a priviledged port (from the point of view of the network). @@ -227,12 +224,11 @@ P2P over SMTP, read the instructions at http://gnunet.org/smtp.php3 Stay tuned ========== -* http://www.gnu.org/software/gnunet/ -* http://gnunet.org/ +* https://gnunet.org/ * https://gnunet.org/bugs/ -* https://ng.gnunet.org/ +* https://gnunet.org/svn/ +* http://www.gnu.org/software/gnunet/ * http://mail.gnu.org/mailman/listinfo/gnunet-developers * http://mail.gnu.org/mailman/listinfo/help-gnunet * http://mail.gnu.org/mailman/listinfo/info-gnunet * http://mail.gnu.org/mailman/listinfo/gnunet-svn - diff --git a/TODO b/TODO index 478692ee4..012987b94 100644 --- a/TODO +++ b/TODO @@ -4,6 +4,8 @@ [On W32, we need to select after calling socket before doing connect etc.] * TRANSPORT: - HTTP backend [MW] + - UDP fails non-deterministically on niko; suspect that it uses hostname-derived, + invalid 10.0.0.x address for test-message (which creates transmission failure) [CG] - Jun 27 11:51:54 core-7670 ERROR Assertion failed at gnunet-service-core.c:3616. (transport notified us that we connected to ourselves!!!) * CORE: @@ -11,25 +13,22 @@ disconnect in a timely fashion (waits for connection timeout); need to figure a way to make it near-instant in those cases (i.e. rapid reduction in timeout on transport-level disconnect) [CG] + => may have been fixed with instant-notification of disconnect + to core on session-oriented connection hick-up; not perfect but + likely good enough until we get ATS going; still should be tested... - derived key generation [Nils] * DHT: [Nate] - implement DHT service - - implement testcases - implement performance tests * FS: [CG] - service: + trust: do not charge when "idle" / load considerations (migration, routing) + artificial delays - + active reply route caching design & implementation of service; gap extension! + + active reply route caching design & implementation; gap extension! - non-anonymous FS service (needs DHT) + DHT integration for search + CS-DHT-functions (DHT-put of LOC) + P2P-functions (DHT-get) - - implement FS performance tests - + insert - + download - + search - + unindex * GNUNET-GTK: [CG] - implement publish dialog details: + open (import directory structure) @@ -56,9 +55,6 @@ - right-clicking on NS list in search dialog should open menu that allows + viewing full meta data + deletion of namespace info - - extend peer dialog with green-yellow-red connectivity status lights - - extend peer dialog with country flags and names - - implement statistics tabs - do meaningful update to status line (starting up, peer running, #connections, shutdown, ...) - implement unindex operation (use dialog with all indexed files for selection) - implement download by URI dialog; figure out where to display those downloads! @@ -74,10 +70,10 @@ 0.9.0pre3: +* Determine RC bugs and fix those! * TRACEKIT: [MW] - good to have for DV/DHT evaluation! -* Determine RC bugs and fix those! -* DATASTORE: +* DATASTORE: [NN] - GNUNET_DATASTORE_cancel method not tested * TESTING: [Nate] - test basic peer re-configure @@ -97,14 +93,29 @@ - reconstruct IBLOCKS from DBLOCKS if possible (during download; see FIXME in fs_download) - add support for pushing "already seen" search results to FS service for bloomfilter (can wait) - use different 'priority' for probe downloads vs. normal downloads + - implement FS performance tests + + insert + + download + + search + + unindex * ARM: [CG/Safey] - better tracking of which config changes actually need to cause process restarts by ARM. - handle gnunet-arm -k in combination with auto-start magic (what is the right thing here?) - discover dependencies between services -* STATISTICS: +* STATISTICS: [NN] - test notification-based statistics API + - implement statistics GUI +* PEERINFO: [NN] + - move peerinfo to new GUI? + - extend peer dialog with green-yellow-red connectivity status lights + - extend peer dialog with country flags and names * SETUP: - design & implement new setup tool +* GNUNET-GTK: + - add tool bar +* MONKEY: [Safey] + - '-m EMAIL' option for alternative e-mail TO + - '-f FILENAME' option to write report to file instead of e-mail (for testing!) 0.9.0: * new webpage: @@ -145,13 +156,13 @@ (Note: build library always, build service when libxml2/etc. are available) * FS: [CG] - Remove KBlocks in gnunet-unindex (see discussion with Kenneth Almquist on gnunet-devs in 9/2009) -* PEERINFO: [CG] +* PEERINFO: [NN] - expire 'ancient' HELLOs (those without valid addresses AND that we have not 'used' (for their public keys) in a while; need a way to track actual 'use') - make sue we also trigger notifications whenever HELLOs expire * VPN -* UTIL: +* UTIL: [CG] - allow limiting UNIX socket access by UID/GID diff --git a/contrib/report.sh b/contrib/report.sh index 521226c94..aab0c2082 100755 --- a/contrib/report.sh +++ b/contrib/report.sh @@ -7,8 +7,6 @@ else WHICH=type fi -echo "On some systems, you may need to change /bin/sh to point to bash" -echo echo "Please submit the following information with your bug report: " echo "--------------------------------------------------------------" OS=`uname -s 2>/dev/null` @@ -86,9 +84,17 @@ fi TEST=`$WHICH gnunetd 2>/dev/null` if test -n "$TEST"; then gnunetd -v | sed -e "s/v//" 2>/dev/null |\ - awk '{print "GNUnet : "$2}' + awk '{print "GNUnet 0.8 : "$2 (may conflict!)}' else - echo "GNUnet : Not Found" + echo "GNUnet 0.8 : Not Found (good)" +fi + +TEST=`$WHICH gnunet-arm 2>/dev/null` +if test -n "$TEST"; then + gnunet-arm -v | sed -e "s/v//" 2>/dev/null |\ + awk '{print "GNUnet 0.9 : "$2}' +else + echo "GNUnet 0.9 : Not Found" fi TEST=`$WHICH libgcrypt-config 2> /dev/null` @@ -169,13 +175,6 @@ else fi -TEST=`grep "Id:" /usr/include/adns.h 2> /dev/null` -if test -n "$TEST"; then - echo "$TEST" | awk '{print "GNU adns : "$4}' -else - echo "GNU adns : Not found" -fi - TEST=`which qmake 2> /dev/null` if test -x "$TEST"; then qmake --version | tail -n 1 | awk '{print "Qt : "$4}' diff --git a/mkinstalldirs b/mkinstalldirs index ef7e16fda..4191a45db 100755 --- a/mkinstalldirs +++ b/mkinstalldirs @@ -1,7 +1,7 @@ #! /bin/sh # mkinstalldirs --- make directory hierarchy -scriptversion=2006-05-11.19 +scriptversion=2009-04-28.21; # UTC # Original author: Noah Friedman # Created: 1993-05-16 @@ -157,5 +157,6 @@ exit $errstatus # eval: (add-hook 'write-file-hooks 'time-stamp) # time-stamp-start: "scriptversion=" # time-stamp-format: "%:y-%02m-%02d.%02H" -# time-stamp-end: "$" +# time-stamp-time-zone: "UTC" +# time-stamp-end: "; # UTC" # End: -- 2.25.1