X-Git-Url: https://git.librecmc.org/?a=blobdiff_plain;ds=sidebyside;f=TODO;h=6c9e18429ffad3247a84f8f2a2fb955545773f49;hb=b0d073db4d1f0006ec1d6a7152c7543c80cea268;hp=e37e667396c5737fcd6b5e7be04389864674c307;hpb=18eca849801470fc35f5a484672f4ed3f57c5730;p=oweals%2Fgnunet.git

diff --git a/TODO b/TODO
index e37e66739..6c9e18429 100644
--- a/TODO
+++ b/TODO
@@ -1,148 +1,146 @@
-Implementable right now (but not necessarily important), with caveats
-(unavailable components that will limit what can be implemented right
-away), in order in which they will likely be done:
-* TESTING
-* FS (DHT not available)
-* SETUP
-* DV (distributed testing not available)
-* TBENCH (distributed testing not available)
-* TRACEKIT (distributed testing not available)
-* HTTP transport
-* FRAGMENTATION
-* MySQL / Postgres plugins (datastore, datacache)
-* UPNP
-
-
-Urgent items (before announcing ng.gnunet.org):
-* TESTING (needed for DV, DHT, Topology)
-  - implement library for local testing
-    + modify configuration to allow controlling
-      connections for non-local starts
-    + CORE service does not start with valid peer ID (all zeros)
-      -- testcase fails!
-    + consider changing API for peer-group termination
-      to call continuation when done
-    + testbed creation with topology (needs working F2F topology) [Nate]
-    + testbed with churn [Nate]
-  - implement testcases for library
-    + get test for basic peer start to work!
-    + test basic peer connect
-    + test group start
-    + test basic peer re-configure [Nate]
-    + test topology creation [Nate]
-    + test churn generation [Nate]
-* TOPOLOGY:
-  - implement testcases (needs TESTING)
-* HOSTLIST:
-  - implement testcases
-* FS (basic anonymous FS only)
-  - design network structs (P2P)
-    + query
-    + response
-  - implement FS service
-    + CS-functions (query BF, on-demand encoding, indexed-list serialization/deserialization)
-    + P2P-functions (query, response)
+0.9.0pre1:
+* PEERINFO:
+  - trust: need *fast* way to check/update trust in peers
+           (async peerinfo would not be right; certainly not with the
+            current API)
+* UTIL:
+  - scheduler should change OS process priority based on task priority;   
+    should make better use of task priorities in general
+  - only connect() sockets that are ready (select()) [Nils]
+    [On W32, we need to select after calling socket before
+     doing connect etc.]
+* SETUP:
+  - design & implement new setup tool
+* TBENCH: [MW]
+  - good to have for transport/DV evaluation! 
+* DV: [Nate]
+  - write DV API (need to move declarations from dv_api.c to gnunet_dv_service.h!)
+  - implement DV service 
+  - implement DV library (looks done)
+  - implement DV transport plugin
   - implement testcases 
-    + URI API
-    + getopt API
-    + sharing API
-      ~ file-information
-      ~ publish (insert, index)
-      ~ download
-      ~ search
-      ~ unindex
-    + directory API
-* new webpage
-  - prevent?
-  - migrate Mantis!?
-  - run peer => have a 0.9.x hostlist
-  - improve basic documentation (configure, dependencies, what works, etc.)
-  - write minimal welcome/introduction to 0.9.x-development/webpage/status
-=> Deploy(able) development network
-
+  - implement performance tests
+* STATISTICS:
+  - does not seem to work with timeouts (especially if service is not running)
+* TOPOLOGY:
+  - needs more testing (especially F2F topology)
+  - needs to re-try connecting after disconnect (currently, it
+    initially triggers a connection request, but if that connection
+    fails / goes down, it does not retry in a timely fashion;
+    cause seems to be the 'blacklist_after_attempt' being set to 1h,
+    which is rather long -- and should probably be adjusted based on
+    the number of connections / known peers)
+  - If the topology daemon crashes, peers that were put on the
+    blacklist with transport will never be removed from it (until
+    transport service dies); we should use the blacklist notification
+    API to learn about the exact set of blacklisted peers at all times
+    (FIXME: the transport_api implementation of blacklisting
+     also does not work nicely for this since it won't let us know about
+     disconnect-reconnect events and the implicit whitelisting
+     that might happen here; that's not so bad since we will
+     re-blacklist on pre-connect attempts anyway, so this is 
+     a minor issue; OTOH, we might want to be more explicit about
+     allowing/forbidding connects on pre-connect to avoid 
+     entering connect attempts to just be blacklisted shortly afterwards).
+  - the code uses the term 'blacklist' for both peers that are forbidden
+    to connect (i.e. F2F mode) as well as peers that we currently
+    won't try to actively connect to ourselves (since we just tried);
+    This is confusing.  We need two distinct terms (greylist?).
+  - move code to use hash table instead of linked list
+  - instead of periodically discarding blacklisted entries,
+    simply add task that is triggered at the right time (earlier free,
+    more balanced load)
+  - check if new HELLO learned is different from old HELLO
+    before resetting entire state!
+* FS: [CG]
+  - support recursive download even if filename is NULL and we hence
+    do not generate files on disk (use temp_filename)
+  - bound parallelism (# fs downloads)
+  - distinguish in performance tracking and event signalling between
+    downloads that are actually running and those that are merely in the queue
+  - persistence support (publish, unindex, search, download)
+  - active migration support (in fs or in datastore or new daemon?)
+  - gnunet-service-fs (hot-path routing, load-based routing, nitpicks)
+  - [gnunet-service-fs.c:208]: member 'LocalGetContext::results_bf_size' is never used
+  - [gnunet-service-fs.c:501]: member 'PendingRequest::used_pids_size' is never used
+  - [gnunet-service-fs.c:654]: member 'ConnectedPeer::last_client_replies' is never used
+  - [gnunet-service-fs.c:669]: member 'ConnectedPeer::avg_delay' is never used
+  - [gnunet-service-fs.c:675]: member 'ConnectedPeer::avg_priority' is never used
+  - [gnunet-service-fs.c:688]: member 'ConnectedPeer::pending_requests' is never used
+  - [gnunet-service-fs.c:694]: member 'ConnectedPeer::last_p2p_replies_woff' is never used
+  - [gnunet-service-fs.c:700]: member 'ConnectedPeer::last_client_replies_woff' is never used
+  - GAP improvements:
+    + active reply route caching design & implementation of service; gap extension!
+* DATASTORE:
+  - API lacks cancellation methods (needed? or is disconnect enough?); 
+    may also want to integrate request queuing here instead of gnunet-service-fs_drq.c 
+* GNUNET-GTK:
+  - use g_main_context_set_poll_func to integrate GTK with GNUnet Scheduler!? (YUCK!)
+  - OR: add scheduler API to enable integration with GTK main loop instead of doing our own select
+  - use g_main_context_pending, g_main_context_query / g_main_context_check / g_main_context_dispatch
+    and NEVER g_main_loop_run (can this be done? might be the clean way to do this! But how
+    to integrate this with "tak_main"?  Docu says:
+    "It's OK to use the GLib main loop directly instead of gtk_main(), though it involves 
+     slightly more typing. See GMainLoop in the GLib documentation."
+    => so maybe it "just works"?
 
-0.9.0pre0:
-* Module features to implement:
-  - advanced FS API parts
-    + namespaces
-      ~ fundamental namespace API
-    + search: SBlocks, KSBlocks, probes
-    + collection
-    + location URIs (publish, search, download)
-    + persistence support (publish, unindex, search, download)
-    + indexing: index-failure-cleanup
-    + download: management/bounding of parallel downloads (for recursive dl.)
-    + datastore reservation (publishing)
-    + gnunet-publish (URI args)
-    + gnunet-download (directory)
-    + gnunet-search (options, incl. namespace search)
-    + gnunet-download (many options)
-    + gnunet-directory (man page, options)
-    + gnunet-pseudonym (all of it)
-  - implement adv. FS testcases 
-    + insert: sblocks, loc uris
-    + download: loc uris
-    + search: sblocks, skblocks, loc uris
-    + namespaces
-    + collection
-  - implement FS performance tests (needs TESTING)
+0.9.0pre2:
+* TRACEKIT: [MW]
+  - good to have for DV/DHT evaluation!
+* DHT: [Nate]
+  - implement DHT service 
+  - implement DHT library
+  - implement testcases
+  - implement performance tests
+* TRANSPORT:
+  - HTTP backend [MW]
+* ARM: [Safey]
+  - need to get rid of synchronous API for service starts (cause all kinds of problems)
+    [=> eliminate for need to tell ARM about service starts most of the time!] [Safey]
+  - better tracking of which config changes actually need to cause process restarts by ARM.
+  - listen for requests to discover dependencies between services (and avoid
+    having to explicitly program start requests)
+  - better crash management (attach debugging support, capture and analyze
+    debug output, detect random vs. deterministic crashes)
+  - shutdown sequence?
+* FS: [CG]
+  - datastore reservation (publishing)
+  - search: availability probes
+  - location URIs (publish, search, download)
+  - non-anonymous FS service (needs DHT)
+    + DHT integration for search
+    + CS-DHT-functions (DHT-put of LOC)
+    + P2P-functions (DHT-get)
+  - collection API & tests
+    + gnunet-pseudonym (collection support)
+  - implement FS performance tests
     + insert
     + download
     + search
     + unindex
-  - non-anonymous FS service (needs DHT)
-    + basic DHT integration
-    + CS-DHT-functions (DHT-put of LOC)
-    + P2P-functions (DHT-get)
-  - setup (RC-pre0)
-    + default generation
-    + need to settle basic design; do we want to keep guile?
-  - testing (RC-pre0)
-    + implement library for distributed testing [Nate]
-    + implement testcases for distributed testing [Nate]
-
-0.9.0pre1:
-* Module features to implement:
-  - tbench (RC-pre1)
-    + good to have for DV evaluation!
-  - DV (RC-pre1)
-    + write DV API 
-    + implement DV service [Nate & CG]
-    + implement DV library [Nate]
-    + implement DV transport plugin [Nate & CG]
-    + implement testcases [Nate]
-    + implement performance tests [Nate]
-* GUIs to implement:
-  - gtk (RC-pre1)
-    + how to integrate scheduler with GTK event loop!
-
-0.9.0pre2:
-* Module features to implement:
-  - tracekit (RC-pre2)
-    + good to have for DV/DHT evaluation!
-  - DHT (RC-pre2)
-    + implement DHT service (needs DV, DATACACHE)
-    + implement DHT library
-    + implement testcases
-    + implement performance tests
-* GUIs to implement:
-  - fuse (RC-pre2)
-* Plugins to implement:
-  - UDP backend (RC-pre2)
-    + Fragmentation library 
-    + actual plugin
-  - HTTP backend (RC-pre2)
 
 0.9.0pre3:
-* GUIs to implement:
-  - qt (RC-pre3)
-    + see discussions @ FISL about integration with event loop!
 * Determine RC bugs and fix those!
+* TESTING: [Nate]
+  - modify configuration to allow controlling connections for non-local starts
+  - testbed creation with topology (needs working F2F topology) 
+  - testbed with churn 
+  - implement testcases for distributed testing
+  - test basic peer re-configure 
+  - test topology creation 
+  - test churn generation 
+  - consider changing API for peer-group termination to 
+    call continuation when done
+* NAT/UPNP: [MW]
+  - finalize API design
+  - code clean up
+  - testing
+  - integration with transport service
+* MYSQL database backends: [CG]
+  - datacache
+  - datastore
 
 0.9.0:
-* Documentation
-  - update man pages
-  - update webpage documentation
 * new webpage:
   - expand bibliography
   - convert documentation pages to books
@@ -150,23 +148,82 @@ Urgent items (before announcing ng.gnunet.org):
   - create good Drupal theme for GNUnet
   - make a NICE download page and figure out how to 
     enable developers to publish TGZs nicely
-  - port "contact" page; add impressum
+  - port "contact" page
   - add content type for "todo" items?
-  - DNS activation
-* Plugins to implement:
-  - MySQL database backends
-    + datacache
-    + datastore
-  - Postgres database backends
-    + datacache
-    + datastore
-  - vpn
+* POSTGRES database backends: [CG]
+  - datacache
+  - datastore
 * Determine RC bugs and fix those!
 
-
 0.9.x:
-* Plugins to implement:
+* TRANSPORT: [MW]
   - SMTP transport backend
   - HTTPS transport backend
     + improved HTTPS support in MHD
     + actual plugin
+  - Implement method of learning our external addresses from
+    other peers; need some kind of threshold-based
+    scheme, limiting both the total number of addresses that we accept 
+    this way as well as requiring multiple confirmations; also, we
+    should possibly try to confirm that the given address works for
+    us ourselves (loopback-style) before adding it to the list
+    + we may be able to simplify WELCOME messages (no need to add 
+      addresses there anymore, but may help to learn them there anyway...).
+    + we probably want some kind of voting/counting for learning IP addresses
+      (maybe including IP addresses in ads proportional to how often others
+       report them? we at least need some protection against >64k HELLOs!),
+    + provide a way to give the user a list of "learned" IP addresses and
+      a way to easily "veto" addresses off the list!
+      => If MiM attacker uses vetoed address, blacklist the specific IP for
+         the presumed neighbour!
+  - implement gnunet-transport (transport configurator / tester)
+  - UPnP-based IP detection
+    (Note: build library always, build service when libxml2/etc. are available)
+* FS: [CG]
+  - Remove KBlocks in gnunet-unindex (see discussion with Kenneth Almquist on gnunet-devs in 9/2009)
+  - Allow checking of presence of search results and/or content via command-line tools
+    (add options to gnunet-search / gnunet-download to limit search to local peer)
+* PEERINFO: [CG]
+  - expire 'ancient' HELLOs (those without valid addresses AND that 
+    we have not 'used' (for their public keys) in a while; need a way
+    to track actual 'use')
+  - make sue we also trigger notifications whenever HELLOs expire
+* VPN
+
+
+
+Optimizations:
+* TCP:
+  - should use hash map to look up sessions
+* PEERINFO:
+  - api creates many, many short-lived TCP connections; either some
+    clients should use the API differently or we need to change the
+    API to enable re-use of connections to the service
+* STATISTICS:
+  - should use BIO instead of mmap
+* TRANSPORT:
+  - need to periodically probe latency/transport cost changes & possibly switch transport
+  - instantly filter addresses from *other* peers that 
+    are *equal* to our own address + port (i.e., localhost:2086).  We 
+    no longer filter those for outgoing (helps with loopback testing
+    and keeps the code clean), but we should filter strictly *impossible*
+    incoming addresses!  This is for efficiency, not correctness.
+  - should use hash map to look up Neighbours
+* HOSTLIST:
+  - 'server' uses 'GNUNET_PEERINFO_iterate', should probably switch to notification API
+    (for more instant / up-to-date hostlists at lower cost) [OPTIMIZATION]
+* DATASTORE (?):
+  - check for duplicates on insertion (currently, same content is frequently
+    stored again [seen with KBLOCKS and SBLOCKS]!)
+
+Minor features:
+* TCP:
+  - repeatedly resolve hostname and look up interfaces to determine our own IP
+  - [./transport/plugin_transport_tcp.c:391]: (style) struct or union member 'Plugin::address_update_task' is never used (related to issue above)
+* TRANSPORT:
+  - [./transport/gnunet-service-transport.c:173]: (style) struct or union member 'TransportPlugin::rebuild' is never used (related to TCP not refreshing external addresses?)
+* DATACACHE:
+  - add stats (# bytes available, # bytes used, # PUTs, # GETs, # GETs satisfied)
+* FS:
+  - support inline data in directories for recursive file downloads (fs_download)
+