-PHASE #1: (Goal: settle key design questions)
+Implementable right now (but not necessarily important), with caveats
+(unavailable components that will limit what can be implemented right
+away), in order in which they will likely be done:
+* TESTING-TESTBED [Nate]
+* SETUP
+* DV [Nate]
+* TBENCH
+* TRACEKIT
+* FRAGMENTATION [Ji Lu]
+* HTTP transport
+* MySQL / Postgres plugins (datastore, datacache)
+* UPNP [Milan]
+* ARM [Safey]
+* FS [Christian]
-Util:
-* Windows: use events instead of pipes to signal select()s [Nils]
-* only connect() sockets that are ready (select()) [Nils]
- [On W32, we need to select after calling socket before
- doing connect etc.]
-* Add "DISK" API for creating of temporary files
- (as used in datacache/ module)
-
-PHASE #2: (Goal: recover basic file-sharing functionality)
-
-
-TESTCASES WANTED:
-
-For these functions, it would be nice if we had testcases ("make check")
-that would cause them to be executed and check that they are working:
-* gnunet-service-peerinfo:
- - change_host_trust / flush_trust
- - remove_garbage
- - discard_hosts_helper / cron_clean_data_hosts
-* gnunet-service-transport:
- - try_unvalidated_addresses
- - lookup_address_callback
- - lookup_hello_callback
- - plugin_env_lookup_address
- - notify_clients_disconnect
- - list_validated_addresses
- - cleanup_validation
- - disconnect_neighbour
- - handle_set_quota
-* plugin_transport_tcp.c:
- - tcp_plugin_cancel
- - tcp_plugin_address_pretty_printer / append_port
- - tcp_plugin_set_receive_quota
- - delayed_done
-* transport_api:
- - GNUNET_TRANSPORT_set_qutoa / send_set_quota
- - hello_wait_timeout
- - transmit_ready
- - transmit_timeout
- - remove_from_any_list / remove_neighbour
- - GNUNET_TRANSPORT_notify_transmit_ready_cancel
- - Testcases for set_quota, timeouts, disconnects, transmit_ready_cancel
- - gnunet-service-transport HELLO validation (how good is our coverage?)
- - direct test of plugins compliance to plugin API
-* core_api:
- - timeout_request
- - solicit_traffic / copy_and_free
- - GNUNET_CORE_peer_configure / produce_configure_message
-* gnunet-service-core:
- - update_window
- - find_client
- - handle_client_request_configure
- - set_key_retry_task
- - align_and_deliver
- - handle_transport_notify_disconnect
-* hostlist (everything)
-* topology (everything)
+0.9.0pre0:
+* CORE:
+ - SET_KEY is triggered every 2.5 minutes (after session is up); should
+ just do PING (not SET_KEY + PING)
+ - "Forcing disconnect of XXX due to inactivity" -- can happen every few MS!?
+ (disconnect does not really succeed, or what?)
+ Also, core does not seem to actually decrement the number of active
+ sessions even if no connections exist for a while! (likely related...)
+ - transport reports bw quota violations on big-endian systems (core or
+ transport issue?)
+* FS:
+ - gnunet-publish cannot be aborted using CTRL-C
+ - on some systems, keyword search does not find locally published content
+ (need testcase of command-line tools!)
+ - 2-peer download is still too slow (why?)
+ - advanced FS API parts
+ + search: SBlocks, NBlocks, probes, notify FS-service of known results
+ + location URIs (publish, search, download)
+ + indexing: index-failure-cleanup
+ + gnunet-publish (URI args)
+ + gnunet-download (directory)
+ + gnunet-search (options, incl. namespace search)
+ + gnunet-download (many options)
+ + gnunet-directory (man page, options)
+ + gnunet-pseudonym (all of it)
+ + gnunet-service-fs (remove failing on-demand blocks, hot-path routing, load-based routing, nitpicks)
+ - implement adv. FS testcases
+ + getopt API
+ + insert: sblocks, loc uris
+ + download: loc uris
+ + search: sblocks, nblocks, loc uris
+ + namespaces
+ - [./fs/gnunet-service-fs.c:208]: (style) struct or union member 'LocalGetContext::results_bf_size' is never used
+ - [./fs/gnunet-service-fs.c:501]: (style) struct or union member 'PendingRequest::used_pids_size' is never used
+ - [./fs/gnunet-service-fs.c:654]: (style) struct or union member 'ConnectedPeer::last_client_replies' is never used
+ - [./fs/gnunet-service-fs.c:669]: (style) struct or union member 'ConnectedPeer::avg_delay' is never used
+ - [./fs/gnunet-service-fs.c:675]: (style) struct or union member 'ConnectedPeer::avg_priority' is never used
+ - [./fs/gnunet-service-fs.c:688]: (style) struct or union member 'ConnectedPeer::pending_requests' is never used
+ - [./fs/gnunet-service-fs.c:694]: (style) struct or union member 'ConnectedPeer::last_p2p_replies_woff' is never used
+ - [./fs/gnunet-service-fs.c:700]: (style) struct or union member 'ConnectedPeer::last_client_replies_woff' is never used
+* WWW:
+ - Get IPv6 hooked up
+ - change DNS
-Module features to implement:
-* TESTING (needed for DV, DHT, Topology)
- - review TESTING API
- - implement library for local testing
- - implement testcases for library
- - implement library for distributed testing
- - implement testcases for distributed testing
+0.9.0pre1:
+* Module features to implement:
+ - tbench (RC-pre1)
+ + good to have for DV evaluation!
+ - DV (RC-pre1)
+ + write DV API
+ + implement DV service [Nate & CG]
+ + implement DV library [Nate]
+ + implement DV transport plugin [Nate & CG]
+ + implement testcases [Nate]
+ + implement performance tests [Nate]
+* SETUP:
+ - design & implement new setup tool
+* GUIs to implement:
+ - gtk (RC-pre1)
+ + how to integrate scheduler with GTK event loop!
+* STATISTICS:
+ - does not seem to work with timeouts (especially if service is not running)
+* UTIL:
+ - trust: need *fast* way to check/update trust in peers
+ (async peerinfo would not be right)
+ - scheduler should change OS process priority based on task priority;
+ should make better use of task priorities in general
+ - only connect() sockets that are ready (select()) [Nils]
+ [On W32, we need to select after calling socket before
+ doing connect etc.]
+* FS:
+ - datastore reservation (publishing)
+ - persistence support (publish, unindex, search, download)
+ - download: management/bounding of parallel downloads (for recursive dl.)
+ - non-anonymous FS service (needs DHT)
+ + basic DHT integration
+ + CS-DHT-functions (DHT-put of LOC)
+ + P2P-functions (DHT-get)
+ - active migration support (in fs or in datastore or new daemon?)
+ - GAP improvements:
+ + active reply route caching design & implementation of service; gap extension!
* TOPOLOGY:
- - implement testcases (needs TESTING)
-* HOSTLIST:
- - implement testcases
-* DV (needed for DHT)
- - write DV API
- - implement DV service
- - implement DV library
- - implement DV transport plugin
- - implement testcases
- - implement performance tests
-* DHT (needed for FS)
- - review DHT API
- - implement DHT service (needs DV, DATACACHE)
- - implement DHT library
- - implement testcases
- - implement performance tests
-* FS
- - review FS API
- - design network structs (CS)
- - implement FS library
- + URI API
- + getopt API
- + persistence mechanism
- + sharing API
- ~ file-information
- ~ insert
- ~ download
- ~ search
- ~ unindex
- ~ namespaces
- ~ collection
- + directory API
- - design network structs (P2P)
- - datastore request queueing mechanism
- - implement FS service (needs DHT)
- + insert
- + download
- + search
- + unindex
- - implement testcases
- + URI API
- + getopt API
- + sharing API (needs TESTING)
- ~ file-information
- ~ insert
- ~ download
- ~ search
- ~ unindex
- ~ namespaces
- ~ collection
- + directory API
- - implement performance tests (needs TESTING)
+ - needs more testing (especially F2F topology)
+ - needs to re-try connecting after disconnect (currently, it
+ initially triggers a connection request, but if that connection
+ fails / goes down, it does not retry in a timely fashion;
+ cause seems to be the 'blacklist_after_attempt' being set to 1h,
+ which is rather long -- and should probably be adjusted based on
+ the number of connections / known peers)
+ - If the topology daemon crashes, peers that were put on the
+ blacklist with transport will never be removed from it (until
+ transport service dies); we should use the blacklist notification
+ API to learn about the exact set of blacklisted peers at all times
+ (FIXME: the transport_api implementation of blacklisting
+ also does not work nicely for this since it won't let us know about
+ disconnect-reconnect events and the implicit whitelisting
+ that might happen here; that's not so bad since we will
+ re-blacklist on pre-connect attempts anyway, so this is
+ a minor issue).
+ - the code uses the term 'blacklist' for both peers that are forbidden
+ to connect (i.e. F2F mode) as well as peers that we currently
+ won't try to actively connect to ourselves (since we just tried);
+ This is confusing. We need two distinct terms.
+ - move code to use hash table instead of linked list
+ - instead of periodically discarding blacklisted entries,
+ simply add task that is triggered at the right time (earlier free,
+ more balanced load)
+ - check if new HELLO learned is different from old HELLO
+ before resetting entire state!
+* HELLO:
+ - need function to test "equivalency" of HELLOs (or integrate with "merge"?); use in PEERINFO
+
+0.9.0pre2:
+* Module features to implement:
+ - tracekit (RC-pre2)
+ + good to have for DV/DHT evaluation!
+ - DHT (RC-pre2)
+ + implement DHT service (needs DV, DATACACHE)
+ + implement DHT library
+ + implement testcases
+ + implement performance tests
+* GUIs to implement:
+ - fuse (RC-pre2)
+* Plugins to implement:
+ - UDP backend (RC-pre2)
+ + Fragmentation library
+ + actual plugin
+ - HTTP backend (RC-pre2)
+* ARM:
+ - need to get rid of synchronous API for service starts (cause all kinds of problems)
+ [=> eliminate for need to tell ARM about service starts most of the time!] [Safey]
+ - better tracking of which config changes actually need to cause process restarts by ARM.
+ - listen for requests to discover dependencies between services (and avoid
+ having to explicitly program start requests)
+ - better crash management (attach debugging support, capture and analyze
+ debug output, detect random vs. deterministic crashes)
+ - shutdown sequence?
+* FS:
+ - collection API & tests
+ - implement FS performance tests
+ insert
+ download
+ search
+ unindex
-
-PHASE #3: (Goal: ready for pre-release) [completion-goal: end of 2009]
-
-Module features to implement:
-* setup
- - default generation
- - need to settle basic design; do we want to keep guile?
-* tbench
- - good to have for DV evaluation!
-* tracekit
- - good to have for DV/DHT evaluation!
-* vpn
-
-
-GUIs to implement:
-* gtk
-* qt
-* fuse
-
-
-Plugins to implement:
-* UDP backend (need LIBRARY to support (de)fragmentation!)
-* HTTP backend
-
+0.9.0pre3:
+* GUIs to implement:
+ - qt (RC-pre3)
+ + see discussions @ FISL about integration with event loop!
+* Determine RC bugs and fix those!
+* TESTING:
+ - modify configuration to allow controlling connections for non-local starts
+ - testbed creation with topology (needs working F2F topology) [Nate]
+ - testbed with churn [Nate]
+ - implement testcases for distributed testing [Nate]
+ - test basic peer re-configure [Nate]
+ - test topology creation [Nate]
+ - test churn generation [Nate]
+ - consider changing API for peer-group termination to
+ call continuation when done
+0.9.0:
+* Documentation
+ - update man pages
+ - update webpage documentation
+* new webpage:
+ - expand bibliography
+ - convert documentation pages to books
+ - update books (especially for developers)
+ - create good Drupal theme for GNUnet
+ - make a NICE download page and figure out how to
+ enable developers to publish TGZs nicely
+ - port "contact" page
+ - add content type for "todo" items?
+ - DNS activation
+* Plugins to implement:
+ - MySQL database backends
+ + datacache
+ + datastore
+ - Postgres database backends
+ + datacache
+ + datastore
+ - vpn
+* Determine RC bugs and fix those!
-Minor TODO items / known bugs:
-* UTIL:
- - crypto_hash: use libgcrypt (supports SHA-512 since 2003)
- - container_bloomfilter: improve efficiency (see FIXME)
-* SERVER:
- - inefficient memmove
+0.9.x:
* TRANSPORT:
- - transport_api: support forcing disconnects through low quotas!
- (required for working F2F support!)
- - API: consider having core provide deadline information for each message
- (likely important for DV plugin which wants to loop back!)
- - implement transport API to pretty-print transport address
- + transport_api extension (API extension!)
- + service-transport extension (protocol extension)
- - add calls to statistics in various places
- - implement gnunet-transport (transport configurator / tester)
- - UPnP-based IP detection
- (Note: build library always, build service when libxml2/etc. are available)
- - instantly filter addresses from *other* peers that
- are *equal* to our own address + port (i.e., localhost:2086). We
- no longer filter those for outgoing (helps with loopback testing
- and keeps the code clean), but we should filter strictly *impossible*
- incoming addresses! This is for efficiency, not correctness.
- - We currently are happy to take any address told to us in a WELCOME
- to our set of addresses; we should have some minimal threshold-based
+ - SMTP transport backend
+ - HTTPS transport backend
+ + improved HTTPS support in MHD
+ + actual plugin
+ - Implement method of learning our external addresses from
+ other peers; need some kind of threshold-based
scheme, limiting both the total number of addresses that we accept
this way as well as requiring multiple confirmations; also, we
should possibly try to confirm that the given address works for
us ourselves (loopback-style) before adding it to the list
- [SECURITY issue]
+ we may be able to simplify WELCOME messages (no need to add
addresses there anymore, but may help to learn them there anyway...).
+ we probably want some kind of voting/counting for learning IP addresses
a way to easily "veto" addresses off the list!
=> If MiM attacker uses vetoed address, blacklist the specific IP for
the presumed neighbour!
- - not sure current way of doing ACKs works well-enough
- with unreliable transports where the ACK maybe lost;
- the "is_new" check would then possibly prevent future
- ACKs to be delivered, all while we're happily
- receiving messages from that peer! Worse, the other
- peer won't generate another ACK since it thinks we're
- connected just fine...
- Key questions:
- + How necessary is ACKing in the first place? (alternatives?)
- + Should we transmit ACKs in response to every HELLO? (would that
- fully address the problem?)
- - latency measurements implemented in the transport
- plugins makes it only work for bi-di transports
- and results in code replication
- - should latency be included in the ReceiveCallback and
- NotifyConnect or passed on request?
- - FIXME's with latency being simply set to 0 in a few places
- - Memory leak (running valgrind --trace-children=yes on test_transport_api:
- ==28393== 16 bytes in 1 blocks are indirectly lost in loss record 1 of 5
- ==28393== at 0x4C2260E: malloc (vg_replace_malloc.c:207)
- ==28393== by 0x52343E3: GNUNET_xmalloc_unchecked_ (common_allocation.c:62)
- ==28393== by 0x5234389: GNUNET_xmalloc_ (common_allocation.c:53)
- ==28393== by 0x524458A: GNUNET_NETWORK_socket_create_from_accept (network.c:289)
- ==28393== by 0x524B2DA: ??? (server.c:332)
- ==28393== by 0x524A4C7: ??? (scheduler.c:425)
- ==28393== by 0x524A73D: GNUNET_SCHEDULER_run (scheduler.c:510)
- ==28393== by 0x524FF8C: GNUNET_SERVICE_run (service.c:1326)
- ==28393== by 0x405500: main (gnunet-service-transport.c:2645)
- And also:
- ==28393== 65,744 (65,728 direct, 16 indirect) bytes in 1 blocks are definitely lost in loss record 5 of 5
- ==28393== at 0x4C2260E: malloc (vg_replace_malloc.c:207)
- ==28393== by 0x52343E3: GNUNET_xmalloc_unchecked_ (common_allocation.c:62)
- ==28393== by 0x5234389: GNUNET_xmalloc_ (common_allocation.c:53)
- ==28393== by 0x524473E: GNUNET_NETWORK_socket_create_from_accept (network.c:323)
- (rest of trace identical)
-
-* DATASTORE:
- - mysql backend
- - postgres backend
-* SETUP:
- - auto-generate "defaults.conf" using gnunet-setup from "config.scm"
- - integrate all options into "config.scm"
- - change config-file writing to exclude options set to default values
-* ARM:
- - implement exponential back-off for service restarts
- - better tracking of which config changes actually need to cause process restarts by ARM.
- - have way to specify dependencies between services (to manage ARM restarts better)
- - client-API is inefficient since it opens a TCP connection per service that is started
- (instead of re-using connections).
-* CORE:
- - code currently notifies clients about "encrypted" connections being up well before
- we get the encrypted PONG; sometimes this may be OK (for topology killing
- unwanted connnections), but of course not in general. I suspect we want
- to signal on PONG and have topology hook directly into transport to
- kill plaintext connections before they have a chance to become encrypted
- (may require minor hack in transport API)
+ - implement gnunet-transport (transport configurator / tester)
+ - UPnP-based IP detection
+ (Note: build library always, build service when libxml2/etc. are available)
+* FS:
+ - Remove KBlocks in gnunet-unindex (see discussion with Kenneth Almquist on gnunet-devs in 9/2009)
+ - Allow checking of presence of search results and/or content via command-line tools
+ (add options to gnunet-search / gnunet-download to limit search to local peer)
* PEERINFO:
- - have gnunet-peerinfo print actual host addresses again
- - add option to gnunet-peerinfo to modify trust value
-* POSTGRES-DB:
- - finish postgres implementation; simplify other SQLs using new stats
-* HTTPS transport
- - PolariSSL for MHD?
- - https integration
-* GAP improvements:
- - active reply route caching design & implementation of service,
- gap extension!
-* HOSTLIST:
- - implement advertising of hostlist URL
- - implement learning of hostlist URLs
+ - expire 'ancient' HELLOs (those without valid addresses AND that
+ we have not 'used' (for their public keys) in a while; need a way
+ to track actual 'use')
+ - make sue we also trigger notifications whenever HELLOs expire
-=> PRE-RELEASE
-
-PHASE #4: [completion-goal: mid 2010]
-* Documentation
- - Doxygen generation
- - lcov generation
- - update man pages
- - update webpage documentation
-* new webpage:
- - expand bibliography
- - convert documentation pages to books
- - update books (especially for developers)
- - add content type for links and view
- - create good Drupal theme for GNUnet
- - make a NICE download page and figure out how to
- enable developers to publish TGZs nicely
- - port "contact" page; add impressum
- - add content type for "todo" items?
- - DNS activation
-
-
-Stuff to remember:
-* Features eliminated from util:
- - threading (goal: good riddance!)
- - complex logging features [ectx-passing, target-kinds] (goal: good riddance!)
- - complex configuration features [defaults, notifications] (goal: good riddance!)
- - network traffic monitors (goal: eliminate)
- - IPC semaphores (goal: d-bus? / eliminate?)
- - second timers
- - DNS lookup (goal: have async service; issue: still need synchronous resolution in places, current code may not be portable)
- => code shrunk from 61 files to 34, 22k LOC to 15k LOC, 470k to 330k (with symbols)
-* New features in util:
- - scheduler
- - service and program boot-strap code
-* Major changes in util:
- - more expressive server (replaces selector)
-* Open questions:
- - how to integrate scheduler with GTK event loop!
-
-
+Optimizations:
+* TCP:
+ - should use hash map to look up sessions
+* PEERINFO:
+ - api creates many, many short-lived TCP connections; either some
+ clients should use the API differently or we need to change the
+ API to enable re-use of connections to the service
+* STATISTICS:
+ - should use BIO instead of mmap
+* TRANSPORT:
+ - need to periodically probe latency/transport cost changes & possibly switch transport
+ - instantly filter addresses from *other* peers that
+ are *equal* to our own address + port (i.e., localhost:2086). We
+ no longer filter those for outgoing (helps with loopback testing
+ and keeps the code clean), but we should filter strictly *impossible*
+ incoming addresses! This is for efficiency, not correctness.
+ - should use hash map to look up Neighbours
+* HOSTLIST:
+ - 'server' uses 'GNUNET_PEERINFO_iterate', should probably switch to notification API
+ (for more instant / up-to-date hostlists at lower cost) [OPTIMIZATION]
-Test coverage:
-* UTIL : 78.7%
-* HELLO : 93.7%
-* ARM : 69.9%
-* RESOLVER : 60.4%
-* STATISTICS: 82.8%
-* PEERINFO : 71.5%
-* TRANSPORT : 70.9%
-* CORE : 65.8%
-===================
-* TOTAL : 74.9%
+Minor features:
+* TCP:
+ - repeatedly resolve hostname and look up interfaces to determine our own IP
+ - [./transport/plugin_transport_tcp.c:391]: (style) struct or union member 'Plugin::address_update_task' is never used (related to issue above)
+* TRANSPORT:
+ - [./transport/gnunet-service-transport.c:173]: (style) struct or union member 'TransportPlugin::rebuild' is never used (related to TCP not refreshing external addresses?)
+* DATACACHE:
+ - add stats (# bytes available, # bytes used, # PUTs, # GETs, # GETs satisfied)
-Not yet tested:
-* HOSTLIST : 0.0%
-* TOPOLOGY : 0.0%
projects / oweals / gnunet.git / blobdiff