-PHASE #1: (Goal: settle key design questions)
-
-Util:
-* Windows: use events instead of pipes to signal select()s [Nils]
-* only connect() sockets that are ready (select()) [Nils]
- [On W32, we need to select after calling socket before
- doing connect etc.]
-
-PHASE #2: (Goal: recover basic file-sharing functionality)
-
-
-TESTCASES WANTED:
-
-For these functions, it would be nice if we had testcases ("make check")
-that would cause them to be executed and check that they are working:
-* gnunet-service-peerinfo:
- - change_host_trust / flush_trust
- - remove_garbage /
- - discard_hosts_helper / cron_clean_data_hosts
-* gnunet-service-transport:
- - try_unvalidated_addresses
- - lookup_address_callback
- - lookup_hello_callback
- - plugin_env_lookup_address
- - notify_clients_disconnect
- - list_validated_addresses
- - cleanup_validation
- - disconnect_neighbour
- - handle_set_quota
-* plugin_transport_tcp.c:
- - tcp_plugin_cancel
- - tcp_plugin_address_pretty_printer / append_port
- - tcp_plugin_set_receive_quota
- - delayed_done
-* transport_api:
- - GNUNET_TRANSPORT_set_qutoa / send_set_quota
- - hello_wait_timeout
- - transmit_ready
- - transmit_timeout
- - remove_from_any_list / remove_neighbour
- - GNUNET_TRANSPORT_notify_transmit_ready_cancel
- - Testcases for set_quota, timeouts, disconnects, transmit_ready_cancel
- - gnunet-service-transport HELLO validation (how good is our coverage?)
- - direct test of plugins compliance to plugin API
-* core_api:
- - timeout_request
- - solicit_traffic / copy_and_free
- - GNUNET_CORE_peer_configure / produce_configure_message
-* gnunet-service-core:
- - update_window
- - find_client
- - handle_client_request_configure
- - set_key_retry_task
- - align_and_deliver
- - handle_transport_notify_disconnect
-* hostlist (everything)
-* topology (everything)
-
-
-Module features to implement:
-* datacache (needed for DHT)
- - design plugin API
- - implement sqlite-based dstore plugin
- - implement dstore API
- - implement testcases for dstore API
- - implement testcases for dstore plugin
- - implement performance tests
-* TESTING (needed for DV, DHT, Topology)
- - review TESTING API
- - implement library for local testing
- - implement testcases for library
- - implement library for distributed testing
- - implement testcases for distributed testing
+Implementable right now (but not necessarily important), with caveats
+(unavailable components that will limit what can be implemented right
+away), in order in which they will likely be done:
+* TESTING-TESTBED [Nate]
+* SETUP
+* DV [Nate]
+* TBENCH
+* TRACEKIT
+* FRAGMENTATION [Ji Lu]
+* HTTP transport
+* MySQL / Postgres plugins (datastore, datacache) [Radhika]
+* UPNP [Milan]
+
+Urgent items (before announcing ng.gnunet.org):
+* UTIL:
+ - provide higher-level convenience API for servers/services that
+ need to send a stream of notifications to clients; instead
+ of having each service queue messages and "send when ready",
+ simply have a way to add a client to the notification set
+ and to 'notify client' or 'notify all clients'
+ (transport (our hello),
+ core (misc monitoring features)
+ and likely others)
+ - server/service API change for ARM inetd'ing
+ (listen as well as support for start with multiple, already
+ bound & listening sockets!)
+* TRANSPORT:
+ - main service not implemented [Nate]
+ - blacklist not implemented [Christian]
+ - testcases crash & burn (no surprise)
+* CORE:
+ - request disconnect not implemented [Christian]
+ - various notification options not implemented [Christian]
+ - test currently fails spectacularly [segv of transport service]
+ => need transport to work first!
+ - request connect not working [Christian, need transport first]
+* PEERINFO:
+ - make sue we also trigger notifications whenever HELLOs expire
* TOPOLOGY:
- - implement testcases (needs TESTING)
+ - needs testing [need transport first]
+* TESTING:
+ - check that 'GNUNET_TRANSPORT_get_hello' is associated with
+ a cancel request wherever appropriate (other testcases
+ in particular!) [Christian]
* HOSTLIST:
- - implement testcases
-* DV (needed for DHT)
- - write DV API
- - implement DV service
- - implement DV library
- - implement DV transport plugin
- - implement testcases
- - implement performance tests
-* DHT (needed for FS)
- - review DHT API
- - implement DHT service (needs DV, DSTORE)
- - implement DHT library
- - implement testcases
- - implement performance tests
-* FS
- - review FS API
- - design network structs (CS)
- - implement FS library
- + URI API
- + getopt API
- + persistence mechanism
- + sharing API
- ~ file-information
- ~ insert
- ~ download
- ~ search
- ~ unindex
- ~ namespaces
- ~ collection
- + directory API
- - design network structs (P2P)
- - datastore request queueing mechanism
- - implement FS service (needs DHT)
- + insert
- + download
- + search
- + unindex
- - implement testcases
- + URI API
+ - test fails (looks like it works, but that's because of a bad
+ connectivity notification; somehow core is unable to send
+ messages successfully via transport) [need transport first]
+* FS (basic anonymous FS only)
+ - implement FS service (P2P operations)
+ + how to send queries (soliciting is not there in core; do we
+ also want to do pushing sometimes?)
+ + need to bound queueing of replies for other peers
+ - test multi-peer search/download
+* new webpage
+ - run peer => have a 0.9.x hostlist
+ - improve basic documentation (configure, dependencies, what works, etc.)
+ - write minimal welcome/introduction to 0.9.x-development/webpage/status
+=> Deploy(able) development network
+
+
+0.9.0pre0:
+* UTIL:
+ - load: need way to determine network load (up/down)
+ - trust: need *fast* way to check/update trust in peers
+ (async peerinfo would not be right)
+ - OS: existing waitpid call is not nice (not integratable with scheduler! fix this!)
+* STATISTICS:
+ - synchronous/asynchronous API (& implementation) is not nice;
+ => provide notification-based API
+ - does not seem to work with timeouts (especially if service is not running)
+* ARM:
+ - need to get rid of synchronous API for service starts (cause all kinds of problems)
+ [=> eliminate for need to tell ARM about service starts most of the time!]
+* HELLO:
+ - need function to test "equivalency" of HELLOs; use in topology!
+* CORE:
+ - outbound message monitoring not supported
+* Module features to implement:
+ - advanced FS API parts
+ + namespaces: fundamental namespace API
+ + search: SBlocks, KSBlocks, probes, notify FS-service of known results
+ + collection
+ + location URIs (publish, search, download)
+ + persistence support (publish, unindex, search, download)
+ + indexing: index-failure-cleanup
+ + download: management/bounding of parallel downloads (for recursive dl.)
+ + datastore reservation (publishing)
+ + gnunet-publish (URI args)
+ + gnunet-download (directory)
+ + gnunet-search (options, incl. namespace search)
+ + gnunet-download (many options)
+ + gnunet-directory (man page, options)
+ + gnunet-pseudonym (all of it)
+ + gnunet-service-fs (remove failing on-demand blocks, many other nitpicks/features/optimizations)
+ + datastore: do active migration support here?
+ - implement adv. FS testcases
+ getopt API
- + sharing API (needs TESTING)
- ~ file-information
- ~ insert
- ~ download
- ~ search
- ~ unindex
- ~ namespaces
- ~ collection
- + directory API
- - implement performance tests (needs TESTING)
+ + insert: sblocks, loc uris
+ + download: loc uris
+ + search: sblocks, skblocks, loc uris
+ + namespaces
+ + collection
+ - implement FS performance tests (needs TESTING)
+ insert
+ download
+ search
+ unindex
-
-
-
-PHASE #3: (Goal: ready for pre-release) [completion-goal: end of 2009]
-
-Module features to implement:
-* setup
- - default generation
- - need to settle basic design; do we want to keep guile?
-* tbench
- - good to have for DV evaluation!
-* tracekit
- - good to have for DV/DHT evaluation!
-* vpn
-
-
-GUIs to implement:
-* gtk
-* qt
-* fuse
-
-
-Plugins to implement:
-* UDP backend (need LIBRARY to support (de)fragmentation!)
-* HTTP backend
-
-
-
-
-
-Minor TODO items / known bugs:
-* UTIL:
- - crypto_hash: use libgcrypt (supports SHA-512 since 2003)
- - container_bloomfilter: improve efficiency (see FIXME)
-* SERVER:
- - inefficient memmove
-* TRANSPORT:
- - transport_api: support forcing disconnects through low quotas!
- (required for working F2F support!)
- - API: consider having core provide deadline information for each message
- (likely important for DV plugin which wants to loop back!)
- - implement transport API to pretty-print transport address
- + transport_api extension (API extension!)
- + service-transport extension (protocol extension)
- - add calls to statistics in various places
- - implement gnunet-transport (transport configurator / tester)
- - UPnP-based IP detection
- (Note: build library always, build service when libxml2/etc. are available)
- - instantly filter addresses from *other* peers that
- are *equal* to our own address + port (i.e., localhost:2086). We
- no longer filter those for outgoing (helps with loopback testing
- and keeps the code clean), but we should filter strictly *impossible*
- incoming addresses! This is for efficiency, not correctness.
- - We currently are happy to take any address told to us in a WELCOME
- to our set of addresses; we should have some minimal threshold-based
- scheme, limiting both the total number of addresses that we accept
- this way as well as requiring multiple confirmations; also, we
- should possibly try to confirm that the given address works for
- us ourselves (loopback-style) before adding it to the list
- [SECURITY issue]
- + we may be able to simplify WELCOME messages (no need to add
- addresses there anymore, but may help to learn them there anyway...).
- + we probably want some kind of voting/counting for learning IP addresses
- (maybe including IP addresses in ads proportional to how often others
- report them? we at least need some protection against >64k HELLOs!),
- + provide a way to give the user a list of "learned" IP addresses and
- a way to easily "veto" addresses off the list!
- => If MiM attacker uses vetoed address, blacklist the specific IP for
- the presumed neighbour!
- - not sure current way of doing ACKs works well-enough
- with unreliable transports where the ACK maybe lost;
- the "is_new" check would then possibly prevent future
- ACKs to be delivered, all while we're happily
- receiving messages from that peer! Worse, the other
- peer won't generate another ACK since it thinks we're
- connected just fine...
- Key questions:
- + How necessary is ACKing in the first place? (alternatives?)
- + Should we transmit ACKs in response to every HELLO? (would that
- fully address the problem?)
- - latency measurements implemented in the transport
- plugins makes it only work for bi-di transports
- and results in code replication
- - should latency be included in the ReceiveCallback and
- NotifyConnect or passed on request?
- - FIXME's with latency being simply set to 0 in a few places
- - Memory leak (running valgrind --trace-children=yes on test_transport_api:
- ==28393== 16 bytes in 1 blocks are indirectly lost in loss record 1 of 5
- ==28393== at 0x4C2260E: malloc (vg_replace_malloc.c:207)
- ==28393== by 0x52343E3: GNUNET_xmalloc_unchecked_ (common_allocation.c:62)
- ==28393== by 0x5234389: GNUNET_xmalloc_ (common_allocation.c:53)
- ==28393== by 0x524458A: GNUNET_NETWORK_socket_create_from_accept (network.c:289)
- ==28393== by 0x524B2DA: ??? (server.c:332)
- ==28393== by 0x524A4C7: ??? (scheduler.c:425)
- ==28393== by 0x524A73D: GNUNET_SCHEDULER_run (scheduler.c:510)
- ==28393== by 0x524FF8C: GNUNET_SERVICE_run (service.c:1326)
- ==28393== by 0x405500: main (gnunet-service-transport.c:2645)
- And also:
- ==28393== 65,744 (65,728 direct, 16 indirect) bytes in 1 blocks are definitely lost in loss record 5 of 5
- ==28393== at 0x4C2260E: malloc (vg_replace_malloc.c:207)
- ==28393== by 0x52343E3: GNUNET_xmalloc_unchecked_ (common_allocation.c:62)
- ==28393== by 0x5234389: GNUNET_xmalloc_ (common_allocation.c:53)
- ==28393== by 0x524473E: GNUNET_NETWORK_socket_create_from_accept (network.c:323)
- (rest of trace identical)
-
-* DATASTORE:
- - mysql backend
- - postgres backend
-* SETUP:
- - auto-generate "defaults.conf" using gnunet-setup from "config.scm"
- - integrate all options into "config.scm"
- - change config-file writing to exclude options set to default values
-* ARM:
- - implement exponential back-off for service restarts
- - better tracking of which config changes actually need to cause process restarts by ARM.
- - have way to specify dependencies between services (to manage ARM restarts better)
- - client-API is inefficient since it opens a TCP connection per service that is started
- (instead of re-using connections).
-* CORE:
- - code currently notifies clients about "encrypted" connections being up well before
- we get the encrypted PONG; sometimes this may be OK (for topology killing
- unwanted connnections), but of course not in general. I suspect we want
- to signal on PONG and have topology hook directly into transport to
- kill plaintext connections before they have a chance to become encrypted
- (may require minor hack in transport API)
-* PEERINFO:
- - have gnunet-peerinfo print actual host addresses again
- - add option to gnunet-peerinfo to modify trust value
-* POSTGRES-DB:
- - finish postgres implementation; simplify other SQLs using new stats
-* HTTPS transport
- - PolariSSL for MHD?
- - https integration
-* GAP improvements:
- - active reply route caching design & implementation of service,
- gap extension!
-* HOSTLIST:
- - implement advertising of hostlist URL
- - implement learning of hostlist URLs
-
-
-
-=> PRE-RELEASE
-
-PHASE #4: [completion-goal: mid 2010]
+ - non-anonymous FS service (needs DHT)
+ + basic DHT integration
+ + CS-DHT-functions (DHT-put of LOC)
+ + P2P-functions (DHT-get)
+ - setup (RC-pre0)
+ + default generation
+ + need to settle basic design; do we want to keep guile?
+ - testing (RC-pre0)
+ + modify configuration to allow controlling connections for non-local starts
+ + testbed creation with topology (needs working F2F topology) [Nate]
+ + testbed with churn [Nate]
+ + implement library for distributed testing [Nate]
+ + implement testcases for distributed testing [Nate]
+ + test basic peer re-configure [Nate]
+ + test topology creation [Nate]
+ + test churn generation [Nate]
+
+0.9.0pre1:
+* Module features to implement:
+ - tbench (RC-pre1)
+ + good to have for DV evaluation!
+ - DV (RC-pre1)
+ + write DV API
+ + implement DV service [Nate & CG]
+ + implement DV library [Nate]
+ + implement DV transport plugin [Nate & CG]
+ + implement testcases [Nate]
+ + implement performance tests [Nate]
+* GUIs to implement:
+ - gtk (RC-pre1)
+ + how to integrate scheduler with GTK event loop!
+
+0.9.0pre2:
+* Module features to implement:
+ - tracekit (RC-pre2)
+ + good to have for DV/DHT evaluation!
+ - DHT (RC-pre2)
+ + implement DHT service (needs DV, DATACACHE)
+ + implement DHT library
+ + implement testcases
+ + implement performance tests
+* GUIs to implement:
+ - fuse (RC-pre2)
+* Plugins to implement:
+ - UDP backend (RC-pre2)
+ + Fragmentation library
+ + actual plugin
+ - HTTP backend (RC-pre2)
+
+0.9.0pre3:
+* GUIs to implement:
+ - qt (RC-pre3)
+ + see discussions @ FISL about integration with event loop!
+* Determine RC bugs and fix those!
+
+0.9.0:
* Documentation
- - Doxygen generation
- - lcov generation
- update man pages
- update webpage documentation
* new webpage:
- expand bibliography
- convert documentation pages to books
- update books (especially for developers)
- - add content type for links and view
- create good Drupal theme for GNUnet
- make a NICE download page and figure out how to
enable developers to publish TGZs nicely
- - port "contact" page; add impressum
+ - port "contact" page
- add content type for "todo" items?
- DNS activation
-
-
-Stuff to remember:
-* Features eliminated from util:
- - threading (goal: good riddance!)
- - complex logging features [ectx-passing, target-kinds] (goal: good riddance!)
- - complex configuration features [defaults, notifications] (goal: good riddance!)
- - network traffic monitors (goal: eliminate)
- - IPC semaphores (goal: d-bus? / eliminate?)
- - second timers
- - DNS lookup (goal: have async service; issue: still need synchronous resolution in places, current code may not be portable)
- => code shrunk from 61 files to 34, 22k LOC to 15k LOC, 470k to 330k (with symbols)
-* New features in util:
- - scheduler
- - service and program boot-strap code
-* Major changes in util:
- - more expressive server (replaces selector)
-* Open questions:
- - how to integrate scheduler with GTK event loop!
-
-
-
-Test coverage:
-* UTIL : 78.7%
-* HELLO : 93.7%
-* ARM : 69.9%
-* RESOLVER : 60.4%
-* STATISTICS: 82.8%
-* PEERINFO : 71.5%
-* TRANSPORT : 70.9%
-* CORE : 65.8%
-===================
-* TOTAL : 74.9%
-
-Not yet tested:
-* HOSTLIST : 0.0%
-* TOPOLOGY : 0.0%
+* Plugins to implement:
+ - MySQL database backends
+ + datacache
+ + datastore
+ - Postgres database backends
+ + datacache
+ + datastore
+ - vpn
+* Determine RC bugs and fix those!
+
+
+0.9.x:
+* Plugins to implement:
+ - SMTP transport backend
+ - HTTPS transport backend
+ + improved HTTPS support in MHD
+ + actual plugin
+* Remove KBlocks in gnunet-unindex (see discussion with Kenneth Almquist on gnunet-devs in 9/2009)
+* Allow checking of presence of search results and/or content via command-line tools
+ (add options to gnunet-search / gnunet-download to limit search to local peer)
+* STATISTICS:
+ - should use BIO instead of mmap