X-Git-Url: https://git.librecmc.org/?a=blobdiff_plain;ds=sidebyside;f=TODO;h=6c9e18429ffad3247a84f8f2a2fb955545773f49;hb=b0d073db4d1f0006ec1d6a7152c7543c80cea268;hp=1c06ef57e25d58b0d84c48967ed8b7646d5c91a9;hpb=384206a77cc1d53c536faeae8bc498895467194a;p=oweals%2Fgnunet.git diff --git a/TODO b/TODO index 1c06ef57e..6c9e18429 100644 --- a/TODO +++ b/TODO @@ -1,138 +1,146 @@ -Implementable right now (but not necessarily important), with caveats -(unavailable components that will limit what can be implemented right -away), in order in which they will likely be done: -* TESTING-TESTBED [Nate] -* SETUP -* DV [Nate] -* TBENCH -* TRACEKIT -* FRAGMENTATION [Ji Lu] -* HTTP transport -* MySQL / Postgres plugins (datastore, datacache) -* UPNP [Milan] -* ARM [Safey] - -Urgent items (before announcing ng.gnunet.org): -* FS (basic anonymous FS only) - - test multi-peer search/download -* new webpage - - run peer => have a 0.9.x hostlist - - improve basic documentation (configure, dependencies, what works, etc.) - - write minimal welcome/introduction to 0.9.x-development/webpage/status -=> Deploy(able) development network - - -0.9.0pre0: -* UTIL: - - load: need way to determine network load (up/down) +0.9.0pre1: +* PEERINFO: - trust: need *fast* way to check/update trust in peers - (async peerinfo would not be right) -* TRANSPORT: - - disconnect notifications from TCP not always as nice as we might want - them to be (but how can we do this cleanly given that UDP is connectionless?) - - TCP not used bi-directionally (especially important also for PONG!) + (async peerinfo would not be right; certainly not with the + current API) +* UTIL: + - scheduler should change OS process priority based on task priority; + should make better use of task priorities in general + - only connect() sockets that are ready (select()) [Nils] + [On W32, we need to select after calling socket before + doing connect etc.] +* SETUP: + - design & implement new setup tool +* TBENCH: [MW] + - good to have for transport/DV evaluation! +* DV: [Nate] + - write DV API (need to move declarations from dv_api.c to gnunet_dv_service.h!) + - implement DV service + - implement DV library (looks done) + - implement DV transport plugin + - implement testcases + - implement performance tests * STATISTICS: - - synchronous/asynchronous API (& implementation) is not nice; - => provide notification-based API - does not seem to work with timeouts (especially if service is not running) -* ARM: - - need to get rid of synchronous API for service starts (cause all kinds of problems) - [=> eliminate for need to tell ARM about service starts most of the time!] [Safey] -* HELLO: - - need function to test "equivalency" of HELLOs; use in topology! * TOPOLOGY: - needs more testing (especially F2F topology) -* CORE: - - needs more testing -* Module features to implement: - - advanced FS API parts - + namespaces: fundamental namespace API - + search: SBlocks, KSBlocks, probes, notify FS-service of known results - + collection - + location URIs (publish, search, download) - + persistence support (publish, unindex, search, download) - + indexing: index-failure-cleanup - + download: management/bounding of parallel downloads (for recursive dl.) - + datastore reservation (publishing) - + gnunet-publish (URI args) - + gnunet-download (directory) - + gnunet-search (options, incl. namespace search) - + gnunet-download (many options) - + gnunet-directory (man page, options) - + gnunet-pseudonym (all of it) - + gnunet-service-fs (remove failing on-demand blocks, stats, hot-path routing, load-based routing, nitpicks) - + datastore: do active migration support here? - - implement adv. FS testcases - + getopt API - + insert: sblocks, loc uris - + download: loc uris - + search: sblocks, skblocks, loc uris - + namespaces - + collection - - implement FS performance tests (needs TESTING) + - needs to re-try connecting after disconnect (currently, it + initially triggers a connection request, but if that connection + fails / goes down, it does not retry in a timely fashion; + cause seems to be the 'blacklist_after_attempt' being set to 1h, + which is rather long -- and should probably be adjusted based on + the number of connections / known peers) + - If the topology daemon crashes, peers that were put on the + blacklist with transport will never be removed from it (until + transport service dies); we should use the blacklist notification + API to learn about the exact set of blacklisted peers at all times + (FIXME: the transport_api implementation of blacklisting + also does not work nicely for this since it won't let us know about + disconnect-reconnect events and the implicit whitelisting + that might happen here; that's not so bad since we will + re-blacklist on pre-connect attempts anyway, so this is + a minor issue; OTOH, we might want to be more explicit about + allowing/forbidding connects on pre-connect to avoid + entering connect attempts to just be blacklisted shortly afterwards). + - the code uses the term 'blacklist' for both peers that are forbidden + to connect (i.e. F2F mode) as well as peers that we currently + won't try to actively connect to ourselves (since we just tried); + This is confusing. We need two distinct terms (greylist?). + - move code to use hash table instead of linked list + - instead of periodically discarding blacklisted entries, + simply add task that is triggered at the right time (earlier free, + more balanced load) + - check if new HELLO learned is different from old HELLO + before resetting entire state! +* FS: [CG] + - support recursive download even if filename is NULL and we hence + do not generate files on disk (use temp_filename) + - bound parallelism (# fs downloads) + - distinguish in performance tracking and event signalling between + downloads that are actually running and those that are merely in the queue + - persistence support (publish, unindex, search, download) + - active migration support (in fs or in datastore or new daemon?) + - gnunet-service-fs (hot-path routing, load-based routing, nitpicks) + - [gnunet-service-fs.c:208]: member 'LocalGetContext::results_bf_size' is never used + - [gnunet-service-fs.c:501]: member 'PendingRequest::used_pids_size' is never used + - [gnunet-service-fs.c:654]: member 'ConnectedPeer::last_client_replies' is never used + - [gnunet-service-fs.c:669]: member 'ConnectedPeer::avg_delay' is never used + - [gnunet-service-fs.c:675]: member 'ConnectedPeer::avg_priority' is never used + - [gnunet-service-fs.c:688]: member 'ConnectedPeer::pending_requests' is never used + - [gnunet-service-fs.c:694]: member 'ConnectedPeer::last_p2p_replies_woff' is never used + - [gnunet-service-fs.c:700]: member 'ConnectedPeer::last_client_replies_woff' is never used + - GAP improvements: + + active reply route caching design & implementation of service; gap extension! +* DATASTORE: + - API lacks cancellation methods (needed? or is disconnect enough?); + may also want to integrate request queuing here instead of gnunet-service-fs_drq.c +* GNUNET-GTK: + - use g_main_context_set_poll_func to integrate GTK with GNUnet Scheduler!? (YUCK!) + - OR: add scheduler API to enable integration with GTK main loop instead of doing our own select + - use g_main_context_pending, g_main_context_query / g_main_context_check / g_main_context_dispatch + and NEVER g_main_loop_run (can this be done? might be the clean way to do this! But how + to integrate this with "tak_main"? Docu says: + "It's OK to use the GLib main loop directly instead of gtk_main(), though it involves + slightly more typing. See GMainLoop in the GLib documentation." + => so maybe it "just works"? + +0.9.0pre2: +* TRACEKIT: [MW] + - good to have for DV/DHT evaluation! +* DHT: [Nate] + - implement DHT service + - implement DHT library + - implement testcases + - implement performance tests +* TRANSPORT: + - HTTP backend [MW] +* ARM: [Safey] + - need to get rid of synchronous API for service starts (cause all kinds of problems) + [=> eliminate for need to tell ARM about service starts most of the time!] [Safey] + - better tracking of which config changes actually need to cause process restarts by ARM. + - listen for requests to discover dependencies between services (and avoid + having to explicitly program start requests) + - better crash management (attach debugging support, capture and analyze + debug output, detect random vs. deterministic crashes) + - shutdown sequence? +* FS: [CG] + - datastore reservation (publishing) + - search: availability probes + - location URIs (publish, search, download) + - non-anonymous FS service (needs DHT) + + DHT integration for search + + CS-DHT-functions (DHT-put of LOC) + + P2P-functions (DHT-get) + - collection API & tests + + gnunet-pseudonym (collection support) + - implement FS performance tests + insert + download + search + unindex - - non-anonymous FS service (needs DHT) - + basic DHT integration - + CS-DHT-functions (DHT-put of LOC) - + P2P-functions (DHT-get) - - setup (RC-pre0) - + default generation - + need to settle basic design; do we want to keep guile? - - testing (RC-pre0) - + modify configuration to allow controlling connections for non-local starts - + testbed creation with topology (needs working F2F topology) [Nate] - + testbed with churn [Nate] - + implement library for distributed testing [Nate] - + implement testcases for distributed testing [Nate] - + test basic peer re-configure [Nate] - + test topology creation [Nate] - + test churn generation [Nate] - -0.9.0pre1: -* Module features to implement: - - tbench (RC-pre1) - + good to have for DV evaluation! - - DV (RC-pre1) - + write DV API - + implement DV service [Nate & CG] - + implement DV library [Nate] - + implement DV transport plugin [Nate & CG] - + implement testcases [Nate] - + implement performance tests [Nate] -* GUIs to implement: - - gtk (RC-pre1) - + how to integrate scheduler with GTK event loop! - -0.9.0pre2: -* Module features to implement: - - tracekit (RC-pre2) - + good to have for DV/DHT evaluation! - - DHT (RC-pre2) - + implement DHT service (needs DV, DATACACHE) - + implement DHT library - + implement testcases - + implement performance tests -* GUIs to implement: - - fuse (RC-pre2) -* Plugins to implement: - - UDP backend (RC-pre2) - + Fragmentation library - + actual plugin - - HTTP backend (RC-pre2) 0.9.0pre3: -* GUIs to implement: - - qt (RC-pre3) - + see discussions @ FISL about integration with event loop! * Determine RC bugs and fix those! +* TESTING: [Nate] + - modify configuration to allow controlling connections for non-local starts + - testbed creation with topology (needs working F2F topology) + - testbed with churn + - implement testcases for distributed testing + - test basic peer re-configure + - test topology creation + - test churn generation + - consider changing API for peer-group termination to + call continuation when done +* NAT/UPNP: [MW] + - finalize API design + - code clean up + - testing + - integration with transport service +* MYSQL database backends: [CG] + - datacache + - datastore 0.9.0: -* Documentation - - update man pages - - update webpage documentation * new webpage: - expand bibliography - convert documentation pages to books @@ -142,31 +150,80 @@ Urgent items (before announcing ng.gnunet.org): enable developers to publish TGZs nicely - port "contact" page - add content type for "todo" items? - - DNS activation -* Plugins to implement: - - MySQL database backends - + datacache - + datastore - - Postgres database backends - + datacache - + datastore - - vpn +* POSTGRES database backends: [CG] + - datacache + - datastore * Determine RC bugs and fix those! - 0.9.x: -* Plugins to implement: +* TRANSPORT: [MW] - SMTP transport backend - HTTPS transport backend + improved HTTPS support in MHD + actual plugin -* Remove KBlocks in gnunet-unindex (see discussion with Kenneth Almquist on gnunet-devs in 9/2009) -* Allow checking of presence of search results and/or content via command-line tools - (add options to gnunet-search / gnunet-download to limit search to local peer) -* STATISTICS: - - should use BIO instead of mmap -* PEERINFO: + - Implement method of learning our external addresses from + other peers; need some kind of threshold-based + scheme, limiting both the total number of addresses that we accept + this way as well as requiring multiple confirmations; also, we + should possibly try to confirm that the given address works for + us ourselves (loopback-style) before adding it to the list + + we may be able to simplify WELCOME messages (no need to add + addresses there anymore, but may help to learn them there anyway...). + + we probably want some kind of voting/counting for learning IP addresses + (maybe including IP addresses in ads proportional to how often others + report them? we at least need some protection against >64k HELLOs!), + + provide a way to give the user a list of "learned" IP addresses and + a way to easily "veto" addresses off the list! + => If MiM attacker uses vetoed address, blacklist the specific IP for + the presumed neighbour! + - implement gnunet-transport (transport configurator / tester) + - UPnP-based IP detection + (Note: build library always, build service when libxml2/etc. are available) +* FS: [CG] + - Remove KBlocks in gnunet-unindex (see discussion with Kenneth Almquist on gnunet-devs in 9/2009) + - Allow checking of presence of search results and/or content via command-line tools + (add options to gnunet-search / gnunet-download to limit search to local peer) +* PEERINFO: [CG] - expire 'ancient' HELLOs (those without valid addresses AND that we have not 'used' (for their public keys) in a while; need a way to track actual 'use') - make sue we also trigger notifications whenever HELLOs expire +* VPN + + + +Optimizations: +* TCP: + - should use hash map to look up sessions +* PEERINFO: + - api creates many, many short-lived TCP connections; either some + clients should use the API differently or we need to change the + API to enable re-use of connections to the service +* STATISTICS: + - should use BIO instead of mmap +* TRANSPORT: + - need to periodically probe latency/transport cost changes & possibly switch transport + - instantly filter addresses from *other* peers that + are *equal* to our own address + port (i.e., localhost:2086). We + no longer filter those for outgoing (helps with loopback testing + and keeps the code clean), but we should filter strictly *impossible* + incoming addresses! This is for efficiency, not correctness. + - should use hash map to look up Neighbours +* HOSTLIST: + - 'server' uses 'GNUNET_PEERINFO_iterate', should probably switch to notification API + (for more instant / up-to-date hostlists at lower cost) [OPTIMIZATION] +* DATASTORE (?): + - check for duplicates on insertion (currently, same content is frequently + stored again [seen with KBLOCKS and SBLOCKS]!) + +Minor features: +* TCP: + - repeatedly resolve hostname and look up interfaces to determine our own IP + - [./transport/plugin_transport_tcp.c:391]: (style) struct or union member 'Plugin::address_update_task' is never used (related to issue above) +* TRANSPORT: + - [./transport/gnunet-service-transport.c:173]: (style) struct or union member 'TransportPlugin::rebuild' is never used (related to TCP not refreshing external addresses?) +* DATACACHE: + - add stats (# bytes available, # bytes used, # PUTs, # GETs, # GETs satisfied) +* FS: + - support inline data in directories for recursive file downloads (fs_download) +