From a451a2586010ed439a381e2621f29b7c3e197fcf Mon Sep 17 00:00:00 2001 From: Christian Grothoff Date: Thu, 18 Mar 2010 14:21:35 +0000 Subject: [PATCH] update --- TODO | 214 +++++++++---------- src/transport/transport_api_address_lookup.c | 20 +- 2 files changed, 120 insertions(+), 114 deletions(-) diff --git a/TODO b/TODO index 0d2725793..adc41743d 100644 --- a/TODO +++ b/TODO @@ -22,104 +22,21 @@ away), in order in which they will likely be done: transport cost changes - disconnect notifications from TCP not always as nice as we might want them to be (but how can we do this cleanly given that UDP is connectionless?) - - hostlist testcase shows that (randomly) we jump to an address that - seems to be a plugin function *after* the plugin has been unloaded - (directly from scheduler); check that *all* plugin tasks are cancelled - after plugin is unloaded! (need better testcases for that...) - implement transport API to pretty-print transport address + transport_api extension (API extension!) + service-transport extension (protocol extension) - - implement gnunet-transport (transport configurator / tester) - instantly filter addresses from *other* peers that are *equal* to our own address + port (i.e., localhost:2086). We no longer filter those for outgoing (helps with loopback testing and keeps the code clean), but we should filter strictly *impossible* incoming addresses! This is for efficiency, not correctness. - - UPnP-based IP detection - (Note: build library always, build service when libxml2/etc. are available) - - We currently are happy to take any address told to us in a WELCOME - to our set of addresses; we should have some minimal threshold-based - scheme, limiting both the total number of addresses that we accept - this way as well as requiring multiple confirmations; also, we - should possibly try to confirm that the given address works for - us ourselves (loopback-style) before adding it to the list - [SECURITY issue] - + we may be able to simplify WELCOME messages (no need to add - addresses there anymore, but may help to learn them there anyway...). - + we probably want some kind of voting/counting for learning IP addresses - (maybe including IP addresses in ads proportional to how often others - report them? we at least need some protection against >64k HELLOs!), - + provide a way to give the user a list of "learned" IP addresses and - a way to easily "veto" addresses off the list! - => If MiM attacker uses vetoed address, blacklist the specific IP for - the presumed neighbour! - [./transport/gnunet-service-transport.c:173]: (style) struct or union member 'TransportPlugin::rebuild' is never used - [./transport/plugin_transport_tcp.c:391]: (style) struct or union member 'Plugin::address_update_task' is never used -* FS: - - [./fs/gnunet-service-fs.c:208]: (style) struct or union member 'LocalGetContext::results_bf_size' is never used - - [./fs/gnunet-service-fs.c:501]: (style) struct or union member 'PendingRequest::used_pids_size' is never used - - [./fs/gnunet-service-fs.c:654]: (style) struct or union member 'ConnectedPeer::last_client_replies' is never used - - [./fs/gnunet-service-fs.c:669]: (style) struct or union member 'ConnectedPeer::avg_delay' is never used - - [./fs/gnunet-service-fs.c:675]: (style) struct or union member 'ConnectedPeer::avg_priority' is never used - - [./fs/gnunet-service-fs.c:688]: (style) struct or union member 'ConnectedPeer::pending_requests' is never used - - [./fs/gnunet-service-fs.c:694]: (style) struct or union member 'ConnectedPeer::last_p2p_replies_woff' is never used - - [./fs/gnunet-service-fs.c:700]: (style) struct or union member 'ConnectedPeer::last_client_replies_woff' is never used * CORE: - SET_KEY is triggered every 2.5 minutes (after session is up); should just do PING (not SET_KEY + PING) - "Forcing disconnect of XXX due to inactivity" -- can happen every few MS!? (disconnect does not really succeed, or what?) -* TOPOLOGY: - - needs more testing (especially F2F topology) - - needs to re-try connecting after disconnect (currently, it - initially triggers a connection request, but if that connection - fails / goes down, it does not retry in a timely fashion; - cause seems to be the 'blacklist_after_attempt' being set to 1h, - which is rather long -- and should probably be adjusted based on - the number of connections / known peers) - - If the topology daemon crashes, peers that were put on the - blacklist with transport will never be removed from it (until - transport service dies); we should use the blacklist notification - API to learn about the exact set of blacklisted peers at all times - (FIXME: the transport_api implementation of blacklisting - also does not work nicely for this since it won't let us know about - disconnect-reconnect events and the implicit whitelisting - that might happen here; that's not so bad since we will - re-blacklist on pre-connect attempts anyway, so this is - a minor issue). - - the code uses the term 'blacklist' for both peers that are forbidden - to connect (i.e. F2F mode) as well as peers that we currently - won't try to actively connect to ourselves (since we just tried); - This is confusing. We need two distinct terms. - - move code to use hash table instead of linked list - - instead of periodically discarding blacklisted entries, - simply add task that is triggered at the right time (earlier free, - more balanced load) - - check if new HELLO learned is different from old HELLO - before resetting entire state! -* UTIL: - - trust: need *fast* way to check/update trust in peers - (async peerinfo would not be right) - - scheduler should change OS process priority based on task priority; - should make better use of task priorities in general - - only connect() sockets that are ready (select()) [Nils] - [On W32, we need to select after calling socket before - doing connect etc.] -* STATISTICS: - - synchronous/asynchronous API (& implementation) is not nice; - => provide notification-based API - - does not seem to work with timeouts (especially if service is not running) -* DATACACHE: - - add stats (# bytes available, # bytes used, # PUTs, # GETs, # GETs satisfied) -* ARM: - - need to get rid of synchronous API for service starts (cause all kinds of problems) - [=> eliminate for need to tell ARM about service starts most of the time!] [Safey] - - better tracking of which config changes actually need to cause process restarts by ARM. - - listen for requests to discover dependencies between services (and avoid - having to explicitly program start requests) - - better crash management (attach debugging support, capture and analyze - debug output, detect random vs. deterministic crashes) - - shutdown sequence? * HELLO: - need function to test "equivalency" of HELLOs; use in topology! * FS: @@ -127,17 +44,14 @@ away), in order in which they will likely be done: + active reply route caching design & implementation of service, gap extension! - gnunet-publish cannot be aborted using CTRL-C - - gnunet-publish segfaults if file does not exist - on some systems, keyword search does not find locally published content (need testcase of command-line tools!) - 2-peer download is still too slow (why?) -* Module features to implement: - advanced FS API parts + namespaces: fundamental namespace API + search: SBlocks, KSBlocks, probes, notify FS-service of known results + collection + location URIs (publish, search, download) - + persistence support (publish, unindex, search, download) + indexing: index-failure-cleanup + download: management/bounding of parallel downloads (for recursive dl.) + datastore reservation (publishing) @@ -147,8 +61,7 @@ away), in order in which they will likely be done: + gnunet-download (many options) + gnunet-directory (man page, options) + gnunet-pseudonym (all of it) - + gnunet-service-fs (remove failing on-demand blocks, stats, hot-path routing, load-based routing, nitpicks) - + datastore: do active migration support here? + + gnunet-service-fs (remove failing on-demand blocks, hot-path routing, load-based routing, nitpicks) - implement adv. FS testcases + getopt API + insert: sblocks, loc uris @@ -156,30 +69,20 @@ away), in order in which they will likely be done: + search: sblocks, skblocks, loc uris + namespaces + collection - - implement FS performance tests (needs TESTING) + - implement FS performance tests + insert + download + search + unindex - - non-anonymous FS service (needs DHT) - + basic DHT integration - + CS-DHT-functions (DHT-put of LOC) - + P2P-functions (DHT-get) - - setup (RC-pre0) - + default generation - + need to settle basic design; do we want to keep guile? - - testing (RC-pre0) - + modify configuration to allow controlling connections for non-local starts - + testbed creation with topology (needs working F2F topology) [Nate] - + testbed with churn [Nate] - + implement library for distributed testing [Nate] - + implement testcases for distributed testing [Nate] - + test basic peer re-configure [Nate] - + test topology creation [Nate] - + test churn generation [Nate] -* TESTING: - - consider changing API for peer-group termination to - call continuation when done + - [./fs/gnunet-service-fs.c:208]: (style) struct or union member 'LocalGetContext::results_bf_size' is never used + - [./fs/gnunet-service-fs.c:501]: (style) struct or union member 'PendingRequest::used_pids_size' is never used + - [./fs/gnunet-service-fs.c:654]: (style) struct or union member 'ConnectedPeer::last_client_replies' is never used + - [./fs/gnunet-service-fs.c:669]: (style) struct or union member 'ConnectedPeer::avg_delay' is never used + - [./fs/gnunet-service-fs.c:675]: (style) struct or union member 'ConnectedPeer::avg_priority' is never used + - [./fs/gnunet-service-fs.c:688]: (style) struct or union member 'ConnectedPeer::pending_requests' is never used + - [./fs/gnunet-service-fs.c:694]: (style) struct or union member 'ConnectedPeer::last_p2p_replies_woff' is never used + - [./fs/gnunet-service-fs.c:700]: (style) struct or union member 'ConnectedPeer::last_client_replies_woff' is never used + * WWW: - Get IPv6 hooked up - change DNS @@ -203,6 +106,53 @@ away), in order in which they will likely be done: * HOSTLIST: - 'server' uses 'GNUNET_PEERINFO_iterate', should probably switch to notification API (for more instant / up-to-date hostlists at lower cost) [OPTIMIZATION] +* STATISTICS: + - does not seem to work with timeouts (especially if service is not running) +* UTIL: + - trust: need *fast* way to check/update trust in peers + (async peerinfo would not be right) + - scheduler should change OS process priority based on task priority; + should make better use of task priorities in general + - only connect() sockets that are ready (select()) [Nils] + [On W32, we need to select after calling socket before + doing connect etc.] +* DATACACHE: + - add stats (# bytes available, # bytes used, # PUTs, # GETs, # GETs satisfied) +* FS: + - persistence support (publish, unindex, search, download) + - non-anonymous FS service (needs DHT) + + basic DHT integration + + CS-DHT-functions (DHT-put of LOC) + + P2P-functions (DHT-get) + - active migration support (in fs or in datastore?) +* TOPOLOGY: + - needs more testing (especially F2F topology) + - needs to re-try connecting after disconnect (currently, it + initially triggers a connection request, but if that connection + fails / goes down, it does not retry in a timely fashion; + cause seems to be the 'blacklist_after_attempt' being set to 1h, + which is rather long -- and should probably be adjusted based on + the number of connections / known peers) + - If the topology daemon crashes, peers that were put on the + blacklist with transport will never be removed from it (until + transport service dies); we should use the blacklist notification + API to learn about the exact set of blacklisted peers at all times + (FIXME: the transport_api implementation of blacklisting + also does not work nicely for this since it won't let us know about + disconnect-reconnect events and the implicit whitelisting + that might happen here; that's not so bad since we will + re-blacklist on pre-connect attempts anyway, so this is + a minor issue). + - the code uses the term 'blacklist' for both peers that are forbidden + to connect (i.e. F2F mode) as well as peers that we currently + won't try to actively connect to ourselves (since we just tried); + This is confusing. We need two distinct terms. + - move code to use hash table instead of linked list + - instead of periodically discarding blacklisted entries, + simply add task that is triggered at the right time (earlier free, + more balanced load) + - check if new HELLO learned is different from old HELLO + before resetting entire state! 0.9.0pre2: * Module features to implement: @@ -220,12 +170,33 @@ away), in order in which they will likely be done: + Fragmentation library + actual plugin - HTTP backend (RC-pre2) +* ARM: + - need to get rid of synchronous API for service starts (cause all kinds of problems) + [=> eliminate for need to tell ARM about service starts most of the time!] [Safey] + - better tracking of which config changes actually need to cause process restarts by ARM. + - listen for requests to discover dependencies between services (and avoid + having to explicitly program start requests) + - better crash management (attach debugging support, capture and analyze + debug output, detect random vs. deterministic crashes) + - shutdown sequence? + 0.9.0pre3: * GUIs to implement: - qt (RC-pre3) + see discussions @ FISL about integration with event loop! * Determine RC bugs and fix those! +* TESTING: + - modify configuration to allow controlling connections for non-local starts + - testbed creation with topology (needs working F2F topology) [Nate] + - testbed with churn [Nate] + - implement testcases for distributed testing [Nate] + - test basic peer re-configure [Nate] + - test topology creation [Nate] + - test churn generation [Nate] + - consider changing API for peer-group termination to + call continuation when done + 0.9.0: * Documentation @@ -253,14 +224,33 @@ away), in order in which they will likely be done: 0.9.x: -* Plugins to implement: +* TRANSPORT: - SMTP transport backend - HTTPS transport backend + improved HTTPS support in MHD + actual plugin -* Remove KBlocks in gnunet-unindex (see discussion with Kenneth Almquist on gnunet-devs in 9/2009) -* Allow checking of presence of search results and/or content via command-line tools - (add options to gnunet-search / gnunet-download to limit search to local peer) + - Implement method of learning our external addresses from + other peers; need some kind of threshold-based + scheme, limiting both the total number of addresses that we accept + this way as well as requiring multiple confirmations; also, we + should possibly try to confirm that the given address works for + us ourselves (loopback-style) before adding it to the list + + we may be able to simplify WELCOME messages (no need to add + addresses there anymore, but may help to learn them there anyway...). + + we probably want some kind of voting/counting for learning IP addresses + (maybe including IP addresses in ads proportional to how often others + report them? we at least need some protection against >64k HELLOs!), + + provide a way to give the user a list of "learned" IP addresses and + a way to easily "veto" addresses off the list! + => If MiM attacker uses vetoed address, blacklist the specific IP for + the presumed neighbour! + - implement gnunet-transport (transport configurator / tester) + - UPnP-based IP detection + (Note: build library always, build service when libxml2/etc. are available) +* FS: + - Remove KBlocks in gnunet-unindex (see discussion with Kenneth Almquist on gnunet-devs in 9/2009) + - Allow checking of presence of search results and/or content via command-line tools + (add options to gnunet-search / gnunet-download to limit search to local peer) * STATISTICS: - should use BIO instead of mmap * PEERINFO: diff --git a/src/transport/transport_api_address_lookup.c b/src/transport/transport_api_address_lookup.c index bf0b37442..4293b18f8 100644 --- a/src/transport/transport_api_address_lookup.c +++ b/src/transport/transport_api_address_lookup.c @@ -70,10 +70,26 @@ address_response_processor (void *cls, const struct GNUNET_MessageHeader *msg) } } +/** + * Convert a binary address into a human readable address. + * + * @param sched scheduler to use + * @param cfg configuration to use + * @param address address to convert (binary format) + * @param addressLen number of bytes in address + * @param numeric should (IP) addresses be displayed in numeric form + * (otherwise do reverse DNS lookup) + * @param nameTrans name of the transport to which the address belongs + * @param timeout how long is the lookup allowed to take at most + * @param aluc function to call with the results + * @param aluc_cls closure for aluc + */ void GNUNET_TRANSPORT_address_lookup (struct GNUNET_SCHEDULER_Handle *sched, - const struct GNUNET_CONFIGURATION_Handle - *cfg, const char *address, size_t addressLen, + const struct GNUNET_CONFIGURATION_Handle *cfg, + const char *address, + size_t addressLen, + int numeric, const char *nameTrans, struct GNUNET_TIME_Relative timeout, GNUNET_TRANSPORT_AddressLookUpCallback aluc, -- 2.25.1