0.9.0pre2:
* BIG CORE REFACTORING:
- - fix transport service API (ATS!)
- - fix transport plugin API (ATS!)
- - actually transmit ATS data through core API
- - fix FS 'latency' ATS function
- - fix DV
+ - fix transport plugin API (ATS!) [mw]
+ - fix DV [nate]
* Integration test:
- test bootstrapping via hostlist + transport/core/dht connect
-
-0.9.0pre3:
-* Determine RC bugs and fix those (release should have no known real bugs)
-* ARM: [CG/Safey]
- - start_forwarding should not use 'listen_info->service_addr' for connecting since that one can be 0.0.0.0;
- if it is, we need to use (IPv4 & IPv6) loopback [LRN]
- - better tracking of which config changes actually need to cause process restarts by ARM.
- - handle gnunet-arm -k in combination with auto-start magic (what is the right thing here?)
- - discover dependencies between services
* CORE:
- - provide 'reliable' P2P transmission API and queueing infrastructure
- Jun 27 11:51:54 core-7670 ERROR Assertion failed at gnunet-service-core.c:3616.
(transport notified us that we connected to ourselves!!!; can we still reproduce this?)
-* DHT: [Nate]
- - track paths content travels (PUT, reply-to-get) in messages,
- pass to client (client API & protocol already support this!)
-* DATASTORE:
- - GNUNET_DATASTORE_cancel method not tested
-* TESTING: [Nate]
- - test basic peer re-configure
-* TOPOLOGY: [Nate]
- - needs more testing (especially F2F topology) & transport blacklisting
-* TRANSPORT-TCP [MW]:
- - should use hash map to look up sessions
-* TRANSPORT:
+ => Also, we may want to issue a 'connect to ourselves' automatically on start of
+ core_api AND allow messages to self [CG/phillip]
+
+0.9.0pre3:
+* TRANSPORT: [MW]
- [#1585, #1583, #1582] transport-level disconnect (i.e. TCP) does not cause core-level
disconnect in a timely fashion (waits for connection timeout);
need to figure a way to make it near-instant in those cases
likely good enough until we get ATS going; still should be tested...
=> "peers connected (transport)" now instantly goes to ZERO (core statistic),
but "established sessions" stays up...
-* NAT/UPNP: [Milan]
- - finalize API design
+* NAT/UPNP: [Milan / Ayush / MW]
- [#1609] code clean up
- testing
- - integration with transport service
- - also do UPnP-based (external) IP detection
- (Note: build library always, build UPnP service when dependencies like libxml2 are available)
+ - integration with transport service:
+ + test TCP
+ + implement UDP, HTTP/HTTPS
+* DHT: [Nate]
+ - track paths content travels (PUT, reply-to-get) in messages,
+ pass to client (client API & protocol already support this!)
* FS: [CG]
- - library:
- + reconstruct IBLOCKS from DBLOCKS if possible (during download; see FIXME in fs_download)
- + add support for pushing "already seen" search results to FS service for bloomfilter
- + use different 'priority' for probe downloads vs. normal downloads
- implement multi-peer FS performance tests
+ insert
+ download
+ search
- + unindex
- - implement anonymity level > 1
- - re-implement gnunet-auto-share
-* GNUNET-GTK:
- - finish publish dialog details:
- + normalize keywords (edit subdialog)
- - implement download by URI dialog; figure out where to display those downloads!
- - figure out where in the GUI we should show active uploads/unindex operations and allow aborts
- - implement unindex operation (use dialog with all indexed files for selection)
- - events:
+* GNUNET-GTK: [CG]
+ - figure out where in the GUI we should show active upload operations and allow aborts
+ - handle events:
+ search error
+ publish error
- + unindex error
-* MONKEY: [Safey]
- - better crash management (attach debugging support, capture and analyze
- debug output, detect random vs. deterministic crashes)
- - '-m EMAIL' option for alternative e-mail TO
- - '-f FILENAME' option to write report to file instead of e-mail (for testing!)
+* Determine RC bugs and fix those (release should have no known real bugs)
0.9.0:
-* Determine RC bugs and fix those (release should have no known real bugs)
-* UTIL:
- - only connect() sockets that are ready (select()) [Nils]
- [On W32, we need to select after calling socket before doing connect etc.]
-* new webpage: [BL]
- - convert documentation pages to books
+* new webpage:
- update books (especially for developers)
- make a NICE download page and figure out how to enable developers to publish TGZs nicely
- port "contact" page
- - add content type for "todo" items
-* TBENCH: [MW]
- - good to have for transport/DV evaluation!
-* TRACEKIT: [MW]
- - good to have for DHT evaluation!
-* DHT: [Nate]
- - performance tests
-* BLOCK:
- - more testing (KBlock, SBlock, NBlock)
* FS migration:
- - exclude content that will "soon" expire from migration?
- - exclude content with zero-anonymity from gap migration?
+ - exclude content that will "soon" expire from migration
- make migration data rate & datastore IO-rate configurable
- exclude certain peers as targets (based on hash values) in each
iteration => same peer can only be picked every n-th iteration
for the same content => fewer duplicate sending!
-
+* big code review
+* Determine RC bugs and fix those (release should have no known real bugs)
0.9.1:
* TRANSPORT: [MW]
- WiFi transport backend [DB]
- - implement gnunet-transport (transport configurator / tester)
- Implement method of learning our external addresses from
other peers; need some kind of threshold-based
scheme, limiting both the total number of addresses that we accept
=> If MiM attacker uses vetoed address, blacklist the specific IP for
the presumed neighbour!
- need to periodically probe latency/transport cost changes & possibly switch transport
- - should use hash map to look up Neighbours (service AND plugins!)
-* DV: [Nate]
+* DV: [Nate?]
- proper bandwidth allocation
- performance tests
* PEERINFO:
(theoretically reduces overhead; bounds message queue size)
- merge multiple iteration requests over "all" peers in the queue
(theoretically reduces overhead; bounds messgae queue size)
-* STATISTICS: [CG]
+* STATISTICS:
- should use BIO instead of mmap
* FS: [CG]
- Remove KBlocks in gnunet-unindex (see discussion with Kenneth Almquist on gnunet-devs in 9/2009)
- use different queue prioritization for probe-downloads vs. normal downloads
+ - re-implement gnunet-auto-share
* UTIL: [CG]
- allow limiting UNIX socket access by UID/GID
* GNUNET-GTK: [CG]
- right-clicking on NS list in search dialog should open menu that allows
* viewing full meta data
* deletion of namespace info
+ - implement unindex operation (use dialog with all indexed files for selection)
+ - finish publish dialog details:
+ + normalize keywords (edit subdialog)
+ - implement download by URI dialog; figure out where to display those downloads!
+* ARM: [CG/Safey]
+ - better tracking of which config changes actually need to cause process restarts by ARM.
+ - handle gnunet-arm -k in combination with auto-start magic (what is the right thing here?)
+ - discover dependencies between services
+* MONKEY: [Safey]
+ - better crash management (attach debugging support, capture and analyze
+ debug output, detect random vs. deterministic crashes)
+ - '-m EMAIL' option for alternative e-mail TO
+ - '-f FILENAME' option to write report to file instead of e-mail (for testing!)
0.9.2:
* PEERINFO: [NN]
- testcase would be nice
- generic block support for DHT
* STATISTICS:
- - test notification-based statistics API [LT]
+ - test notification-based statistics API
- implement statistics GUI (=> start from gnunet-gtk by button!)
* PEERINFO: [NN]
- move peerinfo to new GUI (=> start from gnunet-gtk by button!)
- improved batching
- resource limit integration with ATS
* VPN [PT]
- - DNS hijacking
- - DNS exit
- TCP entry/exit
- - UDP entry/exit
- internal services
- integration with DHT routing
- optimized routes (beyond DHT/DV)
GNUNET_i2s (&n->peer));
#endif
size = sizeof (struct PeerStatusNotifyMessage) +
- (n->ats_count+1) * sizeof (struct GNUNET_TRANSPORT_ATS_Information);
+ n->ats_count * sizeof (struct GNUNET_TRANSPORT_ATS_Information);
if (size >= GNUNET_SERVER_MAX_MESSAGE_SIZE)
{
GNUNET_break (0);
n->ats_count,
0);
size = sizeof (struct PeerStatusNotifyMessage) +
- (n->ats_count+1) * sizeof (struct GNUNET_TRANSPORT_ATS_Information);
+ n->ats_count * sizeof (struct GNUNET_TRANSPORT_ATS_Information);
}
psnm = (struct PeerStatusNotifyMessage*) buf;
psnm->header.size = htons (size);
n->ats_count,
0);
size = sizeof (struct PeerStatusNotifyMessage) +
- (n->ats_count) * sizeof (struct GNUNET_TRANSPORT_ATS_Information);
+ n->ats_count * sizeof (struct GNUNET_TRANSPORT_ATS_Information);
}
cnm = (struct ConnectNotifyMessage*) buf;
cnm->header.size = htons (size);
n->ats_count,
0);
size = sizeof (struct PeerStatusNotifyMessage) +
- (n->ats_count) * sizeof (struct GNUNET_TRANSPORT_ATS_Information);
+ n->ats_count * sizeof (struct GNUNET_TRANSPORT_ATS_Information);
}
cnm = (struct ConnectNotifyMessage*) buf;
cnm->header.size = htons (size);
(GNUNET_OK !=
GNUNET_BIO_write_int32 (wh, (uint32_t) dc->has_finished)) ||
(GNUNET_OK !=
- GNUNET_BIO_write_int32 (wh, num_pending)) )
+ GNUNET_BIO_write_int32 (wh, num_pending)) ||
+ (GNUNET_OK !=
+ GNUNET_BIO_write_int32 (wh, dc->start_task != GNUNET_SCHEDULER_NO_TASK)) )
{
GNUNET_break (0);
goto cleanup;
uint32_t options;
uint32_t status;
uint32_t num_pending;
+ int32_t start_pending;
uris = NULL;
emsg = NULL;
(GNUNET_OK !=
GNUNET_BIO_read_int32 (rh, &status)) ||
(GNUNET_OK !=
- GNUNET_BIO_read_int32 (rh, &num_pending)) )
+ GNUNET_BIO_read_int32 (rh, &num_pending)) ||
+ (GNUNET_OK !=
+ GNUNET_BIO_read_int32 (rh, &start_pending)) )
{
GNUNET_break (0);
goto cleanup;
signal_download_resume (dc);
}
GNUNET_free (uris);
+ if (start_pending)
+ dc->start_task
+ = GNUNET_SCHEDULER_add_now (&GNUNET_FS_download_start_task_, dc);
return;
cleanup:
GNUNET_free_non_null (uris);
-
/*
This file is part of GNUnet.
(C) 2003, 2004, 2005, 2006, 2007, 2008, 2009 Christian Grothoff (and other contributing authors)
struct GNUNET_FS_DownloadContext *dc);
+/**
+ * Task that creates the initial (top-level) download
+ * request for the file.
+ *
+ * @param cls the 'struct GNUNET_FS_DownloadContext'
+ * @param tc scheduler context
+ */
+void
+GNUNET_FS_download_start_task_ (void *cls,
+ const struct GNUNET_SCHEDULER_TaskContext *tc);
+
+
/**
* Fill in all of the generic fields for
* an unindex event and call the callback.
* when the search is being stopped (if not
* GNUNET_SCHEDULER_NO_TASK). Used for the task that adds some
* artificial delay when trying to reconnect to the FS service.
- */
+o */
GNUNET_SCHEDULER_TaskIdentifier task;
+
+ /**
+ * How many of the entries in the search request
+ * map have been passed to the service so far?
+ */
+ unsigned int search_request_map_offset;
+
+ /**
+ * How many of the keywords in the KSK
+ * map have been passed to the service so far?
+ */
+ unsigned int keyword_offset;
/**
* Anonymity level for the search.
*/
GNUNET_SCHEDULER_TaskIdentifier task;
+ /**
+ * Task used to start the download.
+ */
+ GNUNET_SCHEDULER_TaskIdentifier start_task;
+
/**
* What was the size of the file on disk that we're downloading
* before we started? Used to detect if there is a point in
* data from the meta data yet?
*/
int tried_full_data;
+
+ /**
+ * Have we tried to reconstruct an IBLOCK from disk
+ * and failed (and should hence not try again?)
+ */
+ int reconstruct_failed;
};
*
* TODO:
* - different priority for scheduling probe downloads?
- * - check if iblocks can be computed from existing blocks (can wait, hard)
*/
#include "platform.h"
#include "gnunet_constants.h"
#include "fs.h"
#include "fs_tree.h"
-#define DEBUG_DOWNLOAD GNUNET_YES
+#define DEBUG_DOWNLOAD GNUNET_NO
/**
* Determine if the given download (options and meta data) should cause
}
+
+/**
+ * Closure for 'reconstruct_cont' and 'reconstruct_cb'.
+ */
+struct ReconstructContext
+{
+ /**
+ * File handle open for the reconstruction.
+ */
+ struct GNUNET_DISK_FileHandle *fh;
+
+ /**
+ * the download context.
+ */
+ struct GNUNET_FS_DownloadContext *dc;
+
+ /**
+ * Tree encoder used for the reconstruction.
+ */
+ struct GNUNET_FS_TreeEncoder *te;
+
+ /**
+ * CHK of block we are trying to reconstruct.
+ */
+ struct ContentHashKey chk;
+
+ /**
+ * Request that was generated.
+ */
+ struct DownloadRequest *sm;
+
+ /**
+ * Offset of block we are trying to reconstruct.
+ */
+ uint64_t offset;
+
+ /**
+ * Depth of block we are trying to reconstruct.
+ */
+ unsigned int depth;
+
+};
+
+
+/**
+ * Continuation after a possible attempt to reconstruct
+ * the current IBlock from the existing file.
+ *
+ * @param cls the 'struct ReconstructContext'
+ * @param tc scheduler context
+ */
+static void
+reconstruct_cont (void *cls,
+ const struct GNUNET_SCHEDULER_TaskContext *tc)
+{
+ struct ReconstructContext *rcc = cls;
+
+ if (rcc->te != NULL)
+ GNUNET_FS_tree_encoder_finish (rcc->te, NULL, NULL);
+ rcc->dc->reconstruct_failed = GNUNET_YES;
+ if (rcc->fh != NULL)
+ GNUNET_break (GNUNET_OK == GNUNET_DISK_file_close (rcc->fh));
+ if ( (rcc->dc->th == NULL) &&
+ (rcc->dc->client != NULL) )
+ {
+#if DEBUG_DOWNLOAD
+ GNUNET_log (GNUNET_ERROR_TYPE_DEBUG,
+ "Asking for transmission to FS service\n");
+#endif
+ rcc->dc->th = GNUNET_CLIENT_notify_transmit_ready (rcc->dc->client,
+ sizeof (struct SearchMessage),
+ GNUNET_CONSTANTS_SERVICE_TIMEOUT,
+ GNUNET_NO,
+ &transmit_download_request,
+ rcc->dc);
+ }
+ else
+ {
+#if DEBUG_DOWNLOAD
+ GNUNET_log (GNUNET_ERROR_TYPE_DEBUG,
+ "Transmission request not issued (%p %p)\n",
+ rcc->dc->th,
+ rcc->dc->client);
+#endif
+ }
+ GNUNET_free (rcc);
+}
+
+
+static void
+get_next_block (void *cls,
+ const struct GNUNET_SCHEDULER_TaskContext *tc)
+{
+ struct ReconstructContext *rcc = cls;
+ GNUNET_FS_tree_encoder_next (rcc->te);
+}
+
+
+/**
+ * Function called asking for the current (encoded)
+ * block to be processed. After processing the
+ * client should either call "GNUNET_FS_tree_encode_next"
+ * or (on error) "GNUNET_FS_tree_encode_finish".
+ *
+ * This function checks if the content on disk matches
+ * the expected content based on the URI.
+ *
+ * @param cls closure
+ * @param query the query for the block (key for lookup in the datastore)
+ * @param offset offset of the block
+ * @param type type of the block (IBLOCK or DBLOCK)
+ * @param block the (encrypted) block
+ * @param block_size size of block (in bytes)
+ */
+static void
+reconstruct_cb (void *cls,
+ const GNUNET_HashCode *query,
+ uint64_t offset,
+ unsigned int depth,
+ enum GNUNET_BLOCK_Type type,
+ const void *block,
+ uint16_t block_size)
+{
+ struct ReconstructContext *rcc = cls;
+ struct ProcessResultClosure prc;
+ struct GNUNET_FS_TreeEncoder *te;
+ uint64_t off;
+ uint64_t boff;
+ uint64_t roff;
+ unsigned int i;
+
+ roff = offset / DBLOCK_SIZE;
+ for (i=rcc->dc->treedepth;i>depth;i--)
+ roff /= CHK_PER_INODE;
+ boff = roff * DBLOCK_SIZE;
+ for (i=rcc->dc->treedepth;i>depth;i--)
+ boff *= CHK_PER_INODE;
+ /* convert reading offset into IBLOCKs on-disk offset */
+ off = compute_disk_offset (GNUNET_FS_uri_chk_get_file_size (rcc->dc->uri),
+ boff,
+ depth,
+ rcc->dc->treedepth);
+ if ( (off == rcc->offset) &&
+ (depth == rcc->depth) &&
+ (0 == memcmp (query,
+ &rcc->chk.query,
+ sizeof (GNUNET_HashCode))) )
+ {
+ /* already got it! */
+ GNUNET_log (GNUNET_ERROR_TYPE_DEBUG,
+ _("Block reconstruction at offset %llu and depth %u successful\n"),
+ (unsigned long long) offset,
+ depth);
+ prc.dc = rcc->dc;
+ prc.data = block;
+ prc.size = block_size;
+ prc.type = type;
+ prc.query = rcc->chk.query;
+ prc.do_store = GNUNET_NO;
+ process_result_with_request (&prc,
+ &rcc->chk.key,
+ rcc->sm);
+ te = rcc->te;
+ rcc->te = NULL;
+ GNUNET_FS_tree_encoder_finish (te, NULL, NULL);
+ GNUNET_free (rcc);
+ return;
+ }
+ GNUNET_SCHEDULER_add_continuation (&get_next_block,
+ rcc,
+ GNUNET_SCHEDULER_REASON_PREREQ_DONE);
+}
+
+
+/**
+ * Function called by the tree encoder to obtain
+ * a block of plaintext data (for the lowest level
+ * of the tree).
+ *
+ * @param cls our 'struct ReconstructContext'
+ * @param offset identifies which block to get
+ * @param max (maximum) number of bytes to get; returning
+ * fewer will also cause errors
+ * @param buf where to copy the plaintext buffer
+ * @param emsg location to store an error message (on error)
+ * @return number of bytes copied to buf, 0 on error
+ */
+static size_t
+fh_reader (void *cls,
+ uint64_t offset,
+ size_t max,
+ void *buf,
+ char **emsg)
+{
+ struct ReconstructContext *rcc = cls;
+ struct GNUNET_DISK_FileHandle *fh = rcc->fh;
+ ssize_t ret;
+
+ *emsg = NULL;
+ if (offset !=
+ GNUNET_DISK_file_seek (fh,
+ offset,
+ GNUNET_DISK_SEEK_SET))
+ {
+ *emsg = GNUNET_strdup (strerror (errno));
+ return 0;
+ }
+ ret = GNUNET_DISK_file_read (fh, buf, max);
+ if (ret < 0)
+ {
+ *emsg = GNUNET_strdup (strerror (errno));
+ return 0;
+ }
+ return ret;
+}
+
+
/**
* Schedule the download of the specified block in the tree.
*
GNUNET_HashCode key;
struct MatchDataContext mdc;
struct GNUNET_DISK_FileHandle *fh;
+ struct ReconstructContext *rcc;
- total = GNUNET_ntohll (dc->uri->data.chk.file_length);
+ total = GNUNET_FS_uri_chk_get_file_size (dc->uri);
len = GNUNET_FS_tree_calculate_block_size (total,
dc->treedepth,
offset,
GNUNET_h2s (&chk->query));
#endif
fh = NULL;
- if ( (dc->old_file_size > off) &&
+ if ( ( (dc->old_file_size > off) ||
+ ( (depth < dc->treedepth) &&
+ (dc->reconstruct_failed == GNUNET_NO) ) ) &&
(dc->filename != NULL) )
fh = GNUNET_DISK_file_open (dc->filename,
GNUNET_DISK_OPEN_READ,
GNUNET_DISK_PERM_NONE);
if ( (fh != NULL) &&
+ (dc->old_file_size > off) &&
(off ==
GNUNET_DISK_file_seek (fh,
off,
return;
}
}
- if (fh != NULL)
- GNUNET_break (GNUNET_OK == GNUNET_DISK_file_close (fh));
- if (depth < dc->treedepth)
- {
- // FIXME: try if we could
- // reconstitute this IBLOCK
- // from the existing blocks on disk (can wait)
- // (read block(s), encode, compare with
- // query; if matches, simply return)
- }
-
- if ( (dc->th == NULL) &&
- (dc->client != NULL) )
- {
-#if DEBUG_DOWNLOAD
- GNUNET_log (GNUNET_ERROR_TYPE_DEBUG,
- "Asking for transmission to FS service\n");
-#endif
- dc->th = GNUNET_CLIENT_notify_transmit_ready (dc->client,
- sizeof (struct SearchMessage),
- GNUNET_CONSTANTS_SERVICE_TIMEOUT,
- GNUNET_NO,
- &transmit_download_request,
- dc);
- }
- else
+ rcc = GNUNET_malloc (sizeof (struct ReconstructContext));
+ rcc->fh = fh;
+ rcc->dc = dc;
+ rcc->sm = sm;
+ rcc->chk = *chk;
+ rcc->offset = off;
+ rcc->depth = depth;
+ if ( (depth < dc->treedepth) &&
+ (dc->reconstruct_failed == GNUNET_NO) &&
+ (fh != NULL) )
{
-#if DEBUG_DOWNLOAD
- GNUNET_log (GNUNET_ERROR_TYPE_DEBUG,
- "Transmission request not issued (%p %p)\n",
- dc->th,
- dc->client);
-#endif
-
+ rcc->te = GNUNET_FS_tree_encoder_create (dc->h,
+ dc->old_file_size,
+ rcc,
+ fh_reader,
+ &reconstruct_cb,
+ NULL,
+ &reconstruct_cont);
+ GNUNET_FS_tree_encoder_next (rcc->te);
+ return;
}
-
+ reconstruct_cont (rcc, NULL);
}
-
/**
* Suggest a filename based on given metadata.
*
}
+/**
+ * Task that creates the initial (top-level) download
+ * request for the file.
+ *
+ * @param cls the 'struct GNUNET_FS_DownloadContext'
+ * @param tc scheduler context
+ */
+void
+GNUNET_FS_download_start_task_ (void *cls,
+ const struct GNUNET_SCHEDULER_TaskContext *tc)
+{
+ struct GNUNET_FS_DownloadContext *dc = cls;
+
+ dc->start_task = GNUNET_SCHEDULER_NO_TASK;
+ schedule_block_download (dc,
+ (dc->uri->type == chk)
+ ? &dc->uri->data.chk.chk
+ : &dc->uri->data.loc.fi.chk,
+ 0,
+ 1 /* 0 == CHK, 1 == top */);
+}
+
+
/**
* Create SUSPEND event for the given download operation
* and then clean up our state (without stop signal).
{
struct GNUNET_FS_DownloadContext *dc = cls;
struct GNUNET_FS_ProgressInfo pi;
-
+
+ if (dc->start_task != GNUNET_SCHEDULER_NO_TASK)
+ {
+ GNUNET_SCHEDULER_cancel (dc->start_task);
+ dc->start_task = GNUNET_SCHEDULER_NO_TASK;
+ }
if (dc->top != NULL)
GNUNET_FS_end_top (dc->h, dc->top);
while (NULL != dc->child_head)
pi.status = GNUNET_FS_STATUS_DOWNLOAD_START;
pi.value.download.specifics.start.meta = meta;
GNUNET_FS_download_make_status_ (&pi, dc);
- schedule_block_download (dc,
- (dc->uri->type == chk)
- ? &dc->uri->data.chk.chk
- : &dc->uri->data.loc.fi.chk,
- 0,
- 1 /* 0 == CHK, 1 == top */);
+ dc->start_task = GNUNET_SCHEDULER_add_now (&GNUNET_FS_download_start_task_, dc);
GNUNET_FS_download_sync_ (dc);
GNUNET_FS_download_start_downloading_ (dc);
return dc;
pi.status = GNUNET_FS_STATUS_DOWNLOAD_START;
pi.value.download.specifics.start.meta = dc->meta;
GNUNET_FS_download_make_status_ (&pi, dc);
- schedule_block_download (dc,
- &dc->uri->data.chk.chk,
- 0,
- 1 /* 0 == CHK, 1 == top */);
+ dc->start_task = GNUNET_SCHEDULER_add_now (&GNUNET_FS_download_start_task_, dc);
GNUNET_FS_download_sync_ (dc);
GNUNET_FS_download_start_downloading_ (dc);
return dc;
}
-
/**
* Start the downloading process (by entering the queue).
*
void
GNUNET_FS_download_start_downloading_ (struct GNUNET_FS_DownloadContext *dc)
{
+ if (dc->completed == dc->length)
+ return;
GNUNET_assert (dc->job_queue == NULL);
dc->job_queue = GNUNET_FS_queue_ (dc->h,
&activate_fs_download,
if (dc->top != NULL)
GNUNET_FS_end_top (dc->h, dc->top);
+ if (dc->start_task != GNUNET_SCHEDULER_NO_TASK)
+ {
+ GNUNET_SCHEDULER_cancel (dc->start_task);
+ dc->start_task = GNUNET_SCHEDULER_NO_TASK;
+ }
if (dc->search != NULL)
{
dc->search->download = NULL;
* @param cls closure
* @param query the query for the block (key for lookup in the datastore)
* @param offset offset of the block in the file
+ * @param depth depth of the block in the file
* @param type type of the block (IBLOCK or DBLOCK)
* @param block the (encrypted) block
* @param block_size size of block (in bytes)
block_proc (void *cls,
const GNUNET_HashCode *query,
uint64_t offset,
+ unsigned int depth,
enum GNUNET_BLOCK_Type type,
const void *block,
uint16_t block_size)
* @file fs/fs_search.c
* @brief Helper functions for searching.
* @author Christian Grothoff
- *
- * TODO:
- * - add support for pushing "already seen" information
- * to FS service for bloomfilter (can wait)
*/
#include "platform.h"
}
+/**
+ * Schedule the transmission of the (next) search request
+ * to the service.
+ *
+ * @param sc context for the search
+ */
+static void
+schedule_transmit_search_request (struct GNUNET_FS_SearchContext *sc);
+
+
+/**
+ * Closure for 'build_result_set'.
+ */
+struct MessageBuilderContext
+{
+ /**
+ * How many entries can we store to xoff.
+ */
+ unsigned int put_cnt;
+
+ /**
+ * How many entries should we skip.
+ */
+ unsigned int skip_cnt;
+
+ /**
+ * Where to store the keys.
+ */
+ GNUNET_HashCode *xoff;
+
+ /**
+ * Search context we are iterating for.
+ */
+ struct GNUNET_FS_SearchContext *sc;
+
+ /**
+ * URI the search result must match, NULL for any
+ */
+ struct GNUNET_FS_Uri *uri;
+};
+
+
+/**
+ * Iterating over the known results, pick those
+ * matching the given result range and store
+ * their keys at 'xoff'.
+ *
+ * @param cls the 'struct MessageBuilderContext'
+ * @param key key for a result
+ * @param value the search result
+ * @return GNUNET_OK to continue iterating
+ */
+static int
+build_result_set (void *cls,
+ const GNUNET_HashCode *key,
+ void *value)
+{
+ struct MessageBuilderContext *mbc = cls;
+ struct GNUNET_FS_SearchResult *sr = value;
+
+ if ( (mbc->uri != NULL) &&
+ (GNUNET_YES != GNUNET_FS_uri_test_equal (mbc->uri,
+ sr->uri)) )
+ return GNUNET_OK;
+ if (mbc->skip_cnt > 0)
+ {
+ mbc->skip_cnt--;
+ return GNUNET_OK;
+ }
+ if (mbc->put_cnt == 0)
+ return GNUNET_SYSERR;
+ mbc->sc->search_request_map_offset++;
+ mbc->xoff[--mbc->put_cnt] = *key;
+ return GNUNET_OK;
+}
+
+
+/**
+ * Iterating over the known results, count those
+ * matching the given result range and increment
+ * put count for each.
+ *
+ * @param cls the 'struct MessageBuilderContext'
+ * @param key key for a result
+ * @param value the search result
+ * @return GNUNET_OK to continue iterating
+ */
+static int
+find_result_set (void *cls,
+ const GNUNET_HashCode *key,
+ void *value)
+{
+ struct MessageBuilderContext *mbc = cls;
+ struct GNUNET_FS_SearchResult *sr = value;
+
+ if ( (mbc->uri != NULL) &&
+ (GNUNET_YES != GNUNET_FS_uri_test_equal (mbc->uri,
+ sr->uri)) )
+ return GNUNET_OK;
+ mbc->put_cnt++;
+ return GNUNET_OK;
+}
+
+
/**
* We're ready to transmit the search request to the
* file-sharing service. Do it.
void *buf)
{
struct GNUNET_FS_SearchContext *sc = cls;
+ struct MessageBuilderContext mbc;
size_t msize;
struct SearchMessage *sm;
- unsigned int i;
const char *identifier;
GNUNET_HashCode key;
GNUNET_HashCode idh;
+ unsigned int sqms;
if (NULL == buf)
{
try_reconnect (sc);
return 0;
}
+ mbc.sc = sc;
+ mbc.skip_cnt = sc->search_request_map_offset;
+ sm = buf;
+ sm->header.type = htons (GNUNET_MESSAGE_TYPE_FS_START_SEARCH);
+ mbc.xoff = (GNUNET_HashCode* ) &sm[1];
if (GNUNET_FS_uri_test_ksk (sc->uri))
{
- msize = sizeof (struct SearchMessage) * sc->uri->data.ksk.keywordCount;
+ msize = sizeof (struct SearchMessage);
GNUNET_assert (size >= msize);
- sm = buf;
- memset (sm, 0, msize);
- for (i=0;i<sc->uri->data.ksk.keywordCount;i++)
+ mbc.uri = NULL;
+ mbc.put_cnt = 0;
+ GNUNET_CONTAINER_multihashmap_iterate (sc->master_result_map,
+ &find_result_set,
+ &mbc);
+ sqms = mbc.put_cnt;
+ mbc.put_cnt = (size - msize) / sizeof (GNUNET_HashCode);
+ mbc.put_cnt = GNUNET_MIN (mbc.put_cnt,
+ sqms - mbc.skip_cnt);
+ if (sc->search_request_map_offset < sqms)
+ GNUNET_assert (mbc.put_cnt > 0);
+
+ sm->header.size = htons (msize);
+ if (0 != (sc->options & GNUNET_FS_SEARCH_OPTION_LOOPBACK_ONLY))
+ sm->options = htonl (1);
+ else
+ sm->options = htonl (0);
+ sm->type = htonl (GNUNET_BLOCK_TYPE_ANY);
+ sm->anonymity_level = htonl (sc->anonymity);
+ sm->query = sc->requests[sc->keyword_offset].query;
+ msize += sizeof (GNUNET_HashCode) * mbc.put_cnt;
+ GNUNET_CONTAINER_multihashmap_iterate (sc->master_result_map,
+ &build_result_set,
+ &mbc);
+ sm->header.size = htons (msize);
+ if (sqms != sc->search_request_map_offset)
{
- sm[i].header.size = htons (sizeof (struct SearchMessage));
- sm[i].header.type = htons (GNUNET_MESSAGE_TYPE_FS_START_SEARCH);
- if (0 != (sc->options & GNUNET_FS_SEARCH_OPTION_LOOPBACK_ONLY))
- sm[i].options = htonl (1);
- else
- sm[i].options = htonl (0);
- sm[i].type = htonl (GNUNET_BLOCK_TYPE_ANY);
- sm[i].anonymity_level = htonl (sc->anonymity);
- sm[i].query = sc->requests[i].query;
- /* FIXME: should transmit hash codes of all already-known results here!
- (and if they do not fit, add another message with the same
- header and additional already-seen results!) */
+ /* more requesting to be done... */
+ schedule_transmit_search_request (sc);
+ return msize;
+ }
+ sc->keyword_offset++;
+ if (sc->uri->data.ksk.keywordCount !=
+ sc->keyword_offset)
+ {
+ /* more requesting to be done... */
+ schedule_transmit_search_request (sc);
+ return msize;
}
}
else
GNUNET_assert (GNUNET_FS_uri_test_sks (sc->uri));
msize = sizeof (struct SearchMessage);
GNUNET_assert (size >= msize);
- sm = buf;
- memset (sm, 0, msize);
- sm->header.size = htons (sizeof (struct SearchMessage));
- sm->header.type = htons (GNUNET_MESSAGE_TYPE_FS_START_SEARCH);
if (0 != (sc->options & GNUNET_FS_SEARCH_OPTION_LOOPBACK_ONLY))
sm->options = htonl (1);
else
GNUNET_CRYPTO_hash_xor (&idh,
&sm->target,
&sm->query);
- /* FIXME: should transmit hash codes of all already-known results here!
- (and if they do not fit, add another message with the same
- header and additional already-seen results!) */
- }
+ mbc.skip_cnt = sc->search_request_map_offset;
+ mbc.put_cnt = (size - msize) / sizeof (GNUNET_HashCode);
+ sqms = GNUNET_CONTAINER_multihashmap_size (sc->master_result_map);
+ mbc.put_cnt = GNUNET_MIN (mbc.put_cnt,
+ sqms - mbc.skip_cnt);
+ mbc.uri = NULL;
+ if (sc->search_request_map_offset < sqms)
+ GNUNET_assert (mbc.put_cnt > 0);
+ msize += sizeof (GNUNET_HashCode) * mbc.put_cnt;
+ GNUNET_CONTAINER_multihashmap_iterate (sc->master_result_map,
+ &build_result_set,
+ &mbc);
+ sm->header.size = htons (msize);
+ if (sqms != sc->search_request_map_offset)
+ {
+ /* more requesting to be done... */
+ schedule_transmit_search_request (sc);
+ return msize;
+ }
+ }
GNUNET_CLIENT_receive (sc->client,
&receive_results,
sc,
}
+/**
+ * Schedule the transmission of the (next) search request
+ * to the service.
+ *
+ * @param sc context for the search
+ */
+static void
+schedule_transmit_search_request (struct GNUNET_FS_SearchContext *sc)
+{
+ size_t size;
+ unsigned int sqms;
+ unsigned int fit;
+
+ size = sizeof (struct SearchMessage);
+ sqms = GNUNET_CONTAINER_multihashmap_size (sc->master_result_map) - sc->search_request_map_offset;
+ fit = (GNUNET_SERVER_MAX_MESSAGE_SIZE - 1 - size) / sizeof (GNUNET_HashCode);
+ fit = GNUNET_MIN (fit, sqms);
+ size += sizeof (GNUNET_HashCode) * fit;
+ GNUNET_CLIENT_notify_transmit_ready (sc->client,
+ size,
+ GNUNET_CONSTANTS_SERVICE_TIMEOUT,
+ GNUNET_NO,
+ &transmit_search_request,
+ sc);
+
+}
+
+
/**
* Reconnect to the FS service and transmit
* our queries NOW.
{
struct GNUNET_FS_SearchContext *sc = cls;
struct GNUNET_CLIENT_Connection *client;
- size_t size;
-
+
sc->task = GNUNET_SCHEDULER_NO_TASK;
client = GNUNET_CLIENT_connect ("fs",
sc->h->cfg);
return;
}
sc->client = client;
- if (GNUNET_FS_uri_test_ksk (sc->uri))
- size = sizeof (struct SearchMessage) * sc->uri->data.ksk.keywordCount;
- else
- size = sizeof (struct SearchMessage);
- GNUNET_CLIENT_notify_transmit_ready (client,
- size,
- GNUNET_CONSTANTS_SERVICE_TIMEOUT,
- GNUNET_NO,
- &transmit_search_request,
- sc);
+ sc->search_request_map_offset = 0;
+ sc->keyword_offset = 0;
+ schedule_transmit_search_request (sc);
}
GNUNET_HashCode hc;
struct GNUNET_CRYPTO_RsaPublicKeyBinaryEncoded pub;
struct GNUNET_CRYPTO_RsaPrivateKey *pk;
- size_t size;
GNUNET_assert (NULL == sc->client);
- if (GNUNET_FS_uri_test_ksk (sc->uri))
- {
- size = sizeof (struct SearchMessage) * sc->uri->data.ksk.keywordCount;
- }
- else
- {
- GNUNET_assert (GNUNET_FS_uri_test_sks (sc->uri));
- size = sizeof (struct SearchMessage);
- }
- if (size >= GNUNET_SERVER_MAX_MESSAGE_SIZE)
- {
- GNUNET_log (GNUNET_ERROR_TYPE_ERROR,
- _("Too many keywords specified for a single search."));
- return GNUNET_SYSERR;
- }
if (GNUNET_FS_uri_test_ksk (sc->uri))
{
GNUNET_assert (0 != sc->uri->data.ksk.keywordCount);
for (i=0;i<sc->uri->data.ksk.keywordCount;i++)
{
keyword = &sc->uri->data.ksk.keywords[i][1];
- GNUNET_CRYPTO_hash (keyword, strlen (keyword), &hc);
+ GNUNET_CRYPTO_hash (keyword,
+ strlen (keyword),
+ &hc);
pk = GNUNET_CRYPTO_rsa_key_create_from_hash (&hc);
GNUNET_assert (pk != NULL);
GNUNET_CRYPTO_rsa_key_get_public (pk, &pub);
sc->h->cfg);
if (NULL == sc->client)
return GNUNET_SYSERR;
- GNUNET_CLIENT_notify_transmit_ready (sc->client,
- size,
- GNUNET_CONSTANTS_SERVICE_TIMEOUT,
- GNUNET_NO,
- &transmit_search_request,
- sc);
+ schedule_transmit_search_request (sc);
return GNUNET_OK;
}
*/
struct ContentHashKey *chk_tree;
+ /**
+ * Are we currently in 'GNUNET_FS_tree_encoder_next'?
+ * Flag used to prevent recursion.
+ */
+ int in_next;
};
*
* @param te tree encoder to use
*/
-void GNUNET_FS_tree_encoder_next (struct GNUNET_FS_TreeEncoder * te)
+void
+GNUNET_FS_tree_encoder_next (struct GNUNET_FS_TreeEncoder * te)
{
struct ContentHashKey *mychk;
const void *pt_block;
struct GNUNET_CRYPTO_AesInitializationVector iv;
unsigned int off;
+ GNUNET_assert (GNUNET_NO == te->in_next);
+ te->in_next = GNUNET_YES;
if (te->current_depth == te->chk_tree_depth)
{
pt_size = GNUNET_MIN(DBLOCK_SIZE,
GNUNET_SCHEDULER_add_continuation (te->cont,
te->cls,
GNUNET_SCHEDULER_REASON_TIMEOUT);
+ te->in_next = GNUNET_NO;
return;
}
pt_block = iob;
GNUNET_SCHEDULER_add_continuation (te->cont,
te->cls,
GNUNET_SCHEDULER_REASON_PREREQ_DONE);
+ te->in_next = GNUNET_NO;
return;
}
off = compute_chk_offset (te->chk_tree_depth - te->current_depth,
te->proc (te->cls,
&mychk->query,
te->publish_offset,
+ te->current_depth,
(te->current_depth == te->chk_tree_depth)
? GNUNET_BLOCK_TYPE_FS_DBLOCK
: GNUNET_BLOCK_TYPE_FS_IBLOCK,
else
te->current_depth = te->chk_tree_depth;
}
+ te->in_next = GNUNET_NO;
}
* @param cls closure
* @param query the query for the block (key for lookup in the datastore)
* @param offset offset of the block
+ * @param depth depth of the block
* @param type type of the block (IBLOCK or DBLOCK)
* @param block the (encrypted) block
* @param block_size size of block (in bytes)
typedef void (*GNUNET_FS_TreeBlockProcessor)(void *cls,
const GNUNET_HashCode *query,
uint64_t offset,
+ unsigned int depth,
enum GNUNET_BLOCK_Type type,
const void *block,
uint16_t block_size);
* @param cls closure
* @param query the query for the block (key for lookup in the datastore)
* @param offset offset of the block
+ * @param depth depth of the block
* @param type type of the block (IBLOCK or DBLOCK)
* @param block the (encrypted) block
* @param block_size size of block (in bytes)
unindex_process (void *cls,
const GNUNET_HashCode *query,
uint64_t offset,
+ unsigned int depth,
enum GNUNET_BLOCK_Type type,
const void *block,
uint16_t block_size)
(! GNUNET_FS_uri_test_loc (uri)) )
{
fprintf (stderr,
- "Only CHK or LOC URIs supported.\n");
+ _("Only CHK or LOC URIs supported.\n"));
ret = 1;
GNUNET_FS_uri_destroy (uri);
return;
if (NULL == filename)
{
fprintf (stderr,
- "Target filename must be specified.\n");
+ _("Target filename must be specified.\n"));
ret = 1;
GNUNET_FS_uri_destroy (uri);
return;
*/
#define TRUST_FLUSH_FREQ GNUNET_TIME_relative_multiply (GNUNET_TIME_UNIT_MINUTES, 5)
+/**
+ * How quickly do we age cover traffic? At the given
+ * time interval, remaining cover traffic counters are
+ * decremented by 1/16th.
+ */
+#define COVER_AGE_FREQUENCY GNUNET_TIME_relative_multiply (GNUNET_TIME_UNIT_SECONDS, 5)
+
/**
* How often do we at most PUT content into the DHT?
*/
};
-
-
-
/**
* Doubly-linked list of messages we are performing
* due to a pending request.
*/
static struct GNUNET_LOAD_Value *rt_entry_lifetime;
+/**
+ * How many query messages have we received 'recently' that
+ * have not yet been claimed as cover traffic?
+ */
+static unsigned int cover_query_count;
+
+/**
+ * How many content messages have we received 'recently' that
+ * have not yet been claimed as cover traffic?
+ */
+static unsigned int cover_content_count;
+
+/**
+ * ID of our task that we use to age the cover counters.
+ */
+static GNUNET_SCHEDULER_TaskIdentifier cover_age_task;
+
+static void
+age_cover_counters (void *cls,
+ const struct GNUNET_SCHEDULER_TaskContext *tc)
+{
+ cover_content_count = (cover_content_count * 15) / 16;
+ cover_query_count = (cover_query_count * 15) / 16;
+ cover_age_task = GNUNET_SCHEDULER_add_delayed (COVER_AGE_FREQUENCY,
+ &age_cover_counters,
+ NULL);
+}
+
/**
* We've just now completed a datastore request. Update our
* datastore load calculations.
cfg = NULL;
GNUNET_free_non_null (trustDirectory);
trustDirectory = NULL;
+ GNUNET_SCHEDULER_cancel (cover_age_task);
+ cover_age_task = GNUNET_SCHEDULER_NO_TASK;
}
&process_dht_reply,
pr);
}
+
+ if ( (pr->anonymity_level > 1) &&
+ (cover_query_count < pr->anonymity_level - 1) )
+ {
+ delay = get_processing_delay ();
+#if DEBUG_FS
+ GNUNET_log (GNUNET_ERROR_TYPE_DEBUG,
+ "Not enough cover traffic to forward query `%s', will try again in %llu ms!\n",
+ GNUNET_h2s (&pr->query),
+ delay.rel_value);
+#endif
+ pr->task = GNUNET_SCHEDULER_add_delayed (delay,
+ &forward_request_task,
+ pr);
+ return;
+ }
+ /* consume cover traffic */
+ if (pr->anonymity_level > 1)
+ cover_query_count -= pr->anonymity_level - 1;
+
/* (1) select target */
psc.pr = pr;
psc.target_score = -DBL_MAX;
*/
uint32_t priority;
+ /**
+ * Anonymity requirements for this reply.
+ */
+ uint32_t anonymity_level;
+
/**
* Evaluation result (returned).
*/
size_t msize;
unsigned int i;
+ if (NULL == pr->client_request_list)
+ {
+ /* reply will go over the network, check for cover traffic */
+ if ( (prq->anonymity_level > 1) &&
+ (cover_content_count < prq->anonymity_level - 1) )
+ {
+ /* insufficient cover traffic, skip */
+ GNUNET_STATISTICS_update (stats,
+ gettext_noop ("# replies suppressed due to lack of cover traffic"),
+ 1,
+ GNUNET_NO);
+ return GNUNET_YES;
+ }
+ if (prq->anonymity_level > 1)
+ cover_content_count -= prq->anonymity_level - 1;
+ }
#if DEBUG_FS
GNUNET_log (GNUNET_ERROR_TYPE_DEBUG,
"Matched result (type %u) for query `%s' with pending request\n",
GNUNET_break_op (0);
return GNUNET_SYSERR;
}
+ cover_content_count++;
#if DEBUG_FS
GNUNET_log (GNUNET_ERROR_TYPE_DEBUG,
"Received result for query `%s' from peer `%4s'\n",
prq.type = type;
prq.expiration = expiration;
prq.priority = 0;
+ prq.anonymity_level = 1;
prq.finished = GNUNET_NO;
prq.request_found = GNUNET_NO;
GNUNET_CONTAINER_multihashmap_get_multiple (query_request_map,
prq.priority = priority;
prq.finished = GNUNET_NO;
prq.request_found = GNUNET_NO;
+ prq.anonymity_level = anonymity;
if ( (old_rf == 0) &&
(pr->results_found == 0) )
update_datastore_delays (pr->start_time);
GNUNET_break_op (0);
return GNUNET_SYSERR;
}
+ cover_query_count++;
bm = ntohl (gm->hash_bitmap);
bits = 0;
cps = GNUNET_CONTAINER_multihashmap_get (connected_peers,
GNUNET_SERVER_add_handlers (server, handlers);
+ cover_age_task = GNUNET_SCHEDULER_add_delayed (COVER_AGE_FREQUENCY,
+ &age_cover_counters,
+ NULL);
GNUNET_SCHEDULER_add_delayed (GNUNET_TIME_UNIT_FOREVER_REL,
&shutdown_task,
NULL);
projects / oweals / gnunet.git / commitdiff