gnunet-handbook

The GNUnet Handbook
Log | Files | Refs

fs.rst (19047B)


      1 File-sharing
      2 ------------
      3 
      4 .. note::
      5 
      6    File-sharing is under development and has known design issues that affect
      7    usability and performance.  See :doc:`/status`.
      8 
      9 This chapter documents the GNUnet file-sharing application. The original
     10 file-sharing implementation for GNUnet was designed to provide anonymous
     11 file-sharing. However, over time, we have also added support for
     12 non-anonymous file-sharing (which can provide better performance).
     13 Anonymous and non-anonymous file-sharing are quite integrated in GNUnet
     14 and, except for routing, share most of the concepts and implementation.
     15 There are three primary file-sharing operations: publishing, searching
     16 and downloading. For each of these operations, the user specifies an
     17 anonymity level. If both the publisher and the searcher/downloader
     18 specify “no anonymity”, non-anonymous file-sharing is used. If either
     19 user specifies some desired degree of anonymity, anonymous file-sharing
     20 will be used.
     21 
     22 After a short introduction, we will first look at the various concepts
     23 in GNUnet’s file-sharing implementation. Then, we will discuss specifics
     24 as to how they impact users that publish, search or download files.
     25 
     26 Searching
     27 ~~~~~~~~~
     28 
     29 The command ``gnunet-search`` can be used to search for content on
     30 GNUnet. The format is:
     31 
     32 ::
     33 
     34    $ gnunet-search [-t TIMEOUT] KEYWORD
     35 
     36 The ``-t`` option specifies that the query should timeout after
     37 approximately ``TIMEOUT`` seconds. A value of zero (“0”) is interpreted
     38 as no timeout, which is the default. In this case, gnunet-search will
     39 never terminate (unless you press CTRL-C).
     40 
     41 If multiple words are passed as keywords, they will all be considered
     42 optional. Prefix keywords with a “+” to make them mandatory.
     43 
     44 Note that searching using:
     45 
     46 ::
     47 
     48    $ gnunet-search Das Kapital
     49 
     50 is not the same as searching for
     51 
     52 ::
     53 
     54    $ gnunet-search "Das Kapital"
     55 
     56 as the first will match files shared under the keywords “Das” or
     57 “Kapital” whereas the second will match files shared under the keyword
     58 “Das Kapital”.
     59 
     60 Search results are printed like this:
     61 
     62 ::
     63 
     64    #15:
     65    gnunet-download -o "COPYING" gnunet://fs/chk/PGK8M...3EK130.75446
     66 
     67 The whole line is the command you would have to enter to download the
     68 file. The first argument passed to ``-o`` is the suggested filename (you
     69 may change it to whatever you like). It is followed by the key for
     70 decrypting the file, the query for searching the file, a checksum (in
     71 hexadecimal) finally the size of the file in bytes.
     72 
     73 Downloading
     74 ~~~~~~~~~~~
     75 
     76 In order to download a file, you need the whole line returned by
     77 gnunet-search. You can then use the tool ``gnunet-download`` to obtain
     78 the file:
     79 
     80 ::
     81 
     82    $ gnunet-download -o <FILENAME> <GNUNET-URL>
     83 
     84 ``FILENAME`` specifies the name of the file where GNUnet is supposed to
     85 write the result. Existing files are overwritten. If the existing file
     86 contains blocks that are identical to the desired download, those blocks
     87 will not be downloaded again (automatic resume).
     88 
     89 If you want to download the GPL from the previous example, you do the
     90 following:
     91 
     92 ::
     93 
     94    $ gnunet-download -o "COPYING" gnunet://fs/chk/PGK8M...3EK130.75446
     95 
     96 If you ever have to abort a download, you can continue it at any time by
     97 re-issuing gnunet-download with the same filename. In that case, GNUnet
     98 will **not** download blocks again that are already present.
     99 
    100 GNUnet’s file-encoding mechanism will ensure file integrity, even if the
    101 existing file was not downloaded from GNUnet in the first place.
    102 
    103 You may want to use the ``-V`` switch to turn on verbose reporting. In
    104 this case, gnunet-download will print the current number of bytes
    105 downloaded whenever new data was received.
    106 
    107 Publishing
    108 ~~~~~~~~~~
    109 
    110 The command ``gnunet-publish`` can be used to add content to the
    111 network. The basic format of the command is:
    112 
    113 ::
    114 
    115    $ gnunet-publish [-n] [-k KEYWORDS]* [-m TYPE:VALUE] FILENAME
    116 
    117 For example:
    118 
    119 ::
    120 
    121    $ gnunet-publish -m "description:GNU License" -k gpl -k test -m "mimetype:text/plain" COPYING
    122 
    123 The option ``-k`` is used to specify keywords for the file that should
    124 be inserted. You can supply any number of keywords, and each of the
    125 keywords will be sufficient to locate and retrieve the file. Please note
    126 that you must use the ``-k`` option more than once – one for each
    127 expression you use as a keyword for the filename.
    128 
    129 The ``-m`` option is used to specify meta-data, such as descriptions.
    130 You can use ``-m`` multiple times. The ``TYPE`` passed must be from the
    131 list of meta-data types known to libextractor. You can obtain this list
    132 by running ``extract -L``. Use quotes around the entire meta-data
    133 argument if the value contains spaces. The meta-data is displayed to
    134 other users when they select which files to download. The meta-data and
    135 the keywords are optional and may be inferred using GNU libextractor.
    136 
    137 ``gnunet-publish`` has a few additional options to handle namespaces and
    138 directories. Refer to the man-page for details.
    139 
    140 Indexing vs Inserting
    141 ~~~~~~~~~~~~~~~~~~~~~
    142 
    143 By default, GNUnet indexes a file instead of making a full copy. This is
    144 much more efficient, but requires the file to stay unaltered at the
    145 location where it was when it was indexed. If you intend to move, delete
    146 or alter a file, consider using the option ``-n`` which will force
    147 GNUnet to make a copy of the file in the database.
    148 
    149 Since it is much less efficient, this is strongly discouraged for large
    150 files. When GNUnet indexes a file (default), GNUnet does **not** create
    151 an additional encrypted copy of the file but just computes a summary (or
    152 index) of the file. That summary is approximately two percent of the
    153 size of the original file and is stored in GNUnet’s database. Whenever a
    154 request for a part of an indexed file reaches GNUnet, this part is
    155 encrypted on-demand and send out. This way, there is no need for an
    156 additional encrypted copy of the file to stay anywhere on the drive.
    157 This is different from other systems, such as Freenet, where each file
    158 that is put online must be in Freenet’s database in encrypted format,
    159 doubling the space requirements if the user wants to preserve a directly
    160 accessible copy in plaintext.
    161 
    162 Thus indexing should be used for all files where the user will keep
    163 using this file (at the location given to gnunet-publish) and does not
    164 want to retrieve it back from GNUnet each time. If you want to remove a
    165 file that you have indexed from the local peer, use the tool
    166 gnunet-unindex to un-index the file.
    167 
    168 The option ``-n`` may be used if the user fears that the file might be
    169 found on their drive (assuming the computer comes under the control of
    170 an adversary). When used with the ``-n`` flag, the user has a much
    171 better chance of denying knowledge of the existence of the file, even if
    172 it is still (encrypted) on the drive and the adversary is able to crack
    173 the encryption (e.g. by guessing the keyword).
    174 
    175 .. _fs_002dConcepts:
    176 
    177 Concepts
    178 ~~~~~~~~
    179 
    180 For better results with filesharing it is useful to understand the
    181 following concepts. In addition to anonymous routing GNUnet attempts to
    182 give users a better experience in searching for content. GNUnet uses
    183 cryptography to safely break content into smaller pieces that can be
    184 obtained from different sources without allowing participants to corrupt
    185 files. GNUnet makes it difficult for an adversary to send back bogus
    186 search results. GNUnet enables content providers to group related
    187 content and to establish a reputation. Furthermore, GNUnet allows
    188 updates to certain content to be made available. This section is
    189 supposed to introduce users to the concepts that are used to achieve
    190 these goals.
    191 
    192 .. _Files:
    193 
    194 Files
    195 ^^^^^
    196 
    197 A file in GNUnet is just a sequence of bytes. Any file-format is allowed
    198 and the maximum file size is theoretically :math:`2^64 - 1` bytes,
    199 except that it would take an impractical amount of time to share such a
    200 file. GNUnet itself never interprets the contents of shared files,
    201 except when using GNU libextractor to obtain keywords.
    202 
    203 .. _Keywords:
    204 
    205 Keywords
    206 ^^^^^^^^
    207 
    208 Keywords are the most simple mechanism to find files on GNUnet. Keywords
    209 are **case-sensitive** and the search string must always match
    210 **exactly** the keyword used by the person providing the file. Keywords
    211 are never transmitted in plaintext. The only way for an adversary to
    212 determine the keyword that you used to search is to guess it (which then
    213 allows the adversary to produce the same search request). Since
    214 providing keywords by hand for each shared file is tedious, GNUnet uses
    215 GNU libextractor to help automate this process. Starting a keyword
    216 search on a slow machine can take a little while since the keyword
    217 search involves computing a fresh RSA key to formulate the request.
    218 
    219 .. _Directories:
    220 
    221 Directories
    222 ^^^^^^^^^^^
    223 
    224 A directory in GNUnet is a list of file identifiers with meta data. The
    225 file identifiers provide sufficient information about the files to allow
    226 downloading the contents. Once a directory has been created, it cannot
    227 be changed since it is treated just like an ordinary file by the
    228 network. Small files (of a few kilobytes) can be inlined in the
    229 directory, so that a separate download becomes unnecessary.
    230 
    231 Directories are shared just like ordinary files. If you download a
    232 directory with ``gnunet-download``, you can use ``gnunet-directory`` to
    233 list its contents. The canonical extension for GNUnet directories when
    234 stored as files in your local file-system is \".gnd\". The contents of a
    235 directory are URIs and meta data. The URIs contain all the information
    236 required by ``gnunet-download`` to retrieve the file. The meta data
    237 typically includes the mime-type, description, a filename and other meta
    238 information, and possibly even the full original file (if it was small).
    239 
    240 .. _Egos-and-File_002dSharing:
    241 
    242 Egos and File-Sharing
    243 ^^^^^^^^^^^^^^^^^^^^^
    244 
    245 When sharing files, it is sometimes desirable to build a reputation as a
    246 source for quality information. With egos, publishers can
    247 (cryptographically) sign files, thereby demonstrating that various files
    248 were published by the same entity. An ego thus allows users to link
    249 different publication events, thereby deliberately reducing anonymity to
    250 pseudonymity.
    251 
    252 Egos used in GNUnet's file-sharing for such pseudonymous publishing also
    253 correspond to the egos used to identify and sign zones in the GNU Name
    254 System. However, if the same ego is used for file-sharing and for a GNS
    255 zone, this will weaken the privacy assurances provided by the anonymous
    256 file-sharing protocol.
    257 
    258 Note that an ego is NOT bound to a GNUnet peer. There can be multiple
    259 egos for a single user, and users could (theoretically) share the
    260 private keys of an ego by copying the respective private keys.
    261 
    262 .. _Namespaces:
    263 
    264 Namespaces
    265 ^^^^^^^^^^
    266 
    267 A namespace is a set of files that were signed by the same ego. Today,
    268 namespaces are implemented independently of GNS zones, but in the future
    269 we plan to merge the two such that a GNS zone can basically contain
    270 files using a file-sharing specific record type.
    271 
    272 Files (or directories) that have been signed and placed into a namespace
    273 can be updated. Updates are identified as authentic if the same secret
    274 key was used to sign the update.
    275 
    276 .. _Advertisements:
    277 
    278 Advertisements
    279 ^^^^^^^^^^^^^^
    280 
    281 Advertisements are used to notify other users about the existence of a
    282 namespace. Advertisements are propagated using the normal keyword
    283 search. When an advertisement is received (in response to a search), the
    284 namespace is added to the list of namespaces available in the
    285 namespace-search dialogs of gnunet-fs-gtk and printed by
    286 ``gnunet-identity``. Whenever a namespace is created, an appropriate
    287 advertisement can be generated. The default keyword for the advertising
    288 of namespaces is \"namespace\".
    289 
    290 .. _Anonymity-level:
    291 
    292 Anonymity level
    293 ^^^^^^^^^^^^^^^
    294 
    295 The anonymity level determines how hard it should be for an adversary to
    296 determine the identity of the publisher or the searcher/downloader. An
    297 anonymity level of zero means that anonymity is not required. The
    298 default anonymity level of \"1\" means that anonymous routing is
    299 desired, but no particular amount of cover traffic is necessary. A
    300 powerful adversary might thus still be able to deduce the origin of the
    301 traffic using traffic analysis. Specifying higher anonymity levels
    302 increases the amount of cover traffic required.
    303 
    304 The specific numeric value (for anonymity levels above 1) is simple:
    305 Given an anonymity level L (above 1), each request FS makes on your
    306 behalf must be hidden in L-1 equivalent requests of cover traffic
    307 (traffic your peer routes for others) in the same time-period. The
    308 time-period is twice the average delay by which GNUnet artificially
    309 delays traffic.
    310 
    311 While higher anonymity levels may offer better privacy, they can also
    312 significantly hurt performance.
    313 
    314 .. _Content-Priority:
    315 
    316 Content Priority
    317 ^^^^^^^^^^^^^^^^
    318 
    319 Depending on the peer's configuration, GNUnet peers migrate content
    320 between peers. Content in this sense are individual blocks of a file,
    321 not necessarily entire files. When peers run out of space (due to local
    322 publishing operations or due to migration of content from other peers),
    323 blocks sometimes need to be discarded. GNUnet first always discards
    324 expired blocks (typically, blocks are published with an expiration of
    325 about two years in the future; this is another option). If there is
    326 still not enough space, GNUnet discards the blocks with the lowest
    327 priority. The priority of a block is decided by its popularity (in terms
    328 of requests from peers we trust) and, in case of blocks published
    329 locally, the base-priority that was specified by the user when the block
    330 was published initially.
    331 
    332 .. _Replication:
    333 
    334 Replication
    335 ^^^^^^^^^^^
    336 
    337 When peers migrate content to other systems, the replication level of a
    338 block is used to decide which blocks need to be migrated most urgently.
    339 GNUnet will always push the block with the highest replication level
    340 into the network, and then decrement the replication level by one. If
    341 all blocks reach replication level zero, the selection is simply random.
    342 
    343 .. _Namespace-Management:
    344 
    345 Namespace Management
    346 ~~~~~~~~~~~~~~~~~~~~
    347 
    348 The ``gnunet-identity`` tool can be used to create egos. By default,
    349 ``gnunet-identity --display`` simply lists all locally available egos.
    350 
    351 .. _Creating-Egos:
    352 
    353 Creating Egos
    354 ^^^^^^^^^^^^^
    355 
    356 With the ``--create=NICK`` option it can also be used to create a new
    357 ego. An ego is the virtual identity of the entity in control of a
    358 namespace or GNS zone. Anyone can create any number of egos. The
    359 provided NICK name automatically corresponds to a GNU Name System domain
    360 name. Thus, henceforth name resolution for any name ending in ".NICK"
    361 will use the NICK's zone. You should avoid using NICKs that collide with
    362 well-known DNS names.
    363 
    364 Currently, the IDENTITY subsystem supports two types of identity keys:
    365 ECDSA and EdDSA. By default, ECDSA identities are creates with ECDSA
    366 keys. In order to create an identity with EdDSA keys, you can use the
    367 ``--eddsa`` flag.
    368 
    369 .. _Deleting-Egos:
    370 
    371 Deleting Egos
    372 ^^^^^^^^^^^^^
    373 
    374 With the ``-D NICK`` option egos can be deleted. Once the ego has been
    375 deleted it is impossible to add content to the corresponding namespace
    376 or zone. However, the existing GNS zone data is currently not dropped.
    377 This may change in the future.
    378 
    379 Deleting the pseudonym does not make the namespace or any content in it
    380 unavailable.
    381 
    382 .. _File_002dSharing-URIs:
    383 
    384 File-Sharing URIs
    385 ~~~~~~~~~~~~~~~~~
    386 
    387 GNUnet (currently) uses four different types of URIs for file-sharing.
    388 They all begin with \"gnunet://fs/\". This section describes the four
    389 different URI types in detail.
    390 
    391 For FS URIs empty KEYWORDs are not allowed. Quotes are allowed to denote
    392 whitespace between words. Keywords must contain a balanced number of
    393 double quotes. Doubles quotes can not be used in the actual keywords.
    394 This means that the string '\"\"foo bar\"\"' will be turned into two
    395 OR-ed keywords 'foo' and 'bar', not into '\"foo bar\"'.
    396 
    397 .. _Encoding-of-hash-values-in-URIs:
    398 
    399 Encoding of hash values in URIs
    400 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    401 
    402 Most URIs include some hash values. Hashes are encoded using base32hex
    403 (RFC 2938).
    404 
    405 chk-uri
    406 .. _Content-Hash-Key-_0028chk_0029:
    407 
    408 Content Hash Key (chk)
    409 ^^^^^^^^^^^^^^^^^^^^^^
    410 
    411 A chk-URI is used to (uniquely) identify a file or directory and to
    412 allow peers to download the file. Files are stored in GNUnet as a tree
    413 of encrypted blocks. The chk-URI thus contains the information to
    414 download and decrypt those blocks. A chk-URI has the format
    415 \"gnunet://fs/chk/KEYHASH.QUERYHASH.SIZE\". Here, \"SIZE\" is the size
    416 of the file (which allows a peer to determine the shape of the tree),
    417 KEYHASH is the key used to decrypt the file (also the hash of the
    418 plaintext of the top block) and QUERYHASH is the query used to request
    419 the top-level block (also the hash of the encrypted block).
    420 
    421 loc-uri
    422 .. _Location-identifiers-_0028loc_0029:
    423 
    424 Location identifiers (loc)
    425 ^^^^^^^^^^^^^^^^^^^^^^^^^^
    426 
    427 For non-anonymous file-sharing, loc-URIs are used to specify which peer
    428 is offering the data (in addition to specifying all of the data from a
    429 chk-URI). Location identifiers include a digital signature of the peer
    430 to affirm that the peer is truly the origin of the data. The format is
    431 \"gnunet://fs/loc/KEYHASH.QUERYHASH.SIZE.PEER.SIG.EXPTIME\". Here,
    432 \"PEER\" is the public key of the peer (in GNUnet format in base32hex),
    433 SIG is the RSA signature (in GNUnet format in base32hex) and EXPTIME
    434 specifies when the signature expires (in milliseconds after 1970).
    435 
    436 ksk-uri
    437 .. _Keyword-queries-_0028ksk_0029:
    438 
    439 Keyword queries (ksk)
    440 ^^^^^^^^^^^^^^^^^^^^^
    441 
    442 A keyword-URI is used to specify that the desired operation is the
    443 search using a particular keyword. The format is simply
    444 \"gnunet://fs/ksk/KEYWORD\". Non-ASCII characters can be specified using
    445 the typical URI-encoding (using hex values) from HTTP. \"+\" can be used
    446 to specify multiple keywords (which are then logically \"OR\"-ed in the
    447 search, results matching both keywords are given a higher rank):
    448 \"gnunet://fs/ksk/KEYWORD1+KEYWORD2\". ksk-URIs must not begin or end
    449 with the plus ('+') character. Furthermore they must not contain '++'.
    450 
    451 sks-uri
    452 .. _Namespace-content-_0028sks_0029:
    453 
    454 Namespace content (sks)
    455 ^^^^^^^^^^^^^^^^^^^^^^^
    456 
    457 **Please note that the text in this subsection is outdated and needs**
    458 **to be rewritten for version 0.10!** **This especially concerns the
    459 terminology of Pseudonym/Ego/Identity.**
    460 
    461 Namespaces are sets of files that have been approved by some (usually
    462 pseudonymous) user --- typically by that user publishing all of the
    463 files together. A file can be in many namespaces. A file is in a
    464 namespace if the owner of the ego (aka the namespace's private key)
    465 signs the CHK of the file cryptographically. An SKS-URI is used to
    466 search a namespace. The result is a block containing meta data, the CHK
    467 and the namespace owner's signature. The format of a sks-URI is
    468 \"gnunet://fs/sks/NAMESPACE/IDENTIFIER\". Here, \"NAMESPACE\" is the
    469 public key for the namespace. \"IDENTIFIER\" is a freely chosen keyword
    470 (or password!). A commonly used identifier is \"root\" which by
    471 convention refers to some kind of index or other entry point into the
    472 namespace.
    473 
    474