fs.rst (19047B)
1 File-sharing 2 ------------ 3 4 .. note:: 5 6 File-sharing is under development and has known design issues that affect 7 usability and performance. See :doc:`/status`. 8 9 This chapter documents the GNUnet file-sharing application. The original 10 file-sharing implementation for GNUnet was designed to provide anonymous 11 file-sharing. However, over time, we have also added support for 12 non-anonymous file-sharing (which can provide better performance). 13 Anonymous and non-anonymous file-sharing are quite integrated in GNUnet 14 and, except for routing, share most of the concepts and implementation. 15 There are three primary file-sharing operations: publishing, searching 16 and downloading. For each of these operations, the user specifies an 17 anonymity level. If both the publisher and the searcher/downloader 18 specify “no anonymity”, non-anonymous file-sharing is used. If either 19 user specifies some desired degree of anonymity, anonymous file-sharing 20 will be used. 21 22 After a short introduction, we will first look at the various concepts 23 in GNUnet’s file-sharing implementation. Then, we will discuss specifics 24 as to how they impact users that publish, search or download files. 25 26 Searching 27 ~~~~~~~~~ 28 29 The command ``gnunet-search`` can be used to search for content on 30 GNUnet. The format is: 31 32 :: 33 34 $ gnunet-search [-t TIMEOUT] KEYWORD 35 36 The ``-t`` option specifies that the query should timeout after 37 approximately ``TIMEOUT`` seconds. A value of zero (“0”) is interpreted 38 as no timeout, which is the default. In this case, gnunet-search will 39 never terminate (unless you press CTRL-C). 40 41 If multiple words are passed as keywords, they will all be considered 42 optional. Prefix keywords with a “+” to make them mandatory. 43 44 Note that searching using: 45 46 :: 47 48 $ gnunet-search Das Kapital 49 50 is not the same as searching for 51 52 :: 53 54 $ gnunet-search "Das Kapital" 55 56 as the first will match files shared under the keywords “Das” or 57 “Kapital” whereas the second will match files shared under the keyword 58 “Das Kapital”. 59 60 Search results are printed like this: 61 62 :: 63 64 #15: 65 gnunet-download -o "COPYING" gnunet://fs/chk/PGK8M...3EK130.75446 66 67 The whole line is the command you would have to enter to download the 68 file. The first argument passed to ``-o`` is the suggested filename (you 69 may change it to whatever you like). It is followed by the key for 70 decrypting the file, the query for searching the file, a checksum (in 71 hexadecimal) finally the size of the file in bytes. 72 73 Downloading 74 ~~~~~~~~~~~ 75 76 In order to download a file, you need the whole line returned by 77 gnunet-search. You can then use the tool ``gnunet-download`` to obtain 78 the file: 79 80 :: 81 82 $ gnunet-download -o <FILENAME> <GNUNET-URL> 83 84 ``FILENAME`` specifies the name of the file where GNUnet is supposed to 85 write the result. Existing files are overwritten. If the existing file 86 contains blocks that are identical to the desired download, those blocks 87 will not be downloaded again (automatic resume). 88 89 If you want to download the GPL from the previous example, you do the 90 following: 91 92 :: 93 94 $ gnunet-download -o "COPYING" gnunet://fs/chk/PGK8M...3EK130.75446 95 96 If you ever have to abort a download, you can continue it at any time by 97 re-issuing gnunet-download with the same filename. In that case, GNUnet 98 will **not** download blocks again that are already present. 99 100 GNUnet’s file-encoding mechanism will ensure file integrity, even if the 101 existing file was not downloaded from GNUnet in the first place. 102 103 You may want to use the ``-V`` switch to turn on verbose reporting. In 104 this case, gnunet-download will print the current number of bytes 105 downloaded whenever new data was received. 106 107 Publishing 108 ~~~~~~~~~~ 109 110 The command ``gnunet-publish`` can be used to add content to the 111 network. The basic format of the command is: 112 113 :: 114 115 $ gnunet-publish [-n] [-k KEYWORDS]* [-m TYPE:VALUE] FILENAME 116 117 For example: 118 119 :: 120 121 $ gnunet-publish -m "description:GNU License" -k gpl -k test -m "mimetype:text/plain" COPYING 122 123 The option ``-k`` is used to specify keywords for the file that should 124 be inserted. You can supply any number of keywords, and each of the 125 keywords will be sufficient to locate and retrieve the file. Please note 126 that you must use the ``-k`` option more than once – one for each 127 expression you use as a keyword for the filename. 128 129 The ``-m`` option is used to specify meta-data, such as descriptions. 130 You can use ``-m`` multiple times. The ``TYPE`` passed must be from the 131 list of meta-data types known to libextractor. You can obtain this list 132 by running ``extract -L``. Use quotes around the entire meta-data 133 argument if the value contains spaces. The meta-data is displayed to 134 other users when they select which files to download. The meta-data and 135 the keywords are optional and may be inferred using GNU libextractor. 136 137 ``gnunet-publish`` has a few additional options to handle namespaces and 138 directories. Refer to the man-page for details. 139 140 Indexing vs Inserting 141 ~~~~~~~~~~~~~~~~~~~~~ 142 143 By default, GNUnet indexes a file instead of making a full copy. This is 144 much more efficient, but requires the file to stay unaltered at the 145 location where it was when it was indexed. If you intend to move, delete 146 or alter a file, consider using the option ``-n`` which will force 147 GNUnet to make a copy of the file in the database. 148 149 Since it is much less efficient, this is strongly discouraged for large 150 files. When GNUnet indexes a file (default), GNUnet does **not** create 151 an additional encrypted copy of the file but just computes a summary (or 152 index) of the file. That summary is approximately two percent of the 153 size of the original file and is stored in GNUnet’s database. Whenever a 154 request for a part of an indexed file reaches GNUnet, this part is 155 encrypted on-demand and send out. This way, there is no need for an 156 additional encrypted copy of the file to stay anywhere on the drive. 157 This is different from other systems, such as Freenet, where each file 158 that is put online must be in Freenet’s database in encrypted format, 159 doubling the space requirements if the user wants to preserve a directly 160 accessible copy in plaintext. 161 162 Thus indexing should be used for all files where the user will keep 163 using this file (at the location given to gnunet-publish) and does not 164 want to retrieve it back from GNUnet each time. If you want to remove a 165 file that you have indexed from the local peer, use the tool 166 gnunet-unindex to un-index the file. 167 168 The option ``-n`` may be used if the user fears that the file might be 169 found on their drive (assuming the computer comes under the control of 170 an adversary). When used with the ``-n`` flag, the user has a much 171 better chance of denying knowledge of the existence of the file, even if 172 it is still (encrypted) on the drive and the adversary is able to crack 173 the encryption (e.g. by guessing the keyword). 174 175 .. _fs_002dConcepts: 176 177 Concepts 178 ~~~~~~~~ 179 180 For better results with filesharing it is useful to understand the 181 following concepts. In addition to anonymous routing GNUnet attempts to 182 give users a better experience in searching for content. GNUnet uses 183 cryptography to safely break content into smaller pieces that can be 184 obtained from different sources without allowing participants to corrupt 185 files. GNUnet makes it difficult for an adversary to send back bogus 186 search results. GNUnet enables content providers to group related 187 content and to establish a reputation. Furthermore, GNUnet allows 188 updates to certain content to be made available. This section is 189 supposed to introduce users to the concepts that are used to achieve 190 these goals. 191 192 .. _Files: 193 194 Files 195 ^^^^^ 196 197 A file in GNUnet is just a sequence of bytes. Any file-format is allowed 198 and the maximum file size is theoretically :math:`2^64 - 1` bytes, 199 except that it would take an impractical amount of time to share such a 200 file. GNUnet itself never interprets the contents of shared files, 201 except when using GNU libextractor to obtain keywords. 202 203 .. _Keywords: 204 205 Keywords 206 ^^^^^^^^ 207 208 Keywords are the most simple mechanism to find files on GNUnet. Keywords 209 are **case-sensitive** and the search string must always match 210 **exactly** the keyword used by the person providing the file. Keywords 211 are never transmitted in plaintext. The only way for an adversary to 212 determine the keyword that you used to search is to guess it (which then 213 allows the adversary to produce the same search request). Since 214 providing keywords by hand for each shared file is tedious, GNUnet uses 215 GNU libextractor to help automate this process. Starting a keyword 216 search on a slow machine can take a little while since the keyword 217 search involves computing a fresh RSA key to formulate the request. 218 219 .. _Directories: 220 221 Directories 222 ^^^^^^^^^^^ 223 224 A directory in GNUnet is a list of file identifiers with meta data. The 225 file identifiers provide sufficient information about the files to allow 226 downloading the contents. Once a directory has been created, it cannot 227 be changed since it is treated just like an ordinary file by the 228 network. Small files (of a few kilobytes) can be inlined in the 229 directory, so that a separate download becomes unnecessary. 230 231 Directories are shared just like ordinary files. If you download a 232 directory with ``gnunet-download``, you can use ``gnunet-directory`` to 233 list its contents. The canonical extension for GNUnet directories when 234 stored as files in your local file-system is \".gnd\". The contents of a 235 directory are URIs and meta data. The URIs contain all the information 236 required by ``gnunet-download`` to retrieve the file. The meta data 237 typically includes the mime-type, description, a filename and other meta 238 information, and possibly even the full original file (if it was small). 239 240 .. _Egos-and-File_002dSharing: 241 242 Egos and File-Sharing 243 ^^^^^^^^^^^^^^^^^^^^^ 244 245 When sharing files, it is sometimes desirable to build a reputation as a 246 source for quality information. With egos, publishers can 247 (cryptographically) sign files, thereby demonstrating that various files 248 were published by the same entity. An ego thus allows users to link 249 different publication events, thereby deliberately reducing anonymity to 250 pseudonymity. 251 252 Egos used in GNUnet's file-sharing for such pseudonymous publishing also 253 correspond to the egos used to identify and sign zones in the GNU Name 254 System. However, if the same ego is used for file-sharing and for a GNS 255 zone, this will weaken the privacy assurances provided by the anonymous 256 file-sharing protocol. 257 258 Note that an ego is NOT bound to a GNUnet peer. There can be multiple 259 egos for a single user, and users could (theoretically) share the 260 private keys of an ego by copying the respective private keys. 261 262 .. _Namespaces: 263 264 Namespaces 265 ^^^^^^^^^^ 266 267 A namespace is a set of files that were signed by the same ego. Today, 268 namespaces are implemented independently of GNS zones, but in the future 269 we plan to merge the two such that a GNS zone can basically contain 270 files using a file-sharing specific record type. 271 272 Files (or directories) that have been signed and placed into a namespace 273 can be updated. Updates are identified as authentic if the same secret 274 key was used to sign the update. 275 276 .. _Advertisements: 277 278 Advertisements 279 ^^^^^^^^^^^^^^ 280 281 Advertisements are used to notify other users about the existence of a 282 namespace. Advertisements are propagated using the normal keyword 283 search. When an advertisement is received (in response to a search), the 284 namespace is added to the list of namespaces available in the 285 namespace-search dialogs of gnunet-fs-gtk and printed by 286 ``gnunet-identity``. Whenever a namespace is created, an appropriate 287 advertisement can be generated. The default keyword for the advertising 288 of namespaces is \"namespace\". 289 290 .. _Anonymity-level: 291 292 Anonymity level 293 ^^^^^^^^^^^^^^^ 294 295 The anonymity level determines how hard it should be for an adversary to 296 determine the identity of the publisher or the searcher/downloader. An 297 anonymity level of zero means that anonymity is not required. The 298 default anonymity level of \"1\" means that anonymous routing is 299 desired, but no particular amount of cover traffic is necessary. A 300 powerful adversary might thus still be able to deduce the origin of the 301 traffic using traffic analysis. Specifying higher anonymity levels 302 increases the amount of cover traffic required. 303 304 The specific numeric value (for anonymity levels above 1) is simple: 305 Given an anonymity level L (above 1), each request FS makes on your 306 behalf must be hidden in L-1 equivalent requests of cover traffic 307 (traffic your peer routes for others) in the same time-period. The 308 time-period is twice the average delay by which GNUnet artificially 309 delays traffic. 310 311 While higher anonymity levels may offer better privacy, they can also 312 significantly hurt performance. 313 314 .. _Content-Priority: 315 316 Content Priority 317 ^^^^^^^^^^^^^^^^ 318 319 Depending on the peer's configuration, GNUnet peers migrate content 320 between peers. Content in this sense are individual blocks of a file, 321 not necessarily entire files. When peers run out of space (due to local 322 publishing operations or due to migration of content from other peers), 323 blocks sometimes need to be discarded. GNUnet first always discards 324 expired blocks (typically, blocks are published with an expiration of 325 about two years in the future; this is another option). If there is 326 still not enough space, GNUnet discards the blocks with the lowest 327 priority. The priority of a block is decided by its popularity (in terms 328 of requests from peers we trust) and, in case of blocks published 329 locally, the base-priority that was specified by the user when the block 330 was published initially. 331 332 .. _Replication: 333 334 Replication 335 ^^^^^^^^^^^ 336 337 When peers migrate content to other systems, the replication level of a 338 block is used to decide which blocks need to be migrated most urgently. 339 GNUnet will always push the block with the highest replication level 340 into the network, and then decrement the replication level by one. If 341 all blocks reach replication level zero, the selection is simply random. 342 343 .. _Namespace-Management: 344 345 Namespace Management 346 ~~~~~~~~~~~~~~~~~~~~ 347 348 The ``gnunet-identity`` tool can be used to create egos. By default, 349 ``gnunet-identity --display`` simply lists all locally available egos. 350 351 .. _Creating-Egos: 352 353 Creating Egos 354 ^^^^^^^^^^^^^ 355 356 With the ``--create=NICK`` option it can also be used to create a new 357 ego. An ego is the virtual identity of the entity in control of a 358 namespace or GNS zone. Anyone can create any number of egos. The 359 provided NICK name automatically corresponds to a GNU Name System domain 360 name. Thus, henceforth name resolution for any name ending in ".NICK" 361 will use the NICK's zone. You should avoid using NICKs that collide with 362 well-known DNS names. 363 364 Currently, the IDENTITY subsystem supports two types of identity keys: 365 ECDSA and EdDSA. By default, ECDSA identities are creates with ECDSA 366 keys. In order to create an identity with EdDSA keys, you can use the 367 ``--eddsa`` flag. 368 369 .. _Deleting-Egos: 370 371 Deleting Egos 372 ^^^^^^^^^^^^^ 373 374 With the ``-D NICK`` option egos can be deleted. Once the ego has been 375 deleted it is impossible to add content to the corresponding namespace 376 or zone. However, the existing GNS zone data is currently not dropped. 377 This may change in the future. 378 379 Deleting the pseudonym does not make the namespace or any content in it 380 unavailable. 381 382 .. _File_002dSharing-URIs: 383 384 File-Sharing URIs 385 ~~~~~~~~~~~~~~~~~ 386 387 GNUnet (currently) uses four different types of URIs for file-sharing. 388 They all begin with \"gnunet://fs/\". This section describes the four 389 different URI types in detail. 390 391 For FS URIs empty KEYWORDs are not allowed. Quotes are allowed to denote 392 whitespace between words. Keywords must contain a balanced number of 393 double quotes. Doubles quotes can not be used in the actual keywords. 394 This means that the string '\"\"foo bar\"\"' will be turned into two 395 OR-ed keywords 'foo' and 'bar', not into '\"foo bar\"'. 396 397 .. _Encoding-of-hash-values-in-URIs: 398 399 Encoding of hash values in URIs 400 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 401 402 Most URIs include some hash values. Hashes are encoded using base32hex 403 (RFC 2938). 404 405 chk-uri 406 .. _Content-Hash-Key-_0028chk_0029: 407 408 Content Hash Key (chk) 409 ^^^^^^^^^^^^^^^^^^^^^^ 410 411 A chk-URI is used to (uniquely) identify a file or directory and to 412 allow peers to download the file. Files are stored in GNUnet as a tree 413 of encrypted blocks. The chk-URI thus contains the information to 414 download and decrypt those blocks. A chk-URI has the format 415 \"gnunet://fs/chk/KEYHASH.QUERYHASH.SIZE\". Here, \"SIZE\" is the size 416 of the file (which allows a peer to determine the shape of the tree), 417 KEYHASH is the key used to decrypt the file (also the hash of the 418 plaintext of the top block) and QUERYHASH is the query used to request 419 the top-level block (also the hash of the encrypted block). 420 421 loc-uri 422 .. _Location-identifiers-_0028loc_0029: 423 424 Location identifiers (loc) 425 ^^^^^^^^^^^^^^^^^^^^^^^^^^ 426 427 For non-anonymous file-sharing, loc-URIs are used to specify which peer 428 is offering the data (in addition to specifying all of the data from a 429 chk-URI). Location identifiers include a digital signature of the peer 430 to affirm that the peer is truly the origin of the data. The format is 431 \"gnunet://fs/loc/KEYHASH.QUERYHASH.SIZE.PEER.SIG.EXPTIME\". Here, 432 \"PEER\" is the public key of the peer (in GNUnet format in base32hex), 433 SIG is the RSA signature (in GNUnet format in base32hex) and EXPTIME 434 specifies when the signature expires (in milliseconds after 1970). 435 436 ksk-uri 437 .. _Keyword-queries-_0028ksk_0029: 438 439 Keyword queries (ksk) 440 ^^^^^^^^^^^^^^^^^^^^^ 441 442 A keyword-URI is used to specify that the desired operation is the 443 search using a particular keyword. The format is simply 444 \"gnunet://fs/ksk/KEYWORD\". Non-ASCII characters can be specified using 445 the typical URI-encoding (using hex values) from HTTP. \"+\" can be used 446 to specify multiple keywords (which are then logically \"OR\"-ed in the 447 search, results matching both keywords are given a higher rank): 448 \"gnunet://fs/ksk/KEYWORD1+KEYWORD2\". ksk-URIs must not begin or end 449 with the plus ('+') character. Furthermore they must not contain '++'. 450 451 sks-uri 452 .. _Namespace-content-_0028sks_0029: 453 454 Namespace content (sks) 455 ^^^^^^^^^^^^^^^^^^^^^^^ 456 457 **Please note that the text in this subsection is outdated and needs** 458 **to be rewritten for version 0.10!** **This especially concerns the 459 terminology of Pseudonym/Ego/Identity.** 460 461 Namespaces are sets of files that have been approved by some (usually 462 pseudonymous) user --- typically by that user publishing all of the 463 files together. A file can be in many namespaces. A file is in a 464 namespace if the owner of the ego (aka the namespace's private key) 465 signs the CHK of the file cryptographically. An SKS-URI is used to 466 search a namespace. The result is a block containing meta data, the CHK 467 and the namespace owner's signature. The format of a sks-URI is 468 \"gnunet://fs/sks/NAMESPACE/IDENTIFIER\". Here, \"NAMESPACE\" is the 469 public key for the namespace. \"IDENTIFIER\" is a freely chosen keyword 470 (or password!). A commonly used identifier is \"root\" which by 471 convention refers to some kind of index or other entry point into the 472 namespace. 473 474