commit 280997ce59cc80b8dd2d2782190083825d710715
parent 18abc2c1a1af23b4a313561467112f6be786acad
Author: Martin Schanzenbach <schanzen@gnunet.org>
Date: Wed, 18 Nov 2026 20:13:59 +0100
Reorganize docs
Diffstat:
27 files changed, 1771 insertions(+), 3472 deletions(-)
diff --git a/about.rst b/about.rst
@@ -212,7 +212,7 @@ might see it in various places:
UAT1S6PMPITLBKSJ2DGV341JI6KF7B66AC4JVCN9811NNEGQLUN0
-You can find your peer identity by running ``gnunet-core``.
+You can find your peer identity by running ``gnunet-pils --once``.
Almost all peer-to-peer communications in GNUnet are between mutually
authenticated peers.
@@ -240,7 +240,7 @@ International Symposium on Cluster Computing and the Grid (GRID 2003),
Security goals and threat model
--------------------------
+-------------------------------
GNUnet is designed as to subsist in the face of a strong adversaries (malicous, bad actors).
This includes, in decending strength, malicous
@@ -270,9 +270,8 @@ traffic analysis much harder. Naturally, for some applications, it may
still be desirable if even neighbors cannot determine the concrete
contents of a message. In GNUnet, this problem is addressed by the
specific application-level protocols. See for example the following
-sections: `Anonymity <about.md#anonymity>`__, see `How file-sharing
-achieves Anonymity <about.md#how-file-sharing-achieves-anonymity>`__,
-and see `Deniability <about.md#deniability>`__.
+sections: `Anonymity`_, `How file-sharing achieves Anonymity`_,
+and `Deniability`_.
GNUnet attemtps to satisfy the following security goals in the face of those adversaries:
@@ -284,15 +283,22 @@ GNUnet attemtps to satisfy the following security goals in the face of those adv
From the lowest layer to the applications layer, the securty goals and associated subsystems are:
1. Base layer (Communicators/TRANSPORT): This layer optionally provides steganographic and ad-hoc security guarantees against external adversaries that largely depend on the communicator(s) used. For example, use of the HTTP3/QUIC communicator will use TLS and try to validate a certificate signed by the peer we want to connect to. Other communicators may not provide the same properties.
- - QUIC Communicator: Appears to be a regular TLS connection (EdDSA/X25519).
- - TCP/UDP Communicator: Uses Diffie-Hellman with Elligator (`LSD 0011 <https://lsd.gnunet.org/lsd0011/>`_) to look like random noise.
+
+ - QUIC Communicator: Appears to be a regular TLS connection (EdDSA/X25519).
+ - TCP/UDP Communicator: Uses Diffie-Hellman with Elligator (`LSD 0011 <https://lsd.gnunet.org/lsd0011/>`_) to look like random noise.
+
2. Peer connectivity and routing layer (CORE, R5N): This layer provides a secure channel between two (physically) connected peers. Peers are mutually authenticated and a secure cryptographic channel is established, but there is no particular trust required between the communication partners. It does not assume any security guarantees from the previous layer. It provides confidential communication in the face of an external adversary. The R5N uses this layer to establish an overlay network (DHT).
- - CORE: DTLS-style KEMTLS called CAKE with EdDSA and X25519. Specification: `LSD 0012 <https://lsd.gnunet.org/lsd0012/>`_
- - R5N: EdDSA signatures for route recording: `LSD 0004 <https://lsd.gnunet.org/lsd0004/>`_
+
+ - CORE: DTLS-style KEMTLS called CAKE with EdDSA and X25519. Specification: `LSD 0012 <https://lsd.gnunet.org/lsd0012/>`_
+ - R5N: EdDSA signatures for route recording: `LSD 0004 <https://lsd.gnunet.org/lsd0004/>`_
+
3. Peer connectivity layer (CADET): This layer provides an end-to-end secure secure cryptographic channel between two peers. It is assumed that this channel is established between to peers that share a strong trust relationship.
- - CADET: Axolotl 3DH with EdDSA Peer Identities to provide perfect forward secrecy, post-compromise security, secure out-of-order delivery and participant repudication (deniability).
+
+ - CADET: Axolotl 3DH with EdDSA Peer Identities to provide perfect forward secrecy, post-compromise security, secure out-of-order delivery and participant repudication (deniability).
+
4. Application layer: Each :ref:`subsystem <subsystems>` of GNUnet incorporates its own security mechanism taking the existing baseline of the GNUnet network as well as the adversary model into account. See the respective section in the User handbook.
- - GNS: Resource records are signed using EDDSA (or EdDSA) for data origin authentication and encrypted using AES-CTR (or EdDSA+XSalsa20-Poly1305) to achieve data confidentiality against certain adversaries. Public keys are blinded to prevent censorship. Specification: `LSD 0001 <https://lsd.gnunet.org/lsd0012/>`_
+
+ - GNS: Resource records are signed using ECDSA or EdDSA (depending on the zone type) for data origin authentication and encrypted using AES-CTR (ECDSA zones) or XSalsa20-Poly1305 (EdDSA zones) to achieve data confidentiality against certain adversaries. Public keys are blinded to prevent censorship. Specification: `LSD 0001 <https://lsd.gnunet.org/lsd0001/>`_ (:rfc:`9498`)
Cryptography
@@ -306,7 +312,7 @@ implement its protocols. The primary primtives are:
- Digital signatures: For Peer Identities and general (data origin) authentication. Scheme: EdDSA.
- Key exchange and KEMs: For handshakes. Schemes: X25519 (with Ed25519-to-Curve25519 transformations where necessary, such as CORE).
-- Blindable signature keys: For the GNU Name System. Schemes: EDDSA and EdDSA.
+- Blindable signature keys: For the GNU Name System. Schemes: ECDSA and EdDSA.
- Blind signatures: For blind signing. Primarily used by GNU Taler. Scheme: RSA-FDH.
- Symmetric encryption: For secure communication. Schemes: XSalsa20-Poly1305, AES (to be phased out in favor of AEGIS where possible).
- Public-key encryption: To send encrypted messages. Schemes: HPKE (RFC 9180), only DHKEM(X25519, HKDF-SHA256), HKDF-SHA256, ChaCha20-Poly1305.
diff --git a/conf.py b/conf.py
@@ -16,11 +16,12 @@
# -- Project information -----------------------------------------------------
+import datetime
import sys
import os
project = 'GNUnet'
-copyright = '2024, GNUnet Project'
+copyright = f'{datetime.date.today().year}, GNUnet Project'
author = 'GNUnet Project'
sys.path.append(os.path.abspath("_exts"))
@@ -75,8 +76,8 @@ man_make_section_directory = False
man_pages = [
('about', 'gnunet', 'About GNUnet', '', '1'),
('installing', 'gnunet', 'Installing GNUnet', '', '2'),
- ('users/start', 'gnunet', 'Starting and stopping', '', '3'),
- ('users/gns', 'gnunet', 'The GNU Name System', '', '4'),
+ ('users/start', 'gnunet', 'Running your peer', '', '3'),
+ ('gns/index', 'gnunet', 'The GNU Name System', '', '4'),
('users/reclaim', 'gnunet', 're:claimID', '', '5'),
('users/fs', 'gnunet', 'GNUnet File-Sharing', '', '6'),
('users/vpn', 'gnunet', 'GNUnet VPN', '', '7'),
@@ -85,9 +86,9 @@ man_pages = [
('developers/contributing', 'gnunet-dev', 'Contributing', '', '1'),
('developers/style', 'gnunet-dev', 'Style and workflow', '', '2'),
('developers/util', 'gnunet-dev', 'libgnunetutil', '', '3'),
- ('developers/rest/rest', 'gnunet-dev', 'REST API', '', '5'),
+ ('developers/rest-api/index', 'gnunet-dev', 'REST API', '', '5'),
('developers/tutorial', 'gnunet-dev', 'GNUnet Developer Tutorial', '', '6'),
- ('livingstandards', 'lsd', 'Living Standards', '', '1'),
+ ('developers/livingstandards', 'lsd', 'Living Standards', '', '1'),
]
# Add any paths that contain custom static files (such as style sheets) here,
@@ -118,10 +119,9 @@ rst_prolog = f"""
:class: bolditalic
"""
-if os.environ.get("GNUNET_VERSION"):
- rst_prolog = ".. |gnunet_version| replace:: " + os.environ["GNUNET_VERSION"]
-else:
- rst_prolog = ".. |gnunet_version| replace:: latest"
+# Append (do not overwrite) so that the roles defined above stay available.
+rst_prolog += "\n.. |gnunet_version| replace:: " + \
+ os.environ.get("GNUNET_VERSION", "latest") + "\n"
# Whitelist pattern for tags (set to None to ignore all tags)
#smv_tag_whitelist = r'^.*$'
diff --git a/developers/apis/cong.rst b/developers/apis/cong.rst
@@ -370,8 +370,8 @@ Unified Address Format for L2O and libp2p
libp2p peer-id multiaddress?, gnunet-hello
https://github.com/libp2p/specs/blob/master/peer-ids/peer-ids.md
https://github.com/libp2p/specs/blob/master/addressing/README.md#multiaddr-basics
- https://docs.gnunet.org/latest/users/subsystems.html#hellos
- https://docs.gnunet.org/latest/users/subsystems.html#hello-ng
+ https://docs.gnunet.org/latest/developers/subsystems.html#hellos
+ https://docs.gnunet.org/latest/developers/subsystems.html#hello-ng
https://docs.libp2p.io/concepts/fundamentals/addressing/
https://github.com/multiformats/multiaddr/
diff --git a/developers/index.rst b/developers/index.rst
@@ -47,6 +47,7 @@ new chapters, sections or insightful comments.
util.rst
architecture.rst
+ subsystems.rst
cryptographic-material.rst
stability.rst
apis/index.rst
diff --git a/developers/livingstandards.rst b/developers/livingstandards.rst
@@ -10,11 +10,24 @@ LSDs, however, "live" and always represent the current status of the implemented
protocol. The RFC in such cases only represents a snapshot in this case which
may or may not represent the current status of the corresponding LSD.
-- LSD 0000: Reserved
-- `LSD 0001 <https://lsd.gnunet.org/lsd0001>`__ [`RFC 9498 <https://www.rfc-editor.org/rfc/rfc9498.html>`__]: **The GNU Name System**
+The most mature specification is the one for the GNU Name System, which was
+published as :rfc:`9498` and can be used to write an independent,
+interoperable implementation.
+
+- LSD 0000: Reserved
+- `LSD 0001 <https://lsd.gnunet.org/lsd0001>`__ [:rfc:`9498`]: **The GNU Name System**
- `LSD 0002 <https://lsd.gnunet.org/lsd0002>`__: **re:claimID**
- `LSD 0003 <https://lsd.gnunet.org/lsd0003>`__: **Byzantine Fault Tolerant Set Reconciliation**
-- `LSD 0004 <https://lsd.gnunet.org/lsd0004>`__: **The R5N Distributed Hash Table**
+- `LSD 0004 <https://lsd.gnunet.org/lsd0004>`__ [`draft-schanzen-r5n <https://datatracker.ietf.org/doc/draft-schanzen-r5n/>`__]: **The R5N Distributed Hash Table**
- `LSD 0005 <https://lsd.gnunet.org/lsd0005>`__: **The GNU Name System DID Method**
- `LSD 0006 <https://lsd.gnunet.org/lsd0006>`__: **The 'taler' URI scheme for GNU Taler Wallet interactions**
- `LSD 0007 <https://lsd.gnunet.org/lsd0007>`__: **The GNUnet communicators**
+- `LSD 0008 <https://lsd.gnunet.org/lsd0008>`__: **SBOX**
+- `LSD 0009 <https://lsd.gnunet.org/lsd0009>`__: **The GNU Taler protocol**
+- `LSD 0011 <https://lsd.gnunet.org/lsd0011>`__: **Elligator-based HPKE KEM**
+- `LSD 0012 <https://lsd.gnunet.org/lsd0012>`__ [`draft-schanzen-cake <https://datatracker.ietf.org/doc/draft-schanzen-cake/>`__]: **CAKE** (the key exchange used by CORE)
+- `LSD 0013 <https://lsd.gnunet.org/lsd0013>`__: **The 'donau' URI scheme**
+- `LSD 0014 <https://lsd.gnunet.org/lsd0014>`__: **The Peer Identity Lifecycle Service (PILS)**
+
+The full list, including drafts not yet listed here, is at
+https://lsd.gnunet.org/.
diff --git a/users/subsystems.rst b/developers/subsystems.rst
diff --git a/faq.rst b/faq.rst
@@ -32,7 +32,7 @@ How does GNUnet compare to I2P?
I2P is written in Java, and has (asymmetric) tunnels using onion (or garlic) routing as the basis for various (anonymized) applications. I2P is largely used via a Web frontend.
Is GNUnet ready for use on production systems?
- A: GNUnet is still undergoing major development. It is largely not yet ready for usage beyond developers. Your mileage will vary depending on the functionality you use, but you will always likely run into issues with our current low-level transport system. We are currently in the process of rewriting it (Project "Transport Next Generation [TNG]")
+ A: Not yet. GNUnet is alpha software and the network protocols still change between major releases. The most mature component is the GNU Name System (GNS), which is ready for early adopters: start with the :doc:`quickstart`. The other applications are still in development. See :doc:`status` for details.
Is GNUnet build using distributed ledger technologies?
A: No. GNUnet is a new network protocol stack for building secure, distributed, and privacy-preserving applications. While a ledger could be built using GNUnet, we currently have no plans in doing so.
@@ -41,7 +41,7 @@ Features
--------
What can I do with GNUnet?
- A: GNUnet is a peer-to-peer framework, by which we mostly mean that it can do more than just one thing. Naturally, the implementation and documentation of some of the features that exist are more advanced than others. For users, GNUnet offers anonymous and non-anonymous file-sharing, a fully decentralized and censorship-resistant replacement for DNS and a mechanism for IPv4-IPv6 protocol translation and tunneling (NAT-PT with DNS-ALG). See also: Applications.
+ A: GNUnet is a peer-to-peer framework, by which we mostly mean that it can do more than just one thing. Naturally, the implementation and documentation of some of the features that exist are more advanced than others. The most mature is the :doc:`GNU Name System <gns/index>`, a fully decentralized and censorship-resistant alternative to DNS. In development are a self-sovereign identity provider (re:claimID), decentralized messaging, anonymous and non-anonymous file-sharing, and a mechanism for IPv4-IPv6 protocol translation and tunneling (NAT-PT with DNS-ALG). See :doc:`status`.
Is it possible to surf the WWW anonymously with GNUnet?
A: It is not possible use GNUnet for anonymous browsing at this point. We recommend that you use Tor for anonymous surfing.
@@ -58,94 +58,11 @@ Which operating systems does GNUnet run on?
GNU Name System
---------------
-Who runs the GNS root zone?
- A: Short answer: you. The long answer is the GNUnet will ship with a default configuration of top-level domains. The governance of this default configuration is not yet established. In any case, the user will be able to modify this configuration at will. We expect normal users to have no need to edit their own GNS zone(s) unless they host services themselves.
-
-Where is the per-user GNS database kept?
- A: The short answer is that the database is kept at the user's GNUnet peer. Now, a user may run multiple GNUnet peers, in which case the database could be kept at each peer (however, we don't have code for convenient replication). Similarly, multiple GNUnet peers can share one instance of the database --- the "gnunet-service-namestore" can be accessed from remote (via TCP). The actual data can be stored in a Postgres database, for which various replication options are again applicable. Ultimately, there are many options for how users can store (and secure) their GNS database.
-
-What is the expected average size of a GNS namestore database?
- A: Pretty small. Based on our user study where we looked at browser histories and the number of domains visited, we expect that GNS databases will only grow to a few tens of thousands of entries, small enough to fit even on mobile devices.
-
-Is GNS resistant to the attacks on DNS used by the US?
- A: We believe so, as there is no entity that any government could force to change the mapping for a name except for each individual user (and then the changes would only apply to the names that this user is the authority for). So if everyone used GNS, the only practical attack of a government would be to force the operator of a server to change the GNS records for his server to point elsewhere. However, if the owner of the private key for a zone is unavailable for enforcement, the respective zone cannot be changed and any other zone delegating to this zone will achieve proper resolution.
-
-
-How does GNS compare to other name systems?
- A: A scientific paper on this topic `has been published <https://grothoff.org/christian/ns2018.pdf>`_ and below is a table from the publication. For detailed descriptions please refer to the paper.
-
-
-What is the difference between GNS and CoDoNS?
- A: CoDoNS decentralizes the DNS database (using a DHT) but preserves the authority structure of DNS. With CoDoNS, IANA/ICANN are still in charge, and there are still registrars that determine who owns a name.
- With GNS, we decentralize the database and also decentralize the responsibility for naming: each user runs their own personal root zone and is thus in complete control of the names they use. GNS also has many additional features (to keep names short and enable migration) which don't even make sense in the context of CoDoNS.
-
-What is the difference between GNS and SocialDNS?
- A: Like GNS, SocialDNS allows each user to create DNS mappings. However, with SocialDNS the mappings are shared through the social network and subjected to ranking. As the social relationships evolve, names can thus change in surprising ways. With GNS, names are primarily shared via delegation, and thus mappings will only change if the user responsible for the name (the authority) manually changes the record.
-
-What is the difference between GNS and ODDNS?
- A: ODDNS is primarily designed to bypass the DNS root zone and the TLD registries (such as those for ".com" and ".org"). Instead of using those, each user is expected to maintain a database of (second-level) domains (like "gnu.org") and the IP addresses of the respective name servers. Resolution will fail if the target name servers change IPs.
-
-What is the difference between GNS and Handshake?
- A: Handshake is a blockchain-based method for root zone governance. Hence, it does not address the name resolution process itself but delegates resolution into DNS after the initial TLD resolution. Not taking sustainablility considerations into account, Handshake could be used as an additional supporting GNS root zone governance model, but we currently do not have such plans in mind.
-
-What is the difference between GNS and TrickleDNS?
- A: TrickleDNS pushes ("critical") DNS records between DNS resolvers of participating domains to provide "better availability, lower query resolution times, and faster update propagation". Thus TrickleDNS is focused on defeating attacks on the availability (and performance) of record propagation in DNS, for example via DDoS attacks on DNS root servers. TrickleDNS is thus concerned with how to ensure distribution of authoritative records, and authority remains derived from the DNS hierarchy.
-
-Does GNS require real-world introduction (secure PKEY exchange) in the style of the PGP web of trust?
- A: For security, it is well known that an initial trust path between the two parties must exist. However, for applications where this is not required, weaker mechanisms can be used. For example, we have implemented a first-come-first-served (FCFS) authority which allows arbitrary users to register arbitrary names. The key of this authority is included with every GNUnet installation. Thus, any name registered with FCFS is in fact global and requires no further introduction. However, the security of these names depends entirely on the trustworthiness of the FCFS authority. The authority can be queried under the ".pin" TLD.
-
-How can a legitimate domain owner tell other people to not use his name in GNS?
- A: Names have no owners in GNS, so there cannot be a "legitimate" domain owner. Any user can claim any name (as his preferred name or "pseudonym") in his NICK record. Similarly, all other users can choose to ignore this preference and use a name of their choice (or even assign no name) for this user.
-
-Did you consider the privacy implications of making your personal GNS zone visible?
- A: Each record in GNS has a flag "private". Records are shared with other users (via DHT or zone transfers) only if this flag is not set. Thus, users have full control over what information about their zones is made public.
-
-Are "Legacy Host" (LEHO) records not going to be obsolete with IPv6?
- A: The question presumes that (a) virtual hosting is only necessary because of IPv4 address scarcity, and (b) that LEHOs are only useful in the context of virtual hosting. However, LEHOs are also useful to help with X.509 certificate validation (as they specify for which legacy hostname the certificate should be valid). Also, even with IPv6 fully deployed and "infinite" IP addresses being available, we're not sure that virtual hosting would disappear. Finally, we don't want to have to wait for IPv6 to become commonplace, GNS should work with today's networks.
-
-Why does GNS not use a trust metric or consensus to determine globally unique names?
- A: Trust metrics have the fundamental problem that they have thresholds. As trust relationships evolve, mappings would change their meaning as they cross each others thresholds. We decided that the resulting unpredictability of the resolution process was not acceptable. Furthermore, trust and consensus might be easy to manipulate by adversaries.
-
-
-How do you handle compromised zone keys in GNS?
- A: The owner of a private key can create a revocation message. This one can then be flooded throughout the overlay network, creating a copy at all peers. Before using a public key, peers check if that key has been revoked. All names that involve delegation via a revoked zone will then fail to resolve. Peers always automatically check for the existence of a revocation message when resolving names.
-
-Could the signing algorithm of GNS be upgraded in the future?
- A: Yes. In our efforts to standardize GNS, we have already modified the protocol to support alternative delegation records.
- Naturally, deployed GNS implementations would have to be updated to support the new signature scheme. The new scheme can then be run in parallel with the existing system by using a new record type to indicate the use of a different cipher system.
-
-How can a GNS zone maintain several name servers, e.g. for load balancing?
- A: We don't expect this to be necessary, as GNS records are stored (and replicated) in the R5N DHT. Thus the authority will typically not be contacted whenever clients perform a lookup. Even if the authority goes (temporarily) off-line, the DHT will cache the records for some time. However, should having multiple servers for a zone be considered truly necessary, the owner of the zone can simply run multiple peers (and share the zone's key and database among them).
-
-Why do you believe it is worth giving up unique names for censorship resistance?
- A: The GNU Name system offers an alternative to DNS that is censorship resistant. As with any security mechanism, this comes at a cost (names are not globally unique). To draw a parallel, HTTPS connections use more bandwidth and have higher latency than HTTP connections. Depending on your application, HTTPS may not be worth the cost. However, for users that are experiencing censorship (or are concerned about it), giving up globally unique names may very well be worth the cost. After all, what is a "globally" unique name worth, if it does not resolve?
-
-Why do you say that DNS is 'centralized' and 'distributed'?
- A: We say that DNS is 'centralized' because it has a central component / central point of failure --- the root zone and its management by IANA/ICANN. This centralization creates vulnerabilities. For example, the US government was able to reassign the management of the country-TLDs of Afganistan and Iraq during the wars at the beginning of the 21st century.
-
-How does GNS protect against layer-3 censorship?
- A: GNS does not directly help with layer-3 censorship, but it does help indirectly in two ways:
- Many websites today use virtual hosting, so blocking a particular IP address causes much more collateral damage than blocking a DNS name. It thus raises the cost of censorship.
- Existing layer-3 circumvention solutions (such as Tor) would benefit from a censorship resistant naming system. Accessing Tor's ".onion" namespace currently requires users to use unmemorable cryptographic identifiers. With nicer names, Tor and tor2web-like services would be even easier to use.
-
-Does GNS work with search engines?
- A: GNS creates no significant problems for search engines, as they can use GNS to perform name resolution as well as any normal user. Naturally, while we typically expect normal users to install custom software for name resolution, this is unlikely to work for search engines today. However, the DNS2GNS gateway allows search engines to use DNS to resolve GNS names, so they can still index GNS resources. However, as using DNS2GNS gateways breaks the cryptographic chain of trust, legacy search engines will obviously not obtain censorship-resistant names.
-
-How does GNS compare to the Unmanaged Internet Architecture (UIA)?
- A: UIA and GNS both share the same basic naming model, which actually originated with Rivest's SDSI. However, UIA is not concerned about integration with legacy applications and instead focuses on universal connectivity between a user's many machines. In contrast, GNS was designed to interoperate with DNS as much as possible, and to also work as much as possible with the existing Web infrastructure. UIA is not at all concerned about legacy systems (clean slate).
-
-Doesn't GNS increase the trusted-computing base compared to DNS(SEC)?
- A: First of all, in GNS you can explicitly see the trust chain, so you know if a name you are resolving belongs to a friend, or a friend-of-a-friend, and can thus decide how much you trust the result. Naturally, the trusted-computing base (TCB) can become arbitrarily large this way --- however, given the name length restriction, for an individual name it is always less than about 128 entities.
-
-How does GNS handle SRV/TLSA records where service and protocol are part of the domain name?
- A: When GNS splits a domain name into labels for resolution, it detects the "_Service._Proto" syntax, converts "Service" to the corresponding port number and "Proto" to the corresponding protocol number. The rest of the name is resolved as usual. Then, when the result is presented, GNS looks for the GNS-specific "BOX" record type. A BOX record is a record that contains another record (such as SRV or TLSA records) and adds a service and protocol number (and the original boxed record type) to it.
+Questions about the GNU Name System are answered in :ref:`gns-faq`.
Error messages
--------------
-I receive many "WARNING Calculated flow delay for X at Y for Z". Should I worry?
- A: Right now, this is expected and a known cause for high latency in GNUnet. We have started a major rewrite to address this and other problems, but until the Transport Next Generation (TNG) is ready, these warnings are expected.
-
Error opening ``/dev/net/tun: No such file or directory``?
A: If you get this error message, the solution is simple. Issue the following commands (as root) to create the required device file
diff --git a/gns/concepts.rst b/gns/concepts.rst
@@ -0,0 +1,210 @@
+.. _gns-concepts:
+
+Concepts
+========
+
+This page explains the handful of ideas you need to use GNS: zones, records,
+start zones, delegation and the record types GNS adds to those of DNS. For
+the complete, normative description see :rfc:`9498`.
+
+Zones and egos
+--------------
+
+A GNS *zone* is a set of records controlled by a key pair. Whoever holds
+the private key can add, change and delete records; everybody else
+identifies the zone by its public key. There is no registry: creating a
+zone is as simple as creating a key pair.
+
+In GNUnet, key pairs that you own are called **egos** and are managed with
+``gnunet-identity``. Every ego can serve as a GNS zone. Egos may stand for
+pseudonyms, organizations or your real-world identity; it is a good idea to
+use separate egos for separate activities.
+
+.. code-block:: console
+
+ $ gnunet-identity -C myzone
+ $ gnunet-identity -d
+ myzone - 000G0507HZ8M09QWA93N2YX9Y9357777C0HTPK7PA0HTKJS5GCXXPGKBWC - EdDSA
+
+The name ``myzone`` is only a local nickname for the ego, visible to you
+alone. The long string is the zone's public key: that is what identifies
+the zone to everybody else.
+
+New zones use EdDSA keys. Zones created with older versions may use ECDSA
+keys; these remain supported, but creating new ones is deprecated. The key
+type matters in one place: delegation records to EdDSA zones have the type
+``EDKEY``, and those to ECDSA zones the type ``PKEY``.
+
+Records
+-------
+
+A zone maps *labels* (such as ``www``) to *record sets*. Each record has
+
+- a **type**, such as ``A`` (IPv4 address) or ``EDKEY`` (delegation),
+- a **value**,
+- an **expiration time**, after which resolvers discard it, and
+- **flags**, most importantly whether the record is *public* or *private*.
+
+Records are **private by default**. Private records only resolve on your
+own peer; they are never published. Records marked *public* are signed,
+encrypted and published in the DHT so that others can resolve them. The
+special label ``@`` denotes the zone apex, the equivalent of the bare domain
+name in DNS.
+
+How names are resolved
+----------------------
+
+GNS names look like DNS names and are read from right to left:
+
+1. The rightmost part of the name selects a **start zone** (see below).
+ In ``www.alice.myzone``, that is your own zone ``myzone``.
+2. Each remaining label, right to left, is looked up in the current zone.
+ If the label holds a **delegation record**, resolution continues in the
+ zone it points to. Here ``alice`` in ``myzone`` delegates to Alice's
+ zone.
+3. The last label (``www``) is looked up in the final zone, and its records
+ are returned.
+
+Every record set is verified against the key of the zone that is supposed
+to have published it, so a peer that delivers a forged answer is detected.
+
+Start zones
+-----------
+
+In GNS you are the master of your own root: the mapping from name suffixes
+to zones that DNS delegates to ICANN is, in GNS, your own configuration.
+There are three kinds of start zones.
+
+Your own zones
+ Every ego is a start zone. Names ending in ``.myzone`` are resolved in
+ your zone ``myzone``.
+
+Configured suffixes
+ The ``[gns]`` configuration section maps name suffixes to zone keys.
+ GNUnet ships with a small default set, currently
+
+ - ``.pin.gns.alt``: a first-come, first-served registry run by GNUnet
+ e.V., where anyone can register a name (see :doc:`sharing`), and
+ - ``.gnunet.gns.alt``: the zone of the GNUnet project.
+
+ The default set is maintained in `GANA <https://gana.gnunet.org>`__ under
+ a `restrictive registration policy
+ <https://git.gnunet.org/gana.git/tree/gnu-name-system-default-tlds/POLICY>`__.
+ You can list the configured suffixes, and add your own, with
+ ``gnunet-config``:
+
+ .. code-block:: console
+
+ $ gnunet-config -s gns | grep '^\.'
+ $ gnunet-config -s gns -o .myfriend -V 000G0...ZONEKEY
+
+Zone keys (zTLDs)
+ A zone's public key can itself be used as the top-level domain. This
+ works without any configuration and is mostly useful in programs and
+ when exchanging names with people whose zone you have not (yet) added:
+
+ .. code-block:: console
+
+ $ gnunet-gns -u www.000G0507HZ8M09QWA93N2YX9Y9357777C0HTPK7PA0HTKJS5GCXXPGKBWC
+
+Delegation and petnames
+-----------------------
+
+Delegation is what makes GNS a *system* of names rather than a collection of
+private address books. If your friend Alice gives you her zone key, you
+store it under a label of your choice, say ``alice``, as an ``EDKEY`` record:
+
+.. code-block:: console
+
+ $ gnunet-namestore -z myzone -a -n alice -t EDKEY -V 000G0...ALICEKEY -e 1d
+
+From then on, every public record in Alice's zone is reachable below
+``alice.myzone``: her ``www`` record as ``www.alice.myzone``. If Alice in
+turn has a public delegation to Carol under ``carol``, then Carol's ``www``
+is ``www.carol.alice.myzone`` for you.
+
+The label you chose, ``alice``, is a **petname**: a name that is meaningful
+to you. Somebody else may know Alice's zone as ``ally`` or ``work``. Names
+in GNS are therefore not globally unique; they are relative to whoever
+resolves them. A zone can suggest a preferred name for itself with a
+``NICK`` record (``gnunet-namestore -z myzone -i alice``), but nobody is
+obliged to use it.
+
+Relative names
+--------------
+
+Some record types, for example ``CNAME``, ``MX`` and ``REDIRECT``, contain
+names. A name ending in ``.+`` is *relative to the zone it was found in*:
+the resolver replaces the ``+`` with that zone's key. If Alice's zone
+contains an ``MX`` record pointing to ``mail.+``, the answer you get is
+``mail.000G0...ALICEKEY``, which resolves correctly no matter what petname
+you use for Alice's zone. Relative names can also be used in links on web
+pages served under a GNS name.
+
+Record types
+------------
+
+GNS supports the common DNS record types, including ``A``, ``AAAA``,
+``CNAME``, ``MX``, ``TXT``, ``SRV`` and ``TLSA``. In addition it defines
+record types of its own, registered in `GANA
+<https://git.gnunet.org/gana.git/tree/gnu-name-system-record-types/registry.rec>`__.
+The ones you are most likely to meet are:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 20 80
+
+ * - Type
+ - Purpose
+ * - ``EDKEY``, ``PKEY``
+ - Delegation to another GNS zone (EdDSA or ECDSA key).
+ * - ``GNS2DNS``
+ - Delegation to a zone in DNS: the rest of the name is resolved via
+ the given DNS name servers.
+ * - ``NICK``
+ - The zone's preferred name for itself.
+ * - ``LEHO``
+ - The legacy (DNS) host name of a service, used for HTTP virtual
+ hosting and for validating TLS certificates.
+ * - ``BOX``
+ - Wraps ``SRV`` and ``TLSA`` records with their service and protocol,
+ so that names like ``_443._tcp.www`` work.
+ * - ``REDIRECT``
+ - Redirects resolution of a name to another name.
+ * - ``VPN``
+ - Points to a service offered by a GNUnet peer through the GNUnet VPN;
+ the resolver allocates a local IP address for it (see
+ :doc:`/users/vpn`).
+
+Several other types are used internally by re:claimID and the messenger.
+For the exact format and processing rules of all records see
+:rfc:`9498`.
+
+Security and privacy properties
+-------------------------------
+
+What GNS protects:
+
+- **Integrity and authenticity.** Record sets are signed with the zone key;
+ forged or modified answers are rejected.
+- **Query privacy.** Records are stored in the DHT under a key derived from
+ the zone key and the label, and encrypted with a key derived from the same
+ pair. Peers that store or forward them cannot tell which name they belong
+ to or what it resolves to, unless they already know both the zone key and
+ the label.
+- **Censorship resistance.** There is no registrar or root operator that
+ could be compelled to change or remove a name. Only the holder of the
+ private key can change a zone.
+
+What GNS does *not* protect against:
+
+- Someone who knows a zone's key can check whether a label they guess exists
+ in that zone. Do not rely on a label being secret.
+- If the private key of a zone is stolen, the thief can publish records in
+ your name. Prepare a revocation certificate in advance (see
+ :ref:`gns-revocation`).
+- A delegation is only as trustworthy as the way you obtained the key. A
+ name like ``www.carol.alice.myzone`` makes the chain of trust explicit:
+ you rely on Alice to have Carol's correct key.
+- GNS does not hide that your peer participates in GNUnet, and it does not
+ anonymize the connections you make *after* resolving a name.
diff --git a/gns/faq.rst b/gns/faq.rst
@@ -0,0 +1,83 @@
+.. _gns-faq:
+
+Frequently asked questions about GNS
+====================================
+
+Who runs the GNS root zone?
+ A: Short answer: you. The long answer is that GNUnet ships with a small default configuration of top-level domains (see :ref:`gns-concepts`), which is registered in GANA under a restrictive policy. In any case, the user can modify this configuration at will. We expect normal users to have no need to edit their own GNS zone(s) unless they host services themselves.
+
+Where is the per-user GNS database kept?
+ A: The short answer is that the database is kept at the user's GNUnet peer. Now, a user may run multiple GNUnet peers, in which case the database could be kept at each peer (however, we don't have code for convenient replication). Similarly, multiple GNUnet peers can share one instance of the database --- the "gnunet-service-namestore" can be accessed from remote (via TCP). The actual data can be stored in a Postgres database, for which various replication options are again applicable. Ultimately, there are many options for how users can store (and secure) their GNS database.
+
+What is the expected average size of a GNS namestore database?
+ A: Pretty small. Based on our user study where we looked at browser histories and the number of domains visited, we expect that GNS databases will only grow to a few tens of thousands of entries, small enough to fit even on mobile devices.
+
+Is GNS resistant to the attacks on DNS used by the US?
+ A: We believe so, as there is no entity that any government could force to change the mapping for a name except for each individual user (and then the changes would only apply to the names that this user is the authority for). So if everyone used GNS, the only practical attack of a government would be to force the operator of a server to change the GNS records for his server to point elsewhere. However, if the owner of the private key for a zone is unavailable for enforcement, the respective zone cannot be changed and any other zone delegating to this zone will achieve proper resolution.
+
+How does GNS compare to other name systems?
+ A: A scientific paper comparing GNS with other decentralized name systems `has been published <https://grothoff.org/christian/ns2018.pdf>`_. The answers below summarize the differences to some of them.
+
+What is the difference between GNS and CoDoNS?
+ A: CoDoNS decentralizes the DNS database (using a DHT) but preserves the authority structure of DNS. With CoDoNS, IANA/ICANN are still in charge, and there are still registrars that determine who owns a name.
+ With GNS, we decentralize the database and also decentralize the responsibility for naming: each user runs their own personal root zone and is thus in complete control of the names they use. GNS also has many additional features (to keep names short and enable migration) which don't even make sense in the context of CoDoNS.
+
+What is the difference between GNS and SocialDNS?
+ A: Like GNS, SocialDNS allows each user to create DNS mappings. However, with SocialDNS the mappings are shared through the social network and subjected to ranking. As the social relationships evolve, names can thus change in surprising ways. With GNS, names are primarily shared via delegation, and thus mappings will only change if the user responsible for the name (the authority) manually changes the record.
+
+What is the difference between GNS and ODDNS?
+ A: ODDNS is primarily designed to bypass the DNS root zone and the TLD registries (such as those for ".com" and ".org"). Instead of using those, each user is expected to maintain a database of (second-level) domains (like "gnu.org") and the IP addresses of the respective name servers. Resolution will fail if the target name servers change IPs.
+
+What is the difference between GNS and Handshake?
+ A: Handshake is a blockchain-based method for root zone governance. Hence, it does not address the name resolution process itself but delegates resolution into DNS after the initial TLD resolution. Not taking sustainablility considerations into account, Handshake could be used as an additional supporting GNS root zone governance model, but we currently do not have such plans in mind.
+
+What is the difference between GNS and TrickleDNS?
+ A: TrickleDNS pushes ("critical") DNS records between DNS resolvers of participating domains to provide "better availability, lower query resolution times, and faster update propagation". Thus TrickleDNS is focused on defeating attacks on the availability (and performance) of record propagation in DNS, for example via DDoS attacks on DNS root servers. TrickleDNS is thus concerned with how to ensure distribution of authoritative records, and authority remains derived from the DNS hierarchy.
+
+Does GNS require real-world introduction (secure PKEY exchange) in the style of the PGP web of trust?
+ A: For security, it is well known that an initial trust path between the two parties must exist. However, for applications where this is not required, weaker mechanisms can be used. For example, we have implemented a first-come-first-served (FCFS) authority which allows arbitrary users to register arbitrary names. The key of this authority is included with every GNUnet installation. Thus, any name registered with FCFS is in fact global and requires no further introduction. However, the security of these names depends entirely on the trustworthiness of the FCFS authority. The authority can be queried under the ".pin.gns.alt" TLD; see :ref:`gns-sharing`.
+
+How can a legitimate domain owner tell other people to not use his name in GNS?
+ A: Names have no owners in GNS, so there cannot be a "legitimate" domain owner. Any user can claim any name (as his preferred name or "pseudonym") in his NICK record. Similarly, all other users can choose to ignore this preference and use a name of their choice (or even assign no name) for this user.
+
+Did you consider the privacy implications of making your personal GNS zone visible?
+ A: Each record in GNS has a flag "private". Records are shared with other users (via DHT or zone transfers) only if this flag is not set. Thus, users have full control over what information about their zones is made public.
+
+Are "Legacy Host" (LEHO) records not going to be obsolete with IPv6?
+ A: The question presumes that (a) virtual hosting is only necessary because of IPv4 address scarcity, and (b) that LEHOs are only useful in the context of virtual hosting. However, LEHOs are also useful to help with X.509 certificate validation (as they specify for which legacy hostname the certificate should be valid). Also, even with IPv6 fully deployed and "infinite" IP addresses being available, we're not sure that virtual hosting would disappear. Finally, we don't want to have to wait for IPv6 to become commonplace, GNS should work with today's networks.
+
+Why does GNS not use a trust metric or consensus to determine globally unique names?
+ A: Trust metrics have the fundamental problem that they have thresholds. As trust relationships evolve, mappings would change their meaning as they cross each others thresholds. We decided that the resulting unpredictability of the resolution process was not acceptable. Furthermore, trust and consensus might be easy to manipulate by adversaries.
+
+How do you handle compromised zone keys in GNS?
+ A: The owner of a private key can create a revocation message. This one can then be flooded throughout the overlay network, creating a copy at all peers. Before using a public key, peers check if that key has been revoked. All names that involve delegation via a revoked zone will then fail to resolve. Peers always automatically check for the existence of a revocation message when resolving names.
+
+Could the signing algorithm of GNS be upgraded in the future?
+ A: Yes. In our efforts to standardize GNS, we have already modified the protocol to support alternative delegation records.
+ Naturally, deployed GNS implementations would have to be updated to support the new signature scheme. The new scheme can then be run in parallel with the existing system by using a new record type to indicate the use of a different cipher system.
+
+How can a GNS zone maintain several name servers, e.g. for load balancing?
+ A: We don't expect this to be necessary, as GNS records are stored (and replicated) in the R5N DHT. Thus the authority will typically not be contacted whenever clients perform a lookup. Even if the authority goes (temporarily) off-line, the DHT will cache the records for some time. However, should having multiple servers for a zone be considered truly necessary, the owner of the zone can simply run multiple peers (and share the zone's key and database among them).
+
+Why do you believe it is worth giving up unique names for censorship resistance?
+ A: The GNU Name system offers an alternative to DNS that is censorship resistant. As with any security mechanism, this comes at a cost (names are not globally unique). To draw a parallel, HTTPS connections use more bandwidth and have higher latency than HTTP connections. Depending on your application, HTTPS may not be worth the cost. However, for users that are experiencing censorship (or are concerned about it), giving up globally unique names may very well be worth the cost. After all, what is a "globally" unique name worth, if it does not resolve?
+
+Why do you say that DNS is 'centralized' and 'distributed'?
+ A: We say that DNS is 'centralized' because it has a central component / central point of failure --- the root zone and its management by IANA/ICANN. This centralization creates vulnerabilities. For example, the US government was able to reassign the management of the country-TLDs of Afganistan and Iraq during the wars at the beginning of the 21st century.
+
+How does GNS protect against layer-3 censorship?
+ A: GNS does not directly help with layer-3 censorship, but it does help indirectly in two ways:
+ Many websites today use virtual hosting, so blocking a particular IP address causes much more collateral damage than blocking a DNS name. It thus raises the cost of censorship.
+ Existing layer-3 circumvention solutions (such as Tor) would benefit from a censorship resistant naming system. Accessing Tor's ".onion" namespace currently requires users to use unmemorable cryptographic identifiers. With nicer names, Tor and tor2web-like services would be even easier to use.
+
+Does GNS work with search engines?
+ A: GNS creates no significant problems for search engines, as they can use GNS to perform name resolution as well as any normal user. Naturally, while we typically expect normal users to install custom software for name resolution, this is unlikely to work for search engines today. However, the DNS2GNS gateway allows search engines to use DNS to resolve GNS names, so they can still index GNS resources. However, as using DNS2GNS gateways breaks the cryptographic chain of trust, legacy search engines will obviously not obtain censorship-resistant names.
+
+How does GNS compare to the Unmanaged Internet Architecture (UIA)?
+ A: UIA and GNS both share the same basic naming model, which actually originated with Rivest's SDSI. However, UIA is not concerned about integration with legacy applications and instead focuses on universal connectivity between a user's many machines. In contrast, GNS was designed to interoperate with DNS as much as possible, and to also work as much as possible with the existing Web infrastructure. UIA is not at all concerned about legacy systems (clean slate).
+
+Doesn't GNS increase the trusted-computing base compared to DNS(SEC)?
+ A: First of all, in GNS you can explicitly see the trust chain, so you know if a name you are resolving belongs to a friend, or a friend-of-a-friend, and can thus decide how much you trust the result. Naturally, the trusted-computing base (TCB) can become arbitrarily large this way --- however, given the name length restriction, for an individual name it is always less than about 128 entities.
+
+How does GNS handle SRV/TLSA records where service and protocol are part of the domain name?
+ A: When GNS splits a domain name into labels for resolution, it detects the "_Service._Proto" syntax, converts "Service" to the corresponding port number and "Proto" to the corresponding protocol number. The rest of the name is resolved as usual. Then, when the result is presented, GNS looks for the GNS-specific "BOX" record type. A BOX record is a record that contains another record (such as SRV or TLSA records) and adds a service and protocol number (and the original boxed record type) to it.
diff --git a/gns/index.rst b/gns/index.rst
@@ -0,0 +1,135 @@
+.. _gns:
+
+###################
+The GNU Name System
+###################
+
+The GNU Name System (GNS) is a decentralized, censorship-resistant and
+privacy-preserving alternative to the Domain Name System (DNS). It is the
+most mature part of GNUnet, its protocol is published as :rfc:`9498`, and it
+is the best place to start if you are new to GNUnet.
+
+.. tip::
+
+ Want to try it right away? The :doc:`/quickstart` gets you a working
+ zone in five commands.
+
+What GNS gives you
+==================
+
+**Names that nobody can take away from you.**
+ In DNS, names are rented from a registrar under rules set by registries
+ and ICANN, and they can be seized, blocked or redirected. In GNS, a
+ *zone* is a key pair that you generate yourself. Only the holder of the
+ private key can change the records in a zone.
+
+**No central root.**
+ Each user is the root of their own namespace: the names that resolve on
+ your system are determined by your own configuration. You link to other
+ people's zones (*delegation*) under names that you choose (*petnames*).
+ GNUnet ships with a small default set of zones so that common names work
+ out of the box.
+
+**Private lookups.**
+ Records are encrypted and stored in GNUnet's distributed hash table
+ (DHT). The peers that store and route a query learn neither the name
+ being looked up nor the answer, unless they already know both the zone
+ and the name.
+
+**Authentic answers.**
+ Every record is signed with the zone key, so a resolver can verify an
+ answer without trusting the network that delivered it.
+
+**DNS compatibility.**
+ GNS supports the usual DNS record types (``A``, ``AAAA``, ``MX``,
+ ``TXT``, ...), can serve unmodified applications through a DNS bridge,
+ and comes with tools to import existing DNS zones.
+
+How it works, in one paragraph
+==============================
+
+Your zone is identified by its public key, for example
+``000G0507HZ8M09QWA93N2YX9Y9357777C0HTPK7PA0HTKJS5GCXXPGKBWC``. You add
+records to it with ``gnunet-namestore``; your peer signs and encrypts the
+records you mark as public and stores them in the DHT. To resolve
+``www.alice.myzone``, your resolver starts in your own zone ``myzone``,
+finds that the label ``alice`` delegates to Alice's zone key, and then
+fetches and decrypts the record set for ``www`` in Alice's zone. Each step
+is verified with the respective zone key. :doc:`concepts` explains this in
+detail.
+
+GNS compared with DNS
+=====================
+
+.. list-table::
+ :header-rows: 1
+ :widths: 30 35 35
+
+ * -
+ - DNS
+ - GNS
+ * - Who controls a name
+ - registrar, registry and ultimately ICANN
+ - whoever holds the zone's private key
+ * - Root of trust
+ - one global root zone
+ - each user's own start zones
+ * - Globally unique names
+ - yes
+ - no: names are relative to the user (petnames)
+ * - Query privacy
+ - queries are visible to resolvers and name servers
+ - queries and answers are hidden from the peers storing and routing
+ them
+ * - Record authenticity
+ - optional (DNSSEC)
+ - always (every record set is signed)
+ * - Censorship
+ - names can be seized or blocked centrally
+ - there is no central point at which to seize a name
+
+The price for this is that names are not globally unique: ``www.alice``
+means whatever *you* decided ``alice`` refers to. The :doc:`faq` discusses
+this trade-off.
+
+Current status
+==============
+
+GNS is the most mature component of GNUnet. The protocol is specified in
+:rfc:`9498`, its C API is marked stable (see :ref:`Subsystem-stability`),
+and managing your own zones and resolving names in them works well. What
+is still rough:
+
+- Resolving names *in other people's zones* goes through the GNUnet
+ peer-to-peer network, which is still small and has known connectivity
+ issues. Such lookups can be slow or fail; names in your own zones always
+ resolve locally.
+- Using GNS names in browsers and other ordinary applications requires
+ some manual configuration (see :doc:`integration`).
+- Exchanging zone keys via printed business cards and QR codes does not
+ currently work (see :doc:`sharing`).
+
+See :doc:`/status` for the state of all GNUnet components.
+
+In this section
+===============
+
+.. toctree::
+ :maxdepth: 1
+
+ concepts
+ zones
+ integration
+ sharing
+ migration
+ faq
+
+Further reading
+===============
+
+- :rfc:`9498`, the normative specification, maintained as
+ `LSD 0001 <https://lsd.gnunet.org/lsd0001>`__
+- `Talks and videos <https://www.gnunet.org/en/video.html>`__ about GNS
+- Research papers in the `GNUnet bibliography <https://bib.gnunet.org>`__
+- The :ref:`developer documentation of the GNS subsystem
+ <GNU-Name-System-_0028GNS_0029>`
diff --git a/gns/integration.rst b/gns/integration.rst
@@ -0,0 +1,184 @@
+.. _gns-integration:
+
+Using GNS names in applications
+===============================
+
+``gnunet-gns`` resolves GNS names on the command line. This page explains
+how to make GNS names work in ordinary applications such as web browsers,
+``curl`` or ``ssh``, which only know how to use DNS.
+
+There are two kinds of applications:
+
+- **GNS-aware applications** use GNS directly through its API. Examples are
+ GNUnet's own :doc:`re:claimID </users/reclaim>` and the
+ :doc:`messenger </users/messenger>`. They need no configuration.
+- **GNS-unaware applications** expect names to be resolved by the operating
+ system or by DNS. For these, you put GNS in the path of their name
+ lookups, using one of the methods below.
+
+.. list-table::
+ :header-rows: 1
+ :widths: 22 39 39
+
+ * - Method
+ - Works for
+ - Notes
+ * - `DNS2GNS`_ (recommended)
+ - every application that uses the system's resolver
+ - answers DNS queries for GNS names and forwards all other queries to
+ a DNS resolver of your choice
+ * - `NSS plugin`_
+ - applications that resolve names through glibc
+ - not available on systems that generate ``/etc/nsswitch.conf``
+ automatically, or on non-glibc systems
+ * - `SOCKS proxy for browsers`_
+ - one browser at a time
+ - can validate TLS certificates via GNS records, but is brittle
+
+.. _DNS2GNS:
+
+DNS2GNS
+-------
+
+``gnunet-dns2gns`` is a small DNS server. It answers queries for GNS names
+using GNS and passes every other query on to an ordinary DNS resolver.
+Pointing your system's resolver at it makes GNS names work everywhere.
+
+By default it listens on ``127.0.0.1`` and ``::1``, port ``15353``, and
+forwards non-GNS queries to ``9.9.9.9``. To use a different upstream
+resolver, for example your router:
+
+.. code-block:: console
+
+ $ gnunet-config -s dns2gns -o OPTIONS -V "-d 192.168.1.1"
+
+Start the service and check that it works:
+
+.. code-block:: console
+
+ $ gnunet-arm -i dns2gns
+ $ dig +short @127.0.0.1 -p 15353 www.myzone
+ 192.0.2.1
+
+To start it automatically with your peer, set ``IMMEDIATE_START = YES`` in
+its configuration section:
+
+.. code-block:: console
+
+ $ gnunet-config -s dns2gns -o IMMEDIATE_START -V YES
+
+Now tell your system to use it. How to do that depends on your system.
+
+**With systemd-resolved** (most current GNU/Linux desktops), set it as the
+DNS server of your network interface, here ``wlan0``:
+
+.. code-block:: console
+
+ $ sudo resolvectl dns wlan0 127.0.0.1:15353
+
+This lasts until the interface is reconfigured. Some distribution
+packages, for example Fedora's, ship a NetworkManager dispatcher script
+that does this automatically each time an interface comes up; it is
+activated by setting ``ENABLE_RESOLVECTL_NMDISPATCHER = YES`` in the
+``[dns2gns]`` section of ``/etc/gnunet.conf``.
+
+**With a plain /etc/resolv.conf**, you cannot specify a port, so DNS2GNS
+must listen on the standard DNS port 53. This requires running it with
+the necessary privileges, typically as part of a :ref:`system-wide
+installation <The-Multi_002dUser-Setup>`. Set ``PORT = 53`` in the
+``[dns2gns]`` section and add this line at the top of
+``/etc/resolv.conf``::
+
+ nameserver 127.0.0.1
+
+Test the result with any application:
+
+.. code-block:: console
+
+ $ curl http://www.myzone/
+
+.. note::
+
+ Using DNS2GNS breaks the end-to-end verification of GNS for the
+ application: the application trusts whatever the local DNS2GNS server
+ tells it. This is fine as long as DNS2GNS runs on your own machine. Do
+ not use a DNS2GNS server run by somebody else.
+
+.. _NSS plugin:
+
+NSS plugin
+----------
+
+On systems with the GNU C library, GNS can be added to the *name service
+switch* (NSS), which most programs use to look up host names.
+
+Distribution packages install the plugin ``libnss_gns.so.2`` for you. When
+building from source, it is installed into the library directory of your
+installation prefix; glibc only finds it there if that directory is in the
+dynamic linker's search path (run ``sudo ldconfig`` after installing to
+``/usr/local``).
+
+To activate the plugin, edit ``/etc/nsswitch.conf``. Find the line
+starting with ``hosts:``, for example::
+
+ hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4
+
+and insert ``gns [NOTFOUND=return]`` after ``files``::
+
+ hosts: files gns [NOTFOUND=return] mdns4_minimal [NOTFOUND=return] dns mdns4
+
+On some systems, in particular those using systemd-resolved,
+``/etc/nsswitch.conf`` is generated automatically and must not be edited.
+Use `DNS2GNS`_ there.
+
+.. warning::
+
+ In GNUnet 0.29, the NSS plugin fails to resolve names: it parses the
+ output of ``gnunet-gns -r``, which now contains a progress indicator.
+ Use `DNS2GNS`_ until this is fixed.
+
+.. _SOCKS proxy for browsers:
+
+SOCKS proxy for browsers
+------------------------
+
+``gnunet-gns-proxy`` is a SOCKS5 proxy that resolves GNS names for a
+browser. Unlike DNS2GNS, it can use GNS records (``LEHO``, ``BOX``/``TLSA``)
+to validate TLS certificates of sites hosted under GNS names, by acting as
+a local certificate authority. It is, however, still brittle: expect
+subtle failures with some sites, and switch it off again if a page behaves
+oddly.
+
+First, create and install the proxy's certificate authority. The script
+supports Firefox and Chromium, and needs the ``certutil`` (from NSS) and
+``openssl`` programs:
+
+.. code-block:: console
+
+ $ gnunet-gns-proxy-setup-ca
+
+Then start the proxy and check that it is running:
+
+.. code-block:: console
+
+ $ gnunet-arm -i gns-proxy
+ $ gnunet-arm -I | grep gns-proxy
+
+The proxy listens on ``localhost``, port ``7777``. Configure your browser
+to use it:
+
+Chromium
+ Start the browser with
+ ``chromium --proxy-server=socks5://localhost:7777``. Type URLs including
+ the ``http://`` or ``https://`` prefix; otherwise Chromium may run a web
+ search instead.
+
+Firefox
+ In *Settings*, *Network Settings*, choose *Manual proxy configuration*,
+ enter ``localhost`` and port ``7777`` as *SOCKS Host*, select *SOCKS v5*,
+ and enable *Proxy DNS when using SOCKS v5*. Then open ``about:config``
+ and set ``browser.fixup.alternate.enabled`` to ``false``, so that Firefox
+ does not rewrite a name like ``www.myzone`` into ``www.myzone.com``.
+
+Note that accessing ordinary websites through GNS is experimental and not
+the primary goal of GNS at this time. We welcome help with testing.
diff --git a/gns/migration.rst b/gns/migration.rst
@@ -0,0 +1,126 @@
+.. _gns-migration:
+
+Migrating DNS zones to GNS
+==========================
+
+If you run a DNS zone, you can make it available in GNS as well. Which tool
+to use depends on where the authoritative data lives:
+
+- **You have the zone file**, because you are the operator: import it with
+ `gnunet-namestore-zonefile`_, and keep DNS and GNS in sync yourself.
+- **The zone supports zone transfers (AXFR/IXFR)**: use `Ascension`_, which
+ mirrors the zone and keeps it updated.
+- **The zone only publishes a list of delegated names**, as some top-level
+ domain registries do: use `gnunet-zoneimport`_.
+
+When importing very large zones (millions of records), see also
+:ref:`Large zones <gns-zones>`. In addition, the NAMESTORE by default
+pre-populates the NAMECACHE, which is cryptographically expensive. On a
+system that only serves to import a large zone, and does not resolve names
+itself, disable the namecache by setting ``DISABLE = YES`` in the
+``[namecache]`` section.
+
+gnunet-namestore-zonefile
+-------------------------
+
+``gnunet-namestore-zonefile`` reads a DNS zone file, as used by BIND and
+other DNS servers, from standard input and stores its records in GNS:
+
+.. code-block:: console
+
+ $ gnunet-namestore-zonefile < myzonefile
+
+For each ``$ORIGIN`` directive in the file, a zone named after the origin is
+created unless it already exists: ``$ORIGIN example.com.`` creates or uses
+the zone ``example.com``. The records following the directive are imported
+into that zone. If the file has no ``$ORIGIN`` directive at the beginning,
+or you want to override the first one, name the zone explicitly:
+
+.. code-block:: console
+
+ $ gnunet-namestore-zonefile -z myorigin < myzonefile
+
+The conversion is not lossless. GNS maps the origin to the label ``@``
+wherever possible, so equivalent DNS spellings collapse into one. DNS
+``$TTL`` values in seconds are converted to GNS expiration times, which are
+measured in microseconds. Converting a GNS zone back into a zone file would
+therefore not reproduce the original file (and is not implemented).
+
+For example, both of the following lines describe the same record, and
+become one ``MX`` record under ``@`` in GNS:
+
+.. code-block:: text
+
+ $ORIGIN example.com.
+ $TTL 3600
+ example.com. IN MX 10 mail.example.com.
+ @ IN MX 10 mail.example.com.
+
+Ascension
+---------
+
+Ascension is a separate tool that mirrors DNS zones into GNS using zone
+transfers. If the zone has an ``SOA`` record, Ascension uses its serial
+number to request incremental transfers (IXFR); depending on the server, the
+answer is an incremental update or the full zone (AXFR). Ascension keeps
+running and refreshes the zone when the refresh time given in the ``SOA``
+record has elapsed.
+
+With your peer running, the following command mirrors the Syrian top-level
+domain, one of the few that allow zone transfers, and publishes it
+(``-p``):
+
+.. code-block:: console
+
+ $ ascension sy. -n ns1.tld.sy. -p
+
+To let others resolve the mirrored zone, give them its key:
+
+.. code-block:: console
+
+ $ gnunet-identity -d -q -e sy
+
+They can then use the key as top-level domain, or delegate to it as
+described in :ref:`gns-zones`:
+
+.. code-block:: console
+
+ $ gnunet-gns -t SOA -u "$KEY"
+
+DNSCurve-style records are supported and imported as delegations to the
+respective GNS zone.
+
+There are unofficial Debian packages: ``python3-ascension`` adds a system
+user ``ascension`` and runs a GNUnet peer in the background, and
+``ascension-bind`` detects locally running DNS zones that allow zone
+transfers, asks which of them to migrate, and installs a systemd unit that
+keeps them up to date.
+
+gnunet-zoneimport
+-----------------
+
+Some registries, for example AFNIC for ``.fr``, do not permit zone transfers
+but publish the list of registered names. ``gnunet-zoneimport`` reads such a
+list of domain names from standard input, queries the authoritative DNS
+servers for each of them, converts the answers where possible, and stores
+them in the NAMESTORE. It is not perfect at tracking updates and
+expirations, but it makes mirroring such zones possible at all.
+
+For example, with AFNIC's ``NomsDeDomaineEnPointFr`` open-data file:
+
+.. code-block:: console
+
+ $ cut -d';' -f 1 OPENDATA_A-NomsDeDomaineEnPointFr.csv | tail -n+2 \
+ | gnunet-zoneimport 194.0.9.1
+
+This takes the first column of each line as the domain name and skips the
+header line. The argument is the IP address of an authoritative DNS server
+for the zone; it must be authoritative, not a recursive resolver.
+
+The ZONEMASTER service then publishes the imported records in the DHT. To
+make the mirrored zone available under its usual name on a peer, map the
+suffix to the zone key in the ``[gns]`` section, for example:
+
+.. code-block:: console
+
+ $ gnunet-config -s gns -o .fr -V 000G0...ZONEKEY
diff --git a/gns/sharing.rst b/gns/sharing.rst
@@ -0,0 +1,118 @@
+.. _gns-sharing:
+
+Publishing and sharing your zone
+================================
+
+A zone becomes useful to others once they can find it. This page explains
+how to publish records, how to hand out your zone key, and how to get a name
+for your zone that other GNUnet users can resolve without any setup.
+
+Publishing records
+------------------
+
+Only records marked as **public** are published. Add ``-p`` when you create
+a record:
+
+.. code-block:: console
+
+ $ gnunet-namestore -z myzone -a -p -n www -t A -V 192.0.2.1 -e 1d
+
+Your peer signs and encrypts public records and stores them in the GNUnet
+DHT, and republishes them regularly while it is running. Keep your peer
+running (see :doc:`/users/start`): if it stays offline, your records
+eventually expire from the network.
+
+Sharing your zone key
+---------------------
+
+Print the public key of your zone and give it to the people who should be
+able to resolve your names:
+
+.. code-block:: console
+
+ $ gnunet-identity -d -q -e myzone
+ 000G0507HZ8M09QWA93N2YX9Y9357777C0HTPK7PA0HTKJS5GCXXPGKBWC
+
+A friend then adds it to one of their zones under a petname of their
+choice, for example ``you``:
+
+.. code-block:: console
+
+ $ gnunet-namestore -z theirzone -a -n you -t EDKEY -V 000G0507HZ8M...WC -e 1d
+
+and can resolve your public records as ``www.you.theirzone``. Without
+adding anything, they can also use the key itself as the top-level domain:
+``www.000G0507HZ8M...WC``.
+
+How you exchange keys matters: the key *is* the identity of your zone. Use
+a channel that you trust to deliver it unmodified, for example in person or
+in a signed message.
+
+Registering a name under .pin.gns.alt
+-------------------------------------
+
+Every GNUnet installation is configured with the start zone
+``.pin.gns.alt``, a registry operated by GNUnet e.V. on a first-come,
+first-served basis. If you register ``myname`` there, every GNUnet user can
+resolve your records as ``www.myname.pin.gns.alt`` without exchanging keys
+first.
+
+To register, visit https://fcfs.gnunet.org and register your zone key under
+a name of your choice. The registrar is connected to the demo deployment of
+`GNU Taler <https://taler.net/>`__ and asks for a payment in KUDOS, a demo
+currency you get for free when you register at the Taler demo; the service
+is therefore effectively free.
+
+Keep in mind that names in ``.pin.gns.alt`` are only as trustworthy as the
+registry: whoever operates it could, in principle, change the delegation.
+For names that matter, exchange keys directly as well.
+
+Business cards and QR codes
+---------------------------
+
+GNUnet includes ``gnunet-bcd``, which creates printable business cards with
+a QR code of your zone key, and ``gnunet-qr``, which reads such a code with
+a camera and imports the zone.
+
+.. warning::
+
+ In GNUnet 0.29, importing a zone from a QR code does not work: the
+ import step (``gnunet-namestore -u``) fails for current zone keys.
+ Exchange keys as text, as shown above, until this is fixed.
+
+``gnunet-bcd`` is a small web application. It needs a LaTeX installation
+with the ``units``, ``labels``, ``pst-barcode``, ``luatex85``, ``preview``,
+``pdfcrop`` and ``koma-script`` packages (on Debian, ``texlive-full``
+contains all of them). Start it, then open http://localhost:8888/ in a
+browser and fill in the form, including your zone key:
+
+.. code-block:: console
+
+ $ gnunet-arm -i bcd
+
+The result is a PDF with ten business cards.
+
+The default start zones
+-----------------------
+
+The start zones configured by default, such as ``.pin.gns.alt``, are
+registered in `GANA <https://gana.gnunet.org>`__ and follow a `restrictive
+registration policy
+<https://git.gnunet.org/gana.git/tree/gnu-name-system-default-tlds/POLICY>`__.
+Every user can remove them or add others; see :ref:`gns-concepts`.
+
+Running your own registrar
+--------------------------
+
+The software behind https://fcfs.gnunet.org is available as a separate
+package, the GNUnet GNS registrar, from the `GNU FTP mirror
+<https://ftpmirror.gnu.org/gnunet>`__. Build and run it with:
+
+.. code-block:: console
+
+ $ ./configure && make
+ $ ./gnunet-gns-registrar
+
+It requires a `GNU Taler merchant backend <https://taler.net/>`__ for
+payments; see the example configuration in the package for the available
+options.
diff --git a/gns/zones.rst b/gns/zones.rst
@@ -0,0 +1,212 @@
+.. _gns-zones:
+
+Managing your zones
+===================
+
+This page covers the day-to-day work with your own zones: creating them,
+editing records, delegating to other zones, and keeping your keys safe. It
+assumes your peer is running (``gnunet-arm -s``).
+
+Creating, listing and deleting zones
+------------------------------------
+
+.. code-block:: console
+
+ $ gnunet-identity -C myzone # create a zone
+ $ gnunet-identity -d # list all zones with their keys
+ $ gnunet-identity -d -q -e myzone # print only the key of one zone
+ $ gnunet-identity -D myzone # delete a zone (see "Orphaned records")
+
+The public key printed by ``gnunet-identity -d -q -e myzone`` is what you
+give to others so that they can delegate to your zone.
+
+Adding records
+--------------
+
+Records are added with ``gnunet-namestore``. You always name the zone
+(``-z``), the label (``-n``), the type (``-t``), the value (``-V``) and an
+expiration time (``-e``):
+
+.. code-block:: console
+
+ $ gnunet-namestore -z myzone -a -n www -t A -V 192.0.2.1 -e 1d
+
+Expiration times are given as relative times such as ``1h``, ``1d`` or
+``30d``. Resolvers cache records until they expire, so choose a shorter
+time for records you expect to change.
+
+Records are **private** by default: they resolve on your own peer but are
+not published. Add ``-p`` to make a record public, so that people who
+delegate to your zone can resolve it:
+
+.. code-block:: console
+
+ $ gnunet-namestore -z myzone -a -p -n www -t A -V 192.0.2.1 -e 1d
+
+The zone apex is the label ``@``. To give the zone itself an address:
+
+.. code-block:: console
+
+ $ gnunet-namestore -z myzone -a -p -n @ -t A -V 192.0.2.1 -e 1d
+
+Listing and deleting records
+----------------------------
+
+.. code-block:: console
+
+ $ gnunet-namestore -z myzone -D
+ www.myzone:
+ A: 192.0.2.1 (1 day) PUBLIC
+ $ gnunet-namestore -z myzone -D -n www # one label only
+ $ gnunet-namestore -z myzone -d -n www -t A -V 192.0.2.1 # delete one record
+ $ gnunet-namestore -z myzone -d -n www # delete all records under www
+
+To check what others will see, resolve the name the way any client would:
+
+.. code-block:: console
+
+ $ gnunet-gns -u www.myzone
+
+``gnunet-gns`` looks up ``A`` records unless you ask for another type with
+``-t``, for example ``gnunet-gns -t MX -u myzone``.
+
+Delegating to other zones
+-------------------------
+
+To make another zone available below one of your labels, add a delegation
+record holding its key. For zones with EdDSA keys (the default) the record
+type is ``EDKEY``; for older ECDSA zones it is ``PKEY``:
+
+.. code-block:: console
+
+ $ gnunet-namestore -z myzone -a -n bob -t EDKEY -V 000G0...BOBKEY -e 1d
+
+Bob's public records are now reachable as ``NAME.bob.myzone``. Make the
+delegation public (``-p``) if people who delegate to *your* zone should be
+able to reach Bob's zone through it as well. See :ref:`gns-concepts` for
+how delegation chains work.
+
+To delegate a label to a zone that lives in DNS instead, use a ``GNS2DNS``
+record whose value names the DNS domain and a DNS server to ask, in the form
+``domain@server``:
+
+.. code-block:: console
+
+ $ gnunet-namestore -z myzone -a -n gnu -t GNS2DNS -V gnu.org@8.8.8.8 -e 1d
+
+Choosing a database backend
+---------------------------
+
+Your records are stored by the NAMESTORE service. The default ``sqlite``
+backend needs no configuration and suits almost everybody. For very large
+zones (see `Large zones`_) use PostgreSQL instead:
+
+.. code-block:: console
+
+ $ gnunet-config -s namestore -o DATABASE -V postgres
+ $ gnunet-config -s namestore-postgres # review the options
+
+In a multi-user setup, the NAMESTORE database holds the records of all
+users; users cannot see each other's private records.
+
+If you ever need to start over, ``gnunet-namestore-dbtool --reset`` empties
+the database. **This deletes all records in all of your zones.**
+
+.. _gns-backup:
+
+Backing up
+----------
+
+Back up these files from ``~/.local/share/gnunet/`` (or
+``$XDG_DATA_HOME/gnunet/``):
+
+``identity/egos/``
+ The private keys of your zones, one file per zone, named after the zone.
+ **These are the most important files of a GNUnet installation**: whoever
+ has them controls your zones, and if you lose them you lose your zones.
+
+``namestore/``
+ The record database (``sqlite.db`` for the default backend; for
+ PostgreSQL, back up the database instead).
+
+``private_key.ecc``
+ The key of your *peer*, which identifies your node in the network. It is
+ not related to your zones.
+
+The key files are stored unencrypted, so store the backup encrypted.
+
+.. _gns-revocation:
+
+Revoking a compromised zone
+---------------------------
+
+If someone steals the private key of a zone, they can publish records that
+everybody will accept as yours. The remedy is a *revocation*: a message,
+signed with the zone key and flooded through the network, that marks the
+key as invalid. After a revocation, resolution of names in the zone, and
+through it, fails everywhere.
+
+Revocations require a proof of work so that they are expensive to create.
+Prepare one **in advance**, while you still control the key, and store it
+safely:
+
+.. code-block:: console
+
+ $ gnunet-revocation -f myzone.rev -R myzone
+
+The full computation can take days. You can interrupt it with ``Ctrl-C``
+at any time: the file already contains the signature, which is the part that
+needs the private key, and the computation can be resumed later, even
+without access to the key. Unless you expect to need a very fast reaction,
+interrupt it early, keep the file, and finish the computation only if your
+key is ever compromised. This saves considerable energy.
+
+The command that actually revokes a zone is ``gnunet-revocation -p``. It
+cannot be undone, so read ``gnunet-revocation --help`` carefully before you
+use it.
+
+Orphaned records
+----------------
+
+If you delete a zone's ego (``gnunet-identity -D``) while records for it
+remain in the database, those records become *orphaned*: they are still
+published, but you can no longer manage them. To find them:
+
+.. code-block:: console
+
+ $ gnunet-namestore --list-orphans
+
+This prints each orphaned record set as ``<label>.<private key>:`` followed
+by the records. To recover the zone, recreate the ego from the printed
+private key:
+
+.. code-block:: console
+
+ $ gnunet-identity -C myzone --privkey=PRIVATE_KEY
+
+To delete all orphaned records instead:
+
+.. code-block:: console
+
+ $ gnunet-namestore --purge-orphans
+
+Large zones
+-----------
+
+Zones with very many names, such as the mirror of a DNS top-level domain,
+need some care:
+
+- Use the PostgreSQL backend (see `Choosing a database backend`_).
+- Use EdDSA zones, which are considerably faster than ECDSA zones. This is
+ the default for new zones.
+- Import records in bulk: there is a :ref:`C API for bulk imports
+ <NAMESTORE-Subsystem>` and a :ref:`REST endpoint <namestore_rest_api>`
+ that uses it.
+- Let the ZONEMASTER service, which signs and publishes your records, use
+ more worker threads, for example one per CPU core:
+
+ .. code-block:: console
+
+ $ gnunet-config -s zonemaster -o WORKER_COUNT -V 8
+
+See also :doc:`migration` for importing zones from DNS.
diff --git a/index.rst b/index.rst
@@ -1,16 +1,66 @@
-##################################
-Welcome to GNUnet’s documentation!
-##################################
+####################
+GNUnet documentation
+####################
+
+GNUnet is a network protocol stack for building secure, decentralized and
+privacy-preserving applications. Its most mature component is the **GNU
+Name System (GNS)**: a decentralized and censorship-resistant alternative to
+DNS, specified in :rfc:`9498`. This documentation starts there.
+
+:doc:`Quick start <quickstart>`
+ Install GNUnet and create your own GNS zone in five commands.
+
+:doc:`The GNU Name System <gns/index>`
+ What GNS is and how it works (:doc:`gns/concepts`), how to manage your
+ zones (:doc:`gns/zones`), use GNS names in your browser and other
+ applications (:doc:`gns/integration`), share your zone
+ (:doc:`gns/sharing`) and migrate from DNS (:doc:`gns/migration`).
+
+:doc:`What works today <status>`
+ The state of GNS and of GNUnet's other applications, which are still in
+ development.
+
+:doc:`Developer handbook <developers/index>`
+ Architecture, APIs, tutorials, and how to contribute.
.. toctree::
+ :hidden:
+ :caption: Getting started
:maxdepth: 1
- about
+ quickstart
installing
+ users/start
+
+.. toctree::
+ :hidden:
+ :caption: Applications
+ :maxdepth: 1
+
+ gns/index
users/index
- developers/index
- guis/index
+
+.. toctree::
+ :hidden:
+ :caption: Operating a peer
+ :maxdepth: 1
+
+ users/configuration
+
+.. toctree::
+ :hidden:
+ :caption: About GNUnet
+ :maxdepth: 1
+
+ about
+ status
faq
+
+.. toctree::
+ :hidden:
+ :caption: Developers
+ :maxdepth: 1
+
+ developers/index
GANA <https://gana.gnunet.org>
Bibliography <https://bib.gnunet.org>
-
diff --git a/installing.rst b/installing.rst
@@ -1,298 +1,255 @@
-Installation
-============
+.. _installing:
-This guide is intended for those who want to install GNUnet from source.
-For instructions on how to install GNUnet as a binary package please
-refer to the official documentation of your operating system or package
-manager.
+Installing GNUnet
+=================
-For understanding this guide properly it is important to know that there
-are two different ways of running GNUnet:
+You need a *recent* release of GNUnet. The network protocols still change
+between major releases (for example from 0.28.x to 0.29.x), and peers
+running different major versions may be unable to communicate. New
+releases are announced at https://www.gnunet.org/en/news/.
-- the single-user setup
-- the multi-user setup
+There are two ways to install GNUnet: a package from your operating system,
+if it is recent enough, or from source. Afterwards, continue with the
+:doc:`quickstart`.
-The latter variant has a better security model and requires extra
-preparation before running make install and a different configuration.
-Beginners who want to quickly try out GNUnet can use the single-user
-setup.
+Installing a package
+--------------------
-Dependencies
-------------
-
-GNUnet needs few libraries and applications for being able to run and
-another few optional ones for using certain features. Preferably they
-should be installed with a package manager.
-
-The mandatory libraries and applications are
-
-- meson 1.0 or above
-- recutils 1.0 or above (when building from git)
-- gettext
-- glibc (read below, other libcs work)
-- GnuTLS 3.2.12 or above, recommended to be linked against libunbound
-- GNU make 4.0 or higher (other make implementations do work)
-- iptables (on Linux systems)
-- libtool 2.2 or above
-- libltdl (part of libtool)
-- libgcrypt 1.6 or above
-- libidn2 or libidn
-- libmicrohttpd 0.9.63 or above
-- libunistring
-- libjansson
-- libjose (optional, for reclaimID)
-- libgmp
-- libcurl (ideally linked to GnuTLS) 7.35.0 or
- above
-- Texinfo 5.2 or above (for building the documentation)
-- Texlive 2012 or above (for building the documentation, and for
- gnunet-bcd)
-- makeinfo 4.8 or above
-- pkgconf (or pkg-config)
-- zlib
-
-Glibc is required for certain NSS features:
-
-.. code-block:: text
-
- One mechanism of integrating GNS with legacy applications via NSS is
- not available if this is disabled. But applications that don't use the
- glibc for NS resolution won't work anyway with this, so little is lost
- on *BSD systems.
- GNS via direct use or via the HTTP or DNS proxies is unaffected.
-
-Other libcs should work, the resulting builds just don’t include the
-glibc NSS specific code. One example is the build against NetBSD’s libc
-as detailed in https://bugs.gnunet.org/view.php?id=5605.
-
-In addition GNUnet needs at least one of these three databases (at the
-minimum sqlite3)
-
-- sqlite + libsqlite 3.8 or above (the default, requires no further
- configuration)
-- postgres + libpq
-- mysql + libmysqlclient
-
-These are the dependencies only required for certain features
-
-- miniupnpc (for traversing NAT boxes more reliably)
-- libnss
-- libopus (for running the GNUnet conversation telephony application)
-- libogg (for running the GNUnet conversation telephony application)
-- gstreamer OR libpulse (for running the GNUnet conversation telephony
- application)
-- bluez (for bluetooth support)
-- libextractor (optional but highly recommended, read below)
-- texi2mdoc (for automatic mdoc generation)
-- perl5 for some utilities (which are not installed)
-
-About libextractor being optional:
-
-.. code-block:: text
+.. list-table::
+ :header-rows: 1
+ :widths: 25 75
- While libextractor ("LE") is optional, it is recommended to build gnunet
- against it. If you install it later, you won't benefit from libextractor.
- If you are a distributor, we recommend to split LE into basis + plugins
- rather than making LE an option as an afterthought by the user. LE
- itself is very small, but its dependency chain on first, second, third
- etc level can be big. There is a small effect on privacy if your LE
- build differs from one which includes all plugins (plugins are build as
- shared objects): if users publish a directory with a mixture of file
- types (for example mpeg, jpeg, png, gif) the configuration of LE could
- leak which plugins are installed for which filetypes are not providing
- more details. However, this leak is just a minor concern.
+ * - System
+ - How to install
+ * - Fedora
+ - ``sudo dnf copr enable schanzen/gnunet && sudo dnf install gnunet``
+ * - Arch Linux
+ - the `gnunet <https://aur.archlinux.org/packages/gnunet/>`__ package
+ from the AUR
+ * - Alpine Linux
+ - ``apk add gnunet`` (`community repository
+ <https://pkgs.alpinelinux.org/package/edge/community/x86_64/gnunet>`__)
+ * - GNU Guix
+ - ``guix install gnunet``
+ * - Nix and NixOS
+ - the `gnunet
+ <https://search.nixos.org/packages?query=gnunet>`__ package
+ * - macOS
+ - ``brew install gnunet`` (Homebrew)
+ * - OpenWrt
+ - see `Installing on OpenWrt
+ <https://www.gnunet.org/en/install-on-openwrt.html>`__
+ * - Debian, Ubuntu
+ - The distribution packages are **outdated** (Debian 13 ships 0.20)
+ and cannot join the current network. `Install from source`_.
-These are the test-suite requirements:
+Packages are maintained by the distributions, not by the GNUnet project,
+and may lag behind. After installing, check the version:
-- python3.6 or higher
-- gnunet (installation first)
-- some core-utils: which(1), bc(1), curl(1), sed(1), awk(1), etc.
-- a shell (very few Bash scripts, the majority are POSIX sh scripts)
+.. code-block:: console
-These are runtime requirements:
+ $ gnunet-arm --version
-- nss (the certutil binary, for gnunet-gns-proxy-setup-ca)
-- openssl (openssl binary, for gnunet-gns-proxy-setup-ca)
+If it is older than the latest release, install from source instead.
-Getting the Source Code
------------------------
+Some packages set up a :ref:`system-wide installation
+<system-wide-installation>`, with system services that run as the user
+``gnunet``. In that case, add yourself to the ``gnunet`` group and log in
+again:
-You can either download the source code using git (you obviously need
-git installed) or as an archive.
+.. code-block:: console
-Using git type
+ $ sudo usermod -aG gnunet $USER
-::
+Install from source
+-------------------
- $ git clone git://git.gnunet.org/gnunet.git
+Building GNUnet takes a few minutes.
-The release archive can be found at https://ftpmirror.gnu.org/gnu/gnunet/.
-You can find nightly builds at https://buildbot.gnunet.org/artifacts/.
-Extract it using a graphical archive tool or tar:
+1. Install the build dependencies. On Debian 12 or newer and on recent
+ Ubuntu releases:
-::
+ .. code-block:: console
- tar xf gnunet-<VERSION> .tar.gz
+ $ sudo apt install build-essential meson pkgconf gettext \
+ libgcrypt20-dev libsodium-dev libgmp-dev libunistring-dev \
+ libidn2-dev libltdl-dev libjansson-dev libmicrohttpd-dev \
+ libcurl4-gnutls-dev libsqlite3-dev zlib1g-dev libgnutls28-dev \
+ libextractor-dev
-In the next chapter we will assume that the source code is available in
-the home directory at ~/gnunet.
+ For other systems, see `Dependencies`_.
-Create user and groups for the system services
-----------------------------------------------
+2. Download the latest release, ``gnunet-VERSION.tar.gz``, from
+ https://ftpmirror.gnu.org/gnu/gnunet/ and unpack it:
-**For single-user setup this section can be skipped**
+ .. code-block:: console
-The multi-user setup means that there are system services, which are run
-once per machine as a dedicated system user (called gnunet) and user
-services which can be started by every user who wants to use GNUnet
-applications. The user services communicate with the system services
-over unix domain sockets. To gain permissions to read and write those
-sockets the users running GNUnet applications will need to be in the
-gnunet group. In addition the group gnunetdns may be needed (see below).
+ $ tar xf gnunet-VERSION.tar.gz
+ $ cd gnunet-VERSION
-Create user gnunet who is member of the group gnunet (automatically
-created) and specify a home directory where the GNUnet services will
-store persistent data such as information about peers.
+3. Build and install:
-::
+ .. code-block:: console
- $ sudo useradd --system --home-dir /var/lib/gnunet --create-home gnunet
+ $ meson setup --prefix=/usr/local build
+ $ meson compile -C build
+ $ sudo meson install -C build
+ $ sudo ldconfig
-Now add your own user to the gnunet group:
+ Installing as root also enables the few features that need helper
+ programs with elevated privileges: the VPN, system-wide DNS
+ interception, and NAT traversal via ICMP.
-::
+That's it. Continue with the :doc:`quickstart`, starting at step 2.
- $ sudo usermod -aG gnunet <user>
+``meson setup --help`` lists all build options; ``meson configure build``
+shows their current values.
-Create a group gnunetdns. This allows using setgid in a way that only
-the DNS service can run the gnunet-helper-dns binary. This is only
-needed if system-wide DNS interception will be used. For more
-information see `Configuring system-wide DNS
-interception <installing#configuring%20system-wide-dns-interception>`__.
+Building from git
+~~~~~~~~~~~~~~~~~
-::
+To build the development version, clone the repository and run
+``./bootstrap`` before step 3. This additionally requires ``git`` and
+``recutils``:
- $ sudo groupadd gnunetdns
+.. code-block:: console
-Preparing and Compiling the Source Code
----------------------------------------
+ $ git clone https://git.gnunet.org/gnunet.git
+ $ cd gnunet
+ $ ./bootstrap
-Get the source code from git or a source tarball. When running
-`meson setup`, options can be specified to customize the compilation and
-installation process. For details execute:
+Note that the development version may be incompatible with the current
+release, and thus with most of the network.
-::
+Installing without root privileges
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- $ meson setup --help
+You can also install GNUnet into your home directory. The features that
+need privileged helper programs (see above) are then unavailable, which
+does not affect the GNU Name System. Configure a prefix in your home
+directory in step 3, install without ``sudo``, and tell the dynamic linker
+where to find GNUnet's libraries, for example in your ``~/.profile``:
-The following example configures the installation prefix `/usr/local`
-and creates the out-of-tree build folder `build`.
-
-::
-
- $ ./bootstrap (only when installing from git)
- $ meson setup -Dprefix=/usr/local build
-
-After running the bootstrap script and setup successfully the source
-code can be compiled and the compiled binaries can be installed using:
-
-::
+.. code-block:: console
+ $ meson setup --prefix=$HOME/gnunet --libdir=lib build
$ meson compile -C build
$ meson install -C build
+ $ export PATH=$HOME/gnunet/bin:$PATH
+ $ export LD_LIBRARY_PATH=$HOME/gnunet/lib:$LD_LIBRARY_PATH
-The latter command may need to be run as root (or with sudo) because
-some binaries need the suid bit set. Without that some features
-(e.g. the VPN service, system-wide DNS interception, NAT traversal using
-ICMP) will not work.
-
-NSS plugin (optional)
-~~~~~~~~~~~~~~~~~~~~~
-
-**NOTE: The installation of the NSS plugin is only necessary if GNS
-resolution shall be used with legacy applications (that only support
-DNS) and if you cannot do not want to use the DNS2GNS service.**
-
-One important library is the GNS plugin for NSS (the name services
-switch) which allows using GNS (the GNU name system) in the normal DNS
-resolution process. Unfortunately NSS expects it in a specific location
-(probably ``/lib``) which may differ from the installation prefix (see
-``--prefix`` option in the previous section). This is why the plugin has
-to be installed manually.
-
-Find the directory where nss plugins are installed on your system, e.g.:
-
-::
-
- $ ls -l /lib/libnss_*
- /lib/libnss_mymachines.so.2
- /lib/libnss_resolve.so.2
- /lib/libnss_myhostname.so.2
- /lib/libnss_systemd.so.2
-
-Copy the GNS NSS plugin to that directory:
+.. _system-wide-installation:
+.. _Minimal-configuration:
-::
+Optional: a multi-user setup
+----------------------------
- cp ~/gnunet/src/gns/nss/.libs/libnss_gns.so.2 /lib
+By default, each user runs a complete GNUnet peer of their own (the
+*single-user setup*). On a machine with several users, it is more
+efficient and more secure to run the parts that make up the peer once, as
+*system services* under a dedicated user ``gnunet``, while each user runs
+only their own *user services*, such as their zones. The user services
+talk to the system services over UNIX domain sockets; access is controlled
+by membership in the group ``gnunet``.
-Installing the GNS Certificate Authority (Optional)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+This setup requires some preparation before ``meson install``.
-**NOTE: Installing the GNS certificate authority is only necessary if
-GNS shall be used in a browser and if you cannot or do not want to use
-the DNS2GNS service.**
+1. Create the user ``gnunet`` (which also creates the group ``gnunet``),
+ with a home directory for the system services' persistent data:
-The GNS Certificate authority can provide TLS certificates for GNS names
-while downloading webpages from legacy webservers. This allows browsers
-to use HTTPS in combinations with GNS name resolution.
+ .. code-block:: console
-To install it execute the GNS CA-setup script. So far Firefox and
-Chromium are supported.
+ $ sudo useradd --system --home-dir /var/lib/gnunet --create-home gnunet
-::
+2. Add every user who wants to use GNUnet to the group:
- $ gnunet-gns-proxy-setup-ca
+ .. code-block:: console
-A local proxy server, that takes care of the name resolution and
-provides certificates on-the-fly needs to be started:
+ $ sudo usermod -aG gnunet USERNAME
-::
+3. If you want to use system-wide DNS interception, also create the group
+ ``gnunetdns``. It allows only the DNS service to run the
+ ``gnunet-helper-dns`` program:
- $ /usr/lib/gnunet/libexec/gnunet-gns-proxy
+ .. code-block:: console
-Now GNS should work in browsers that are configured to use a SOCKS proxy
-on localhost:7777.
+ $ sudo groupadd gnunetdns
+4. Build and install as described above.
-.. _Minimal-configuration:
+5. Tell the system services to run only the system part, in
+ ``/etc/gnunet.conf``:
-Minimal configuration
----------------------
+ .. code-block:: ini
-GNUnet needs a configuration file to start (see `Config file
-format <#Config-file-format>`__). For the *single-user setup* an empty
-file is sufficient:
+ [arm]
+ START_SYSTEM_SERVICES = YES
+ START_USER_SERVICES = NO
-::
+ and the users to run only their part, in each user's
+ ``~/.config/gnunet.conf``:
- $ touch ~/.config/gnunet.conf
+ .. code-block:: ini
-For the *multi-user setup* we need an extra config file for the system
-services. The default location is ``/etc/gnunet.conf``. The minimal
-content of that file which activates the system services roll is:
+ [arm]
+ START_SYSTEM_SERVICES = NO
+ START_USER_SERVICES = YES
-::
+6. Start the system services as the user ``gnunet``, ideally from your
+ init system, with ``gnunet-arm -c /etc/gnunet.conf -s``. Each user then
+ starts their own services with ``gnunet-arm -s``.
- [arm]
- START_SYSTEM_SERVICES = YES
- START_USER_SERVICES = NO
+See :ref:`The-Multi_002dUser-Setup` and :ref:`Access-Control-for-GNUnet`
+for details and recommendations.
-The config file for the user services (``~/.config/gnunet.conf``) needs
-the opposite configuration to activate the user services roll:
+Dependencies
+------------
-::
+Required:
- [arm]
- START_SYSTEM_SERVICES = NO
- START_USER_SERVICES = YES
+- meson 1.0 or newer, a C compiler, pkgconf (or pkg-config), and gettext
+- libgcrypt
+- libsodium 1.0.18 or newer
+- libgmp
+- libunistring
+- libidn2 (or libidn)
+- libltdl (part of libtool)
+- jansson
+- libmicrohttpd
+- libcurl 7.85.0 or newer, ideally linked against GnuTLS
+- SQLite 3.35.0 or newer
+- zlib
+- recutils (only when building from git)
+
+Optional, for specific features:
+
+- GnuTLS: the HTTP/3 communicator (together with ngtcp2 and nghttp3) and
+ the GNS proxy
+- libextractor: metadata extraction for file-sharing (recommended, see
+ below)
+- libpq (PostgreSQL): an alternative database backend
+- libjose: OpenID Connect support in re:claimID
+- libzbar: ``gnunet-qr``
+- libopus, libogg, and libpulse or GStreamer: the conversation
+ application
+- miniupnpc (the ``upnpc`` program): NAT traversal via UPnP
+- iptables (on GNU/Linux): the VPN and DNS interception
+- a TeX Live installation: ``gnunet-bcd`` (see :ref:`gns-sharing`)
+
+On libextractor: if you build GNUnet without it, installing it later has no
+effect. If you package GNUnet for a distribution, consider splitting
+libextractor into a base package and plugins rather than leaving it out.
+Note that the set of installed libextractor plugins can leak, in a minor
+way, through the metadata of files you publish.
+
+On the C library: GNUnet works with C libraries other than glibc, but then
+lacks the :ref:`NSS plugin <gns-integration>`; GNS still works through
+``gnunet-gns`` and DNS2GNS. See https://bugs.gnunet.org/view.php?id=5605
+for an example build against NetBSD's libc.
+
+For running the test suite, you additionally need Python 3, a POSIX shell,
+and common utilities such as ``which``, ``bc``, ``curl``, ``sed`` and
+``awk``. The test suite only runs after ``meson install``.
+
+At runtime, ``gnunet-gns-proxy-setup-ca`` needs the ``certutil`` program
+from NSS and ``openssl``.
diff --git a/quickstart.rst b/quickstart.rst
@@ -0,0 +1,119 @@
+.. _quickstart:
+
+Quick start
+===========
+
+This page takes you from nothing to your own working zone in the GNU Name
+System (GNS) in five steps. It takes about five minutes and needs no
+configuration file.
+
+.. note::
+
+ GNUnet is alpha software. GNS is its most mature component, but expect
+ rough edges and do not rely on it for anything critical yet. See
+ :doc:`status`.
+
+1. Install GNUnet
+-----------------
+
+Install the ``gnunet`` package of your operating system, for example:
+
+.. code-block:: console
+
+ $ sudo apt install gnunet # Debian, Ubuntu, ...
+ $ guix install gnunet # GNU Guix
+ $ brew install gnunet # macOS with Homebrew
+
+Distribution packages are sometimes outdated, and peers of different major
+versions (say, 0.28.x and 0.29.x) may be unable to talk to each other.
+Check that ``gnunet-arm --version`` reports the `latest release
+<https://www.gnunet.org/en/news/>`__; if it does not, :doc:`install from
+source <installing>`. :doc:`installing` also covers other systems.
+
+Some packages run GNUnet as a system service shared by all users of the
+machine. If yours created a ``gnunet`` group, add yourself to it and log in
+again:
+
+.. code-block:: console
+
+ $ sudo usermod -aG gnunet $USER
+
+2. Start your peer
+------------------
+
+.. code-block:: console
+
+ $ gnunet-arm -s
+
+This starts the GNUnet services in the background and returns immediately.
+(``gnunet-arm -e`` stops them again.)
+
+3. Create a zone
+----------------
+
+A zone is the GNS equivalent of a domain, except that nobody rents it to you
+and nobody can take it away: it is a key pair you generate yourself.
+
+.. code-block:: console
+
+ $ gnunet-identity -C myzone
+
+You now control the top-level domain ``.myzone`` on your machine.
+
+4. Add a record
+---------------
+
+.. code-block:: console
+
+ $ gnunet-namestore -z myzone -a -n www -t A -V 192.0.2.1 -e 1d
+
+This maps the name ``www.myzone`` to the IPv4 address ``192.0.2.1``, valid
+for one day.
+
+5. Resolve it
+-------------
+
+.. code-block:: console
+
+ $ gnunet-gns -u www.myzone
+ >>> Looking for `A' records under `www.myzone'
+ Resolving... <<< 1 record(s) found:
+
+ A: `192.0.2.1'
+
+That's it: you have a working GNS zone.
+
+Next: link to a friend's zone
+-----------------------------
+
+Zones become useful when they point to each other. Records are private
+unless you publish them, so first add a *public* record (``-p``) and print
+your zone's public key:
+
+.. code-block:: console
+
+ $ gnunet-namestore -z myzone -a -p -n blog -t A -V 192.0.2.2 -e 1d
+ $ gnunet-identity -d -q -e myzone
+ 000G0507HZ8M09QWA93N2YX9Y9357777C0HTPK7PA0HTKJS5GCXXPGKBWC
+
+Give this key to a friend, and ask for theirs. If your friend Alice gives
+you her key, add it to your zone under a name of your choice:
+
+.. code-block:: console
+
+ $ gnunet-namestore -z myzone -a -n alice -t EDKEY -V 000G0...ALICEKEY -e 1d
+
+Now ``blog.alice.myzone`` resolves to whatever Alice published as ``blog``.
+Both peers need to be connected to the GNUnet network for this; the first
+lookup can take a while.
+
+Where to go next
+----------------
+
+- :doc:`gns/integration`: use GNS names in your browser and other
+ applications.
+- :doc:`gns/sharing`: get a name under ``.pin.gns.alt`` that every GNUnet
+ user can resolve.
+- :doc:`gns/concepts`: understand zones, delegation and start zones.
+- :doc:`users/start`: keep your peer running, and check that it is
+ connected.
diff --git a/requirements.txt b/requirements.txt
@@ -0,0 +1,3 @@
+certifi==2026.5.20
+meson==1.11.1
+wheel==0.47.0
diff --git a/status.rst b/status.rst
@@ -0,0 +1,117 @@
+.. _status:
+
+What works today
+================
+
+GNUnet is a research project that is still in an early (alpha) stage. Some
+of its components are ready for early adopters, others are work in
+progress. This page tells you which is which, so that you can decide where
+to invest your time.
+
+*Last reviewed for GNUnet 0.29 (November 2026).*
+
+In short
+--------
+
+- **Start with the GNU Name System (GNS).** It is the most mature
+ component, it is specified in :rfc:`9498`, and it is useful on its own.
+ See :doc:`gns/index`.
+- **Everything else is under development.** The other applications work
+ to varying degrees and are worth exploring if you are curious or want to
+ help, but expect bugs, missing features and incompatible changes.
+- **The network is small.** As a consequence it cannot yet provide strong
+ anonymity, and there is not much content to find.
+- **Releases may be incompatible.** Peers of different major versions (for
+ example 0.28.x and 0.29.x) may not be able to communicate. Keep your
+ installation up to date.
+- **There are known privacy issues**, in particular for mobile users.
+
+Applications
+------------
+
+.. list-table::
+ :header-rows: 1
+ :widths: 18 37 15 30
+
+ * - Application
+ - What it does
+ - Status
+ - Notes
+ * - :doc:`GNS <gns/index>`
+ - Decentralized, censorship-resistant name system
+ - **Most mature**
+ - Protocol specified in :rfc:`9498`; stable API. Lookups in
+ remote zones depend on the network and can be slow.
+ * - :doc:`re:claimID <users/reclaim>`
+ - Self-sovereign identity provider with OpenID Connect support
+ - In development
+ - Specified in LSD 0002.
+ * - :doc:`Messenger <users/messenger>`
+ - Decentralized, end-to-end encrypted group messaging
+ - In development
+ - Use the separate applications `messenger-gtk
+ <https://git.gnunet.org/messenger-gtk.git/>`__ or `messenger-cli
+ <https://git.gnunet.org/messenger-cli.git/>`__; the bundled
+ ``gnunet-messenger`` tool is meant for testing.
+ * - :doc:`File-sharing <users/fs>`
+ - Anonymous, censorship-resistant file-sharing
+ - In development
+ - Known moderate design issues that affect usability and
+ performance.
+ * - :doc:`VPN <users/vpn>`
+ - Tunnel IP traffic over GNUnet, with IPv4/IPv6 protocol translation
+ - In development
+ - GNU/Linux only. Provides no anonymity.
+ * - Conversation
+ - Decentralized voice calls
+ - Experimental
+ -
+
+Network foundations
+-------------------
+
+These subsystems carry all of the above. Their state determines how well
+remote operations, such as looking up names in other people's zones, work
+in practice.
+
+.. list-table::
+ :header-rows: 1
+ :widths: 25 40 35
+
+ * - Subsystem
+ - Role
+ - Status
+ * - TRANSPORT and communicators
+ - Connectivity between peers over UDP, TCP, QUIC, ...
+ - In development. GNUnet 0.29 fixed several regressions and
+ instabilities.
+ * - CORE
+ - Encrypted, authenticated links between neighbouring peers
+ - In development; incompatible changes expected.
+ * - CADET
+ - End-to-end encrypted channels across the network
+ - Known moderate implementation limitations that affect
+ performance.
+ * - DHT (R5N)
+ - Distributed hash table used by GNS and others
+ - Being specified in LSD 0004.
+ * - SET, SETU, SETI
+ - Set reconciliation and intersection
+ - Minor implementation limitations that create unnecessary attack
+ surface for availability. The original SET subsystem is deprecated
+ in favour of SETU and SETI.
+ * - RPS
+ - Random peer sampling
+ - Experimental.
+
+For developers, :ref:`Subsystem-stability` lists the expected stability of
+each subsystem's network protocol, IPC protocol and C API. The `bug tracker
+<https://bugs.gnunet.org/>`__ lists the specific open issues.
+
+How you can help
+----------------
+
+Running a peer is the simplest way to help: every additional peer makes the
+network more useful and its anonymity stronger. Reporting bugs, testing
+release candidates and improving this documentation help just as much. See
+:doc:`developers/contributing`.
diff --git a/users/fs.rst b/users/fs.rst
@@ -1,6 +1,11 @@
File-sharing
------------
+.. note::
+
+ File-sharing is under development and has known design issues that affect
+ usability and performance. See :doc:`/status`.
+
This chapter documents the GNUnet file-sharing application. The original
file-sharing implementation for GNUnet was designed to provide anonymous
file-sharing. However, over time, we have also added support for
diff --git a/users/gns.rst b/users/gns.rst
@@ -1,928 +1,15 @@
+:orphan:
+
The GNU Name System
-------------------
-The GNU Name System (GNS) is secure and decentralized naming system. It
-allows its users to register names as top-level domains (TLDs) and
-resolve other namespaces within their TLDs.
-
-GNS is designed to provide:
-
-- Censorship resistance
-- Query privacy
-- Secure name resolution
-- Compatibility with DNS
-
-Unlike DNS, GNS does not rely on central root zones or authorities.
-Instead any user administers their own root and can can create arbitrary
-name value mappings. Furthermore users can delegate resolution to other
-users’ zones just like DNS NS records do. Zones are uniquely identified
-via public keys and resource records are signed using the corresponding
-public key. Delegation to another user’s zone is done using special delegation
-records and petnames. A petname is a name that can be freely chosen by
-the user. This results in non-unique name-value mappings as www.bob to
-one user might be www.friend for someone else.
-
-For a complete specification of the protocol, we refer to `LSD0001 <https://lsd.gnunet.org/lsd0001>`__.
-
-Start Zones
-~~~~~~~~~~~
-
-In GNS you are the master of your own Root.
-This means that you are able (and encouraged!) to manage your own *Start Zones*.
-A *Start Zone* is a mapping from a name suffix (e.g. ``.gnunet.org``) to
-a GNS zone key.
-A user's configuration of *Start Zones* is the equivalent of DNS's `Root Zone <https://en.wikipedia.org/wiki/DNS_root_zone>`_.
-In other words, in GNS you can be your very own `ICANN <https://en.wikipedia.org/wiki/ICANN>`_.
-
-There are three types of *Start Zones*:
-
- 1. Mappings to *remote* GNS zones.
- 2. Mappings to your own *local* GNS zones.
- 3. Explicit zone Top-Level-Domains (zTLDs)
-
-Remote
-""""""
-
-Your GNUnet installation ships with a default configuration of remote
-*Start Zones*.
-The first is ``.gnunet.org``, which points to the authoritative zone of the
-GNUnet project.
-This *Start Zone* allows you to resolve names ending with ``.gnunet.org``.
-It can be used to resolve, for example, ``www.gnunet.org``.
-
-You can try it out for yourself (if you are connected to the peer-to-peer network):
-
-::
-
- $ gnunet-gns --lookup=www.gnunet.org
-
-
-Another *Start Zone* configured by default is ``.pin.gns.alt``.
-It points to a special zone also managed by the GNUnet project.
-Users may register subdomains on a first-come first-served-basis at https://fcfs.gnunet.org.
-
-Use ``gnunet-config -s gns`` to view the GNS configuration, including all
-configured external zones that are operated by other users.
-The respective configuration entry names start with a ``.``, e.g. ``.pin.gns.alt``.
-
-You can change this default configuration at any time and add *Start Zone* to the
-respective zones of your friends! For this, simply obtain the respective
-public key (you will learn how below) and extend the configuration:
-
-::
-
- $ gnunet-config -s gns -o .myfriend -V PUBLIC_KEY
-
-Local
-"""""
-
-In order to create a local GNS zone you have to create a so-called ``ego`` in GNUnet.
-Egos may correspond to pseudonyms or real-world identities.
-It is recommended to use separate egos for separate activities.
-All egos can potentially serve as a GNS zone, but you do not have to use
-all egos as GNS zones.
-All your local zones are used as *Start Zones*.
-
-Technically, an ego is first of all a public-private key pair, and thus
-egos also always correspond to a GNS zone. Egos are managed through the
-IDENTITY service and its tooling.
-The IDENTITY service is used to create and stores private keys with associated
-human-readable nicknames.
-Those mappings serve as local *Start Zones*.
-
-For example, you can create a new ego (or zone) with name *myzone* using the
-``gnunet-identity`` tool using:
-
-::
-
- $ gnunet-identity --create="myzone"
-
-Henceforth, on your system you control the TLD ``myzone``.
-You are able to resolve GNS names ending with ``.myzone``.
-However, the zone is still empty.
-We will look at local zone maintenance in the next section.
-Note that by default, this will create ECDSA key pairs.
-GNS also supports the use of EdDSA keys.
-In order to create a EdDSA zone, execute:
-
-::
-
- $ gnunet-identity --create="myedzone" --eddsa
-
-All of your zones can be listed (displayed) using the gnunet-identity
-command line tool as well:
-
-::
-
- $ gnunet-identity --display
-
-
-
-Zone Top-Level-Domains
-""""""""""""""""""""""
-
-If you know the public key of a GNS zone, you can resolve records in that zone
-without a *Start Zone* mapping.
-To do so, you must provide the string-representation of the zone key
-as Top-Level Domain. Consider the zone key of ``.gnunet.org`` in the default
-configuration. In order to resolve the same record as above using its respective
-zTLD, you can do so as follows:
-
-::
-
- $ gnunet-gns --lookup=www.000G0047M3HN599H57MPXZK4VB59SWK4M9NRD68E1JQFY3RWAHDMKAPN30
-
-The use of zTLDs is mostly useful in the absence of a *Start Zone* configuration
-for that zone or when querying names programmatically.
-
-Local zone maintenance
-~~~~~~~~~~~~~~~~~~~~~~
-
-So you are ready to maintain your own GNS zone? Great.
-First you should review and optionally configure your NAMESTORE database which
-will hold your zone's resource records.
-You have the choice between ``sqlite`` and ``postgres`` as databases.
-Most users will be served well with the ``sqlite`` backend and with its default
-configuration.
-To change the database backend of your NAMESTORE to ``postgres`` execute:
-
-::
-
- $ gnunet-config -s namestore -o DATABASE -V postgres
-
-You can now configure the ``postgres`` database backend to find your database.
-See the ``namestore-postgres`` configuration section on possible options:
-
-::
-
- $ gnunet-config -s namestore-postgres
-
-If you ever need to reset your zone database, you can use ``gnunet-namestore-dbtool``
-with the ``--reset`` option.
-**DANGER: This command will delete ALL of your records in all zones!**
-
-Orphaning
-"""""""""
-
-It can happen that you accidentally or semi-accidentally delete your ego, e.g.
-using ``gnunet-identity --delete=myzone``.
-In this case the records stored in the zone are still in the database but the
-missing ego causes them to be *orphaned*.
-
-Orphaned records are still published in the DHT but access to them is limited.
-For example, managing records through the NAMESTORE API is not longer possible.
-
-You can check for orphaned records using:
-
-::
-
- $ gnunet-namestore --list-orphans
-
-This will output any orphaned records in the format
-
-.. code-block:: text
-
- <$LABEL>.<$PRIVATE_KEY>:
- $RECORD_DATA
-
-In order to recover (and un-orphan) the records, you can re-add the ego by
-executing
-
-::
-
- $ gnunet-identity --create=<myzone> --privkey=$PRIVATE_KEY
-
-If instead you should decide that you want to **purge** all orphaned records,
-you can execute
-
-::
-
- $ gnunet-namestore --purge-orphans
-
-Records
-"""""""
-
-At this point you can add (or edit, or remove) records in your GNS zone using the
-using the gnunet-namestore command-line tool.
-Your records will be stored in a database
-of the NAMESTORE service. Note that for multi-user setups, the NAMESTORE
-database will include the combined records of all users.
-However, users will not be able to see each other’s records if they are marked
-as private.
-
-To provide a short example for editing your own zone, suppose you have
-your own web server with the IP ``1.2.3.4``. Then you can put an A record (A
-records in DNS are for IPv4 IP addresses) into your local zone ``myzone``
-using the command:
-
-::
-
- $ gnunet-namestore --zone=myzone \
- --add \
- --name=www \
- --type=A \
- --value=1.2.3.4 \
- --expiration=1d
-
-Similar commands will work for other types of DNS and GNS records, the
-syntax largely depending on the type of the record. Naturally, most
-users may find editing the zones using the gnunet-namestore-gtk GUI to
-be easier.
-
-Note that by default, records are **private**. This means that they are not
-published and cannot be resolved by other users that may have configured your
-zone as one of their ``Start Zones``.
-In order to create a public record, provide the ``--public`` switch to the
-``gnunet-namestore`` command above.
-
-You can now try to resolve your own GNS record. The method we found
-to be the most uncomplicated is to do this by explicitly resolving using
-``gnunet-gns``.
-In the shell, type:
-
-::
-
- $ gnunet-gns -u www.myzone
-
-You should receive the record as configured above.
-At this point, you can try addition additional records to ``www`` or add other
-names to your zone.
-
-Each zone in GNS has a public-private key. Usually, gnunet-namestore and
-gnunet-setup will access your private key as necessary, so you do not
-have to worry about those. What is important is your public key (or
-rather, the hash of your public key), as you will likely want to give it
-to others so that they can securely link to you.
-
-A central operation in GNS is the ability to securely delegate to other
-zones. Basically, by adding a delegation you make all of the names from
-the other zone available to yourself. This section describes how to
-create delegations.
-
-Suppose you have a friend who you call ’bob’ who also uses GNS. You can
-then delegate resolution of names to Bob’s zone by adding a PKEY record
-to their local zone:
-
-::
-
- $ gnunet-namestore -a -n bob --type PKEY -V $BOBKEY -e 1d -z myzone
-
-Note that ``$BOBKEY`` in the command above must be replaced with the hash of
-Bob’s public key (the output your friend obtained using the
-gnunet-identity command from the previous section and told you, for
-example by giving you a business card containing this information as a
-QR code).
-
-Assuming Bob has an ``A`` record for their website under the name of ``www``
-in his zone, you can then access Bob’s website under ``www.bob.myzone`` -
-as well as any (public) GNS record that Bob has in their zone by
-replacing www with the respective name of the record in Bob’s zone.
-
-Furthermore, if Bob has themselves a (public) delegation to Carol’s zone
-under *carol*, you can access Carol’s records under
-*NAME.carol.bob.myzone* (where *NAME* is the name of Carol’s record you
-want to access).
-
-You can find more detailed information in available record types in the
-`Resource Records`_ section.
-
-Revocation
-""""""""""
-
-Now, in the situation of an attacker gaining access to the private key
-of one of your egos, the attacker can create records in the respective
-GNS zone and publish them as if you published them. Anyone resolving
-your domain will get these new records and when they verify they seem
-authentic because the attacker has signed them with your key.
-
-To address this potential security issue, you can pre-compute a
-revocation certificate corresponding to your ego. This certificate, when
-published on the P2P network, flags your private key as invalid, and all
-further resolutions or other checks involving the key will fail.
-
-A revocation certificate is thus a useful tool when things go out of
-control, but at the same time it should be stored securely. Generation
-of the revocation certificate for a zone can be done through
-``gnunet-revocation``. For example, the following command (as unprivileged
-user) generates a revocation file ``revocation.dat`` for the zone ``myzone``:
-
-::
-
- $ gnunet-revocation -f revocation.dat -R myzone
-
-The above command only pre-computes a revocation certificate. It does
-not revoke the given zone. Pre-computing a revocation certificate
-involves computing a proof-of-work and hence may take up to 4 to 5 days
-on a modern processor. Note that you can abort and resume the
-calculation at any time. Also, even if you did not finish the
-calculation, the resulting file will contain the signature, which is
-sufficient to complete the revocation process even without access to the
-private key. So instead of waiting for a few days, you can just abort
-with ``CTRL-C``, backup the revocation certificate and run the calculation
-only if your key actually was compromised. This has the disadvantage of
-revocation taking longer after the incident, but the advantage of saving
-a significant amount of energy. So unless you believe that a key
-compromise will need a rapid response, we urge you to wait with
-generating the revocation certificate. Also, the calculation is
-deliberately expensive, to deter people from doing this just for fun (as
-the actual revocation operation is expensive for the network, not for
-the peer performing the revocation).
-
-To avoid TL;DR ones from accidentally revoking their zones, we are not
-giving away the command, but it is uncomplicated: the actual revocation
-is performed by using the ``-p`` option of ``gnunet-revocation``.
-
-Backup
-""""""
-
-One should always backup their files, especially in these SSD days (our
-team has suffered 3 SSD crashes over a span of 2 weeks). Backing up peer
-identity and zones is achieved by copying the following files:
-
-The peer identity file can be found in
-``~/.local/share/gnunet/private_key.ecc``.
-
-The private keys of your egos are stored in the directory
-``~/.local/share/gnunet/identity/egos/``. They are stored in files whose
-filenames correspond to the zones’ ego names. These are probably the
-most important files you want to backup from a GNUnet installation.
-
-Note: All these files contain cryptographic keys and they are stored
-without any encryption. So it is advisable to backup encrypted copies of
-them.
-
-This does **NOT** backup your record data.
-Your record data is stored in the respective database backend, and hence
-you may want to backup the respective databases as well.
-
-
-Large Zones
-"""""""""""
-
-Large zones in DNS such as Top-Level Domains can grow to millions of names.
-For example, the ``.fr`` DNS zone contained over 3.9 million entries in 2022.
-Such use cases require good performance and careful consideration of a couple
-of choices in GNS and NAMESTORE.
-
-The cryptographic operations of EdDSA are significantly faster than their
-ECDSA counterparts.
-Most users will be unable to observe this difference.
-But, if you plan on managing large zones, you may want to consider using EdDSA
-zones.
-
-For large zones, using the PostgreSQL NAMESTORE database backend is a must.
-Further, when importing large numbers of records at once, it is advised to make
-use of the bulk import functionality of the NAMESTORE API.
-There is a :ref:`C API for bulk imports <NAMESTORE-Subsystem>`.
-There is also a :ref:`REST API endpoint <namestore_rest_api>` which calls this API.
-
-Zones are published and signed by the ZONEMASTER service.
-In order to speed up this process for large zones, you can edit the number of
-spawned worker threads for the service:
-
-::
-
- $ gnunet-config -s zonemaster -o WORKER_COUNT -V $COUNT
-
-It is reasonable to choose your number of CPUs as ``$COUNT``.
-
-Application integration
-~~~~~~~~~~~~~~~~~~~~~~~
-
-Once you have verified that resolution of names using ``gnunet-gns`` works,
-we can start integrating GNS with applications.
-We can distinguish between two types of applications:
-
- 1. **GNS-aware applications**: Such applications know what GNS is and how it
- can be used to resolve names. Examples are some of GNUnet's services such
- as `re:claimID <reclaimID-Identity-Provider>`_ or `Messenger <Using-the-GNUnet-Messenger>`_.
- 2. **GNS-unaware applications**: Applications that implicitly assume the names
- are resolved through DNS. Such applications use OS-specific APIs or query
- configured DNS servers directly.
-
-In the following, we will discuss integration paths of GNS-unaware applications.
-
-Integration with Browsers (DNS2GNS service)
-"""""""""""""""""""""""""""""""""""""""""""
-
-Most OSes allow you to either modify your ``/etc/resolv.conf`` directly
-or through ``resolvectl``. We are going to configure the dns2gns service
-in order to translate DNS name queries by applications to GNS name
-queries where applicable and else fall back to DNS.
-
-Optionally, you may want to configure your dns2gns service to run on a
-non-privileged port like 5353. But, in case you are going to edit
-``/etc/resolv.conf`` directly, the dns2gns service MUST run on port 53
-as you cannot specify the port number. A ``$FALLBACK_DNS`` variable
-should be a DNS server you trust such as your local router:
-
-::
-
- $ gnunet-config -s dns2gns -o OPTIONS -V "-d $FALLBACK_DNS -p 5252"
- $ gnunet-arm -i dns2gns
-
-If you edit your resolv.conf directly, it should contain and entry like
-this:
-
-::
-
- nameserver 127.0.0.1
-
-In any case, it is very likely that the method of modification of your
-resolver is OS specific. Recently, the combination of NetworkManager and
-systemd-resolved is becoming increasingly popular.
-
-If you use resolvectl and systemd-resolved you can temporarily set the
-nameserver like this:
-
-::
-
- $ resolvectl $INTERFACE 127.0.0.1:5353
-
-Where ``$INTERFACE`` is your network interface such as ``eth0``.
-
-In order to automatically set the DNS2GNS server if it is running
-already you can use NetworkManager-dispatcher. First, enable it:
-
-::
-
- $ sudo systemctl enable NetworkManager-dispatcher.service
- $ sudo systemctl start NetworkManager-dispatcher.service
-
-Then, create a script ``/etc/NetworkManager/dispatch.h/10-dns2-gns.sh``:
-
-.. code-block:: bash
-
- #!/bin/sh
- interface=$1
- status=$2
-
- if [ "$interface" = "eth0" ]; then
- case $status in
- up)
- if nc -u -z 127.0.0.1 5353; then
- resolvectl dns $interface 127.0.0.1:5353
- fi
- ;;
- down)
- ;;
- esac
- fi
-
-Make sure the script is owned by root and executable:
-
-::
-
- $ sudo root:root /etc/NetworkManager/dispatch.d/10-dns2gns.sh
- $ sudo +x /etc/NetworkManager/dispatch.d/10-dns2gns.sh
-
-You can test accessing this website using your browser or curl:
-
-::
-
- $ curl www.gnunet.org
-
-Note that *gnunet.org* is a domain that also exists in DNS and for which
-the GNUnet project webservers can provide trusted TLS certificates. When
-using non-DNS names with GNS or aliases, this may result in issues when
-accessing HTTPS websites with browsers. In order learn how to provide
-relief for this issue, read on.
-
-Integration with Browsers (SOCKS proxy)
-"""""""""""""""""""""""""""""""""""""""
-
-While we recommend integrating GNS using the DNS2GNS service or the
-NSSwitch plugin, you can also integrate GNS directly with your browser
-via the gnunet-gns-proxy. This method can have the advantage that the
-proxy can validate TLS/X.509 records and thus strengthen web security;
-however, the proxy is still a bit brittle, so expect subtle failures. We
-have had reasonable success with Chromium, and various frustrations with
-Firefox in this area recently.
-
-The first step is to start the proxy. As the proxy is (usually) not
-started by default, this is done as a unprivileged user using gnunet-arm
--i gns-proxy. Use gnunet-arm -I as a unprivileged user to check that the
-proxy was actually started. (The most common error for why the proxy may
-fail to start is that you did not run gnunet-gns-proxy-setup-ca during
-installation.) The proxy is a SOCKS5 proxy running (by default) on port
-7777. Thus, you need to now configure your browser to use this proxy.
-With Chromium, you can do this by starting the browser as a unprivileged
-user using chromium –proxy-server=*socks5://localhost:7777* For Firefox
-(or Icecat), select *Edit-Preferences* in the menu, and then select the
-*Advanced* tab in the dialog and then *Network*:
-
-Here, select *Settings…* to open the proxy settings dialog. Select
-*Manual proxy configuration* and enter localhost with port 7777 under
-SOCKS Host. Furthermore, set the checkbox *Proxy DNS when using SOCKS
-v5* at the bottom of the dialog. Finally, push *OK*.
-
-You must also go to about:config and change the
-browser.fixup.alternate.enabled option to false, otherwise the browser
-will autoblunder an address like www.gnu to www.gnu.com. If you want to
-resolve @ in your own TLDs, you must additionally set
-browser.fixup.dns_first_use_for_single_words to true.
-
-After configuring your browser, you might want to first confirm that it
-continues to work as before. (The proxy is still experimental and if you
-experience *odd* failures with some webpages, you might want to disable
-it again temporarily.) Next, test if things work by typing
-*http://test.gnu/* into the URL bar of your browser. This currently
-fails with (my version of) Firefox as Firefox is super-smart and tries
-to resolve *http://www.test.gnu/* instead of *test.gnu*. Chromium can be
-convinced to comply if you explicitly include the *http://* prefix —
-otherwise a Google search might be attempted, which is not what you
-want. If successful, you should see a simple website.
-
-Note that while you can use GNS to access ordinary websites, this is
-more an experimental feature and not really our primary goal at this
-time. Still, it is a possible use-case and we welcome help with testing
-and development.
-
-NSS plugin
-""""""""""
-
-To activate the plugin, you need to edit your
-``/etc/nsswitch.conf`` where you should find a line like this:
-
-::
-
- hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4
-
-The exact details may differ a bit, which is fine. Add the text
-``gns [NOTFOUND=return]`` after ``files``:
-
-::
-
- hosts: files gns [NOTFOUND=return] mdns4_minimal [NOTFOUND=return] dns mdns4
-
-Note that some systems, in particular those running systemd-resolvd, manual modification of ``/etc/nsswitch.conf`` is not possible as it is autogenerated.
-In such cases it is better to use the *DNS2GNS* service approach in combination with ``resolvectl`` as explained above.
-
-Zone dissemination
-~~~~~~~~~~~~~~~~~~
-
-So you have decided to maintain and publish one or more GNS zones.
-You are probably asking yourself how your zone finds its way into the
-*Start Zone* configuration of other users.
-
-Below are pointers on how to make your zone known.
-
-The .pin zone
-"""""""""""""
-
-First, you may want to visit https://fcfs.gnunet.org and register your zone(s)
-with the ``.pin.gns.alt`` top-level domain shipped by default with GNUnet.
-This will allow other GNUnet users to resolve your zone under the name you
-managed to acquire for yourself.
-
-The registration policy of PIN is "first-come, first-served".
-
-Be Social
-"""""""""
-
-Next, you should print out your business card and be social. Find a
-friend, help them install GNUnet and exchange business cards with them.
-Or, if you’re a desperate loner, you might try the next step with your
-own card. Still, it’ll be hard to have a conversation with yourself
-later, so it would be better if you could find a friend. You might also
-want a camera attached to your computer, so you might need a trip to the
-store together.
-
-Before we get started, we need to tell gnunet-qr which zone it should
-import new records into. For this, run:
-
-::
-
- $ gnunet-identity -s namestore -e NAME
-
-where NAME is the name of the zone you want to import records into. In
-our running example, this would be *gnu*.
-
-Henceforth, for every business card you collect, simply run:
-
-::
-
- $ gnunet-qr
-
-to open a window showing whatever your camera points at. Hold up your
-friend’s business card and tilt it until the QR code is recognized. At
-that point, the window should automatically close. At that point, your
-friend’s NICKname and their public key should have been automatically
-imported into your zone.
-
-Assuming both of your peers are properly integrated in the GNUnet
-network at this time, you should thus be able to resolve your friends
-names. Suppose your friend’s nickname is *Bob*. Then, type
-
-::
-
- $ gnunet-gns -u test.bob
-
-to check if your friend was as good at following instructions as you
-were.
-
-
-Creating a business card
-""""""""""""""""""""""""
-
-Before we can really use GNS, you should create a business card. Note
-that this requires having LaTeX installed on your system. If you are
-using a Debian GNU/Linux based operating system, the following command
-should install the required components. Keep in mind that this requires
-3GB of downloaded data and possibly even more when unpacked. On a GNU
-Guix based system texlive 2017 has returns a DAG size of 5032.4 MiB. The
-packages which are confirmed to be required are:
-
-- texlive-units
-- texlive-labels
-- texlive-pst-barcode
-- texlive-luatex85
-- texlive-preview
-- texlive-pdfcrop
-- texlive-koma-script
-
-**We welcome any help in identifying the required components of the
-TexLive Distribution. This way we could just state the required
-components without pulling in the full distribution of TexLive.**
-
-::
-
- apt-get install texlive-full
-
-Start creating a business card by clicking the *Copy* button in
-gnunet-namestore-gtk. Next, you should start the gnunet-bcd program (in
-the terminal, on the command-line). You do not need to pass any options,
-and please be not surprised if there is no output:
-
-::
-
- $ gnunet-bcd
-
-Then, start a browser and point it to http://localhost:8888/ where
-gnunet-bcd is running a Web server!
-
-First, you might want to fill in the *GNS Public Key* field by
-right-clicking and selecting *Paste*, filling in the public key from the
-copy you made in gnunet-namestore-gtk. Then, fill in all of the other
-fields, including your GNS NICKname. Adding a GPG fingerprint is
-optional. Once finished, click *Submit Query*. If your LaTeX
-installation is incomplete, the result will be disappointing. Otherwise,
-you should get a PDF containing fancy 5x2 double-sided translated
-business cards with a QR code containing your public key and a GNUnet
-logo. We’ll explain how to use those a bit later. You can now go back to
-the shell running gnunet-bcd and press CTRL-C to shut down the Web
-server.
-
-The GNUnet default Start Zones
-""""""""""""""""""""""""""""""
-
-The GNUnet default *Start Zones* are managed through `GANA <https://gana.gnunet.org>`_ and
-follows a `restrictive registration policy <https://git.gnunet.org/gana.git/tree/gnu-name-system-default-tlds/POLICY>`_.
-
-The GNUnet GNS Registrar
-""""""""""""""""""""""""
-
-The GNUnet GNS Registrar is a standalone package availiable as download
-in the `GNU FTP Mirror <https://ftpmirror.gnu.org/gnunet>`_.
-
-You can compile it using
-
-::
-
- $ ./configure && make
-
-To run it execute
-
-::
-
- $ ./gnunet-gns-registrar
-
-The registrar requires configuration of a `GNU Taler Merchant <https://www.taler.net/en/>`_.
-You can look at the example configuration for possible options.
-https://fcfs.gnunet.org is running a GNS registrar instance connected to the GNU Taler
-demo exchanges with a merchant running on https://merchant.gnunet.org for testing purposes
-where you can exchange KUDOS (which you get for free upon registration at the GNU Taler Demo
-system).
-Consequently, while requiring KUDOS as payment to register, this is still effectively a
-first-come-first server service.
-
-Resource Records
-~~~~~~~~~~~~~~~~
-
-GNS supports the majority of the DNS records as defined in `RFC
-1035 <http://www.ietf.org/rfc/rfc1035.txt>`_. Additionally, GNS defines
-some new record types the are unique to the GNS system. For example,
-GNS-specific resource records are used to give petnames for zone
-delegation, revoke zone keys and provide some compatibility features.
-
-For some DNS records, GNS does extended processing to increase their
-usefulness in GNS. In particular, GNS introduces special names referred
-to as *zone relative names*. Zone relative names are allowed in some
-resource record types (for example, in NS and CNAME records) and can
-also be used in links on webpages. Zone relative names end in `.+` which
-indicates that the name needs to be resolved relative to the current
-authoritative zone. The extended processing of those names will expand
-the `.+` with the correct delegation chain to the authoritative zone
-(replacing `.+` with the name of the location where the name was
-encountered) and hence generate a valid GNS name.
-
-The GNS currently supports the record types as defined in
-`GANA <https://git.gnunet.org/gana.git/tree/gnu-name-system-record-types/registry.rec>`__.
-In addition, GNS supports DNS record types, such as `A`, `AAAA` or `TXT`.
-
-For a complete description of the records, please refer to the
-specification at `LSD0001 <https://lsd.gnunet.org/lsd0001>`__.
-
-In the following, we discuss GNS records with specific behavior or
-special handling in GNUnet.
-
-
-VPN
-"""
-
-GNS allows easy access to services provided by the GNUnet Virtual Public
-Network. When the GNS resolver encounters a VPN record it will contact
-the VPN service to try and allocate an IPv4/v6 address (if the queries
-record type is an IP address) that can be used to contact the service.
-
-**Example**
-
-I want to provide access to the VPN service *web.gnu.* on port 80 on
-peer ABC012: Name: www; RRType: VPN; Value: 80 ABC012 web.gnu.
-
-The peer ABC012 is configured to provide an exit point for the service
-*web.gnu.* on port 80 to it’s server running locally on port 8080 by
-having the following lines in the gnunet.conf configuration file:
-
-::
-
- [web.gnunet.]
- TCP_REDIRECTS = 80:localhost4:8080
-
-DNS Migration
-~~~~~~~~~~~~~
-
-The following tools facilitate DNS to GNS zone migrations.
-
-The Zoneimport mechanism through ``gnunet-zoneimport`` allows you to mirror delegations in a TLD. We provide an example for AFNIC's ``.fr``.
-Why would you do it this way instead of a zone transfer using AXFR?
-Because some TLD authorities, such as AFNIC, do not support AXFR and only publish a list
-of delegated names.
-In such cases, ``gnunet-zoneimport`` comes in handy. While not perfect in terms
-of mirroring record updates and expirations, it is better than not being able to
-mirror the ``.fr`` TLD at all.
-In general, if the naming authority supports AXFR, then you should consider using Ascension (see below).
-If you are the naming authority, then you should consider manually managing synchronization
-between DNS and GNS or use the ``gnunet-namestore-zonefile`` tool to import DNS zonefiles
-into GNS.
-
-Note that the namestore by default also populates the namecache. This
-pre-population is cryptographically expensive. Thus, on systems that
-only serve to import a large (millions of records) DNS zone and that do
-not have a local gns service in use, it is thus advisable to disable the
-namecache by setting the option *DISABLE* to *YES* in section
-*[namecache]*.
-
-
-Zoneimport
-""""""""""
-
-If you want to support GNS but the master database for a zone is only
-available and maintained in DNS, GNUnet includes the gnunet-zoneimport
-tool to monitor a DNS zone and automatically import records into GNS.
-Today, the tool does not yet support DNS AF(X)R, as we initially used it
-on the *.fr* zone which does not allow us to perform a DNS zone
-transfer. Instead, gnunet-zoneimport reads a list of DNS domain names
-from stdin, issues DNS queries for each, converts the obtained records
-(if possible) and stores the result in the namestore.
-
-The zonemaster service then takes the records from the namestore,
-publishes them into the DHT which makes the result available to the GNS
-resolver. In the GNS configuration, non-local zones can be configured to
-be intercepted by specifying *.tld = PUBLICKEY* in the configuration
-file in the *[gns]* section.
-
-For example, assuming you have the AFNIC NomsDeDomainEnPointFr CSV file you can execute:
-
-::
-
- $ cat OPENDATA_A-NomsDeDomaineEnPointFr.csv | cut -d';' -f 1 | tail -n+2 | gnunet-zoneimport 194.0.9.1
-
-This reads the first value of each line in the CSV as the domain name to import.
-The first line is skipped because it (usually) is a header, not an entry.
-The authoritative DNS server for ``fr`` is provided as IP.
-Note that you need to provide authoritative DNS servers.
-
-Zonefile
-""""""""
-
-In order to import DNS zones which are already defined in a
-`zone file <https://en.wikipedia.org/wiki/Zone_file>`_ in use
-with existing DNS daemons such as BIND you may try to import your DNS zones
-into DNS using ``gnunet-namestore-zonefile``.
-Assuming your zonefile is called ``myzonefile``, execute:
-
-::
-
- $ gnunet-namestore-zonefile < myzonefile
-
-As you can see, the program reads the zonefile from standard input.
-Upon encountering an ``$ORIGIN`` directive a new ego will be created with that
-name unless it already exists.
-Note that ``$ORIGIN`` is a FQDN and ends with a ``.``, e.g. ``example.com.``.
-The ego will be called ``example.com`` (without the ``.``).
-All records following the directive will be imported into the zone identified by
-this origin.
-Multiple ``$ORIGIN`` directives within the same zone file are supported and
-egos will be created accordingly.
-If no ``$ORIGIN`` is explicitly specified in the beginning of the file
-(as is sometimes the case in BIND setups) or if you want to override the (first)
-occurence, you can provide an ego to use:
-
-::
-
- $ gnunet-namestore-zonefile -z myorigin < myzonefile
-
-Note that in the conversion process some information found in the zone file
-is lost and other information is less precise than in GNS.
-For example, records stored in the ``$ORIGIN`` can be specified in multiple
-ways:
-
-.. code-block:: text
-
- $ORIGIN example.com.
- $TTL 3600
- example.com. IN MX 10 mail.example.com. ; mail.example.com is the mailserver for example.com
- @ IN MX 20 mail2.example.com. ; equivalent to above line, "@" represents zone origin
-
-GNS will map the origin ``example.com.`` to the empty label ``@`` whenever possible.
-Further, ``$TTL`` is given in seconds. GNS record expirations are defined in
-microseconds. The seconds will be converted to their equivalent in microseconds.
-Note how this means that conversion of GNS records (in particular their expirations)
-to a DNS zone file cannot be done without loss of information.
-This also means that converting the GNS zone back into a zone file (a function
-currently not implemented) would not yield the same zone file that was imported.
-
-Ascension
-"""""""""
-
-Ascension is a tool to migrate existing DNS zones into GNS.
-
-Compared to the gnunet-zoneimport tool it strictly uses AXFR or IXFR
-depending on whether or not there exists a SOA record for the zone. If
-that is the case it will take the serial as a reference point and
-request the zone. The server will either answer the IXFR request with a
-correct incremental zone or with the entire zone, which depends on the
-server configuration.
-
-Before you can migrate any zone though, you need to start a local GNUnet
-peer. To migrate the Syrian top level domain - one of the few top level
-domains that support zone transfers - into GNS use the following
-command:
-
-::
-
- $ ascension sy. -n ns1.tld.sy. -p
-
-The -p flag will tell GNS to put these records on the DHT so that other
-users may resolve these records by using the public key of the zone.
-
-Once the zone is migrated, Ascension will output a message telling you,
-that it will refresh the zone after the time has elapsed. You can
-resolve the names in the zone directly using GNS or if you want to use
-it with your browser, check out the GNS manual section. Configuring the
-GNU Name System. To resolve the records from another system you need the
-respective zones PKEY. To get the zones public key, you can run the
-following command:
-
-::
-
- $ gnunet-identity -dqe sy
-
-Where *sy* is the name of the zone you want to migrate.
-
-You can share the PKEY of the zone with your friends. They can then
-resolve records in the zone by doing a lookup replacing the zone label
-with your PKEY:
-
-::
-
- $ gnunet-gns -t SOA -u "$PKEY"
-
-The program will continue to run as a daemon and update once the refresh
-time specified in the zones SOA record has elapsed.
-
-DNSCurve style records are supported in the latest release and they are
-added as a PKEY record to be referred to the respective GNS public key.
-Key distribution is still a problem but provided someone else has a
-public key under a given label it can be looked up.
-
-There is an unofficial Debian package called python3-ascension that adds
-a system user ascension and runs a GNUnet peer in the background.
+The documentation of the GNU Name System has moved to its own section:
-Ascension-bind is also an unofficial Debian package that on installation
-checks for running DNS zones and whether or not they are transferable
-using DNS zone transfer (AXFR). It asks the administrator which zones to
-migrate into GNS and installs a systemd unit file to keep the zone up to
-date. If you want to migrate different zones you might want to check the
-unit file from the package as a guide.
+- :doc:`/gns/index`: what GNS is and why to use it
+- :doc:`/gns/concepts`: zones, records, start zones and delegation
+- :doc:`/gns/zones`: managing your zones, backup and revocation
+- :doc:`/gns/integration`: using GNS names in browsers and other
+ applications
+- :doc:`/gns/sharing`: publishing your zone and registering names
+- :doc:`/gns/migration`: importing zones from DNS
+- :doc:`/gns/faq`: frequently asked questions
diff --git a/users/index.rst b/users/index.rst
@@ -1,31 +1,37 @@
-###########
-User Manual
-###########
+.. _applications-in-development:
-This tutorial is supposed to give a first introduction for users trying
-to do something real with GNUnet. Installation and configuration are
-specifically outside of the scope of this tutorial. Instead, we start by
-briefly checking that the installation works, and then dive into
-uncomplicated, concrete practical things that can be done with the
-framework provided by GNUnet.
+###########################
+Applications in development
+###########################
-In short, this chapter of the “GNUnet Reference Documentation” will show
-you how to use the various peer-to-peer applications of the GNUnet
-system. As GNUnet evolves, we will add new sections for the various
-applications that are being created.
+Besides the :doc:`GNU Name System </gns/index>`, GNUnet comes with several
+other applications. They work to varying degrees: they are worth exploring
+if you are curious or want to help, but expect bugs, missing features and
+incompatible changes between releases. See :doc:`/status` for details.
-Comments on the content of this chapter, and extensions of it are always
-welcome.
+:doc:`re:claimID <reclaim>`
+ A self-sovereign identity provider: keep your personal attributes in your
+ own GNS zone and share them selectively with websites via OpenID
+ Connect.
+
+:doc:`Messenger <messenger>`
+ Decentralized, end-to-end encrypted group messaging.
+
+:doc:`File-sharing <fs>`
+ Anonymous, censorship-resistant publishing, searching and downloading of
+ files.
+
+:doc:`VPN <vpn>`
+ Tunnel IP traffic over GNUnet, with IPv4/IPv6 protocol translation.
+
+:doc:`Graphical user interfaces </guis/index>`
+ GTK interfaces for configuring your peer and for file-sharing.
.. toctree::
- :maxdepth: 2
+ :hidden:
- start
- subsystems
- gns
reclaim
+ messenger
fs
vpn
- messenger
- configuration
-
+ /guis/index
diff --git a/users/messenger.rst b/users/messenger.rst
@@ -3,6 +3,11 @@
Messenger
---------
+.. note::
+
+ The messenger is under development; expect bugs and incompatible changes.
+ See :doc:`/status`.
+
The GNUnet Messenger subsystem allows decentralized message-based
communication inside of so called rooms. Each room can be hosted by a
variable amount of peers. Every member of a room has the possibility to
@@ -32,14 +37,16 @@ member can read its content. This will also use additional padding.
Current state
~~~~~~~~~~~~~
-Currently there is only a simplistic CLI application available to use
-the messenger service. You can use this application with the
-``gnunet-messenger`` command.
+For everyday use, there are two separate applications built on the
+client-side library ``libgnunetchat``: the graphical `messenger-gtk
+<https://git.gnunet.org/messenger-gtk.git/>`__ and the terminal-based
+`messenger-cli <https://git.gnunet.org/messenger-cli.git/>`__. The two
+are compatible with each other.
-This application was designed for testing purposes and it does not
-provide full functionality in the current state. It is planned to
-replace this CLI application in later stages with a fully featured one
-using a client-side library designed for messenger applications.
+GNUnet itself only includes a simplistic CLI application, which you can
+use with the ``gnunet-messenger`` command. It was designed for testing
+purposes and does not provide full functionality. The rest of this chapter
+describes it.
.. _Entering-a-room:
diff --git a/users/reclaim.rst b/users/reclaim.rst
@@ -3,6 +3,11 @@
re:claimID
----------
+.. note::
+
+ re:claimID is under development; expect bugs and incompatible changes. See
+ :doc:`/status`.
+
The re:claimID Identity Provider (IdP) is a decentralized IdP service.
It allows its users to manage and authorize third parties to access
their identity attributes such as email or shipping addresses.
diff --git a/users/start.rst b/users/start.rst
@@ -1,77 +1,119 @@
-Getting started
----------------
+.. _running-your-peer:
-Prior to using any GNUnet-based application, one has to start a node:
+Running your peer
+=================
-::
+Your GNUnet *peer* is the set of background services that connect your
+computer to the GNUnet network and that applications such as GNS rely on.
+This page explains how to start and stop it, how to keep it running, and
+how to check that it is healthy.
- $ gnunet-arm -s
+Starting and stopping
+---------------------
-To stop GNUnet:
+.. code-block:: console
-::
+ $ gnunet-arm -s # start the peer
+ $ gnunet-arm -e # stop it
+ $ gnunet-arm -r # restart it
- $ gnunet-arm -e
+``gnunet-arm`` (the *Automatic Restart Manager*) starts the services your
+peer needs and restarts them if they crash. It needs no configuration
+file; settings you change with ``gnunet-config`` are written to
+``~/.config/gnunet.conf``.
-You can usually find the logs under ``~/.cache/gnunet`` and all files
-such as databases and private keys in ``~/.local/share/gnunet``.
+To see which services are running, use ``gnunet-arm -I``. Optional
+services, such as the DNS bridge for GNS, are started and stopped
+individually:
-The list of running services can be displayed using the ``-I`` option.
-It should look similar to this example:
+.. code-block:: console
-::
+ $ gnunet-arm -i dns2gns # start a service
+ $ gnunet-arm -k dns2gns # stop it
- $ gnunet-arm -I
- Running services:
- topology (gnunet-daemon-topology)
- nat (gnunet-service-nat)
- vpn (gnunet-service-vpn)
- gns (gnunet-service-gns)
- cadet (gnunet-service-cadet)
- namecache (gnunet-service-namecache)
- hostlist (gnunet-daemon-hostlist)
- revocation (gnunet-service-revocation)
- zonemaster (gnunet-service-zonemaster)
- zonemaster-monitor (gnunet-service-zonemaster-monitor)
- dht (gnunet-service-dht)
- namestore (gnunet-service-namestore)
- set (gnunet-service-set)
- statistics (gnunet-service-statistics)
- nse (gnunet-service-nse)
- fs (gnunet-service-fs)
- peerstore (gnunet-service-peerstore)
- core (gnunet-service-core)
- rest (gnunet-rest-server)
- transport (gnunet-service-transport)
- datastore (gnunet-service-datastore)
+To start an optional service together with the peer every time, set
+``IMMEDIATE_START = YES`` in its configuration section, for example
+``gnunet-config -s dns2gns -o IMMEDIATE_START -V YES``.
-For the **multi-user** setup first the system services need to be
-started as the system user, i.e. the user gnunet needs to execute
-``gnunet-arm -s``. This should be done by the system’s init system. Then
-the user who wants to start GNUnet applications has to run
-``gnunet-arm -s``, too. It is recommended to automate this, e.g. using
-the user’s crontab.
+Starting your peer automatically
+--------------------------------
-You can check directly connected peers with:
+Your peer is most useful, to you and to others, when it runs all the time.
-::
+- If your distribution's package ships a systemd *user* unit (Fedora's
+ does, for example), enable it with
+ ``systemctl --user enable --now gnunet``.
+- Otherwise, add this line to your crontab (``crontab -e``):
- $ gnunet-core --connection-status
+ .. code-block:: text
-This should return (at least) one established connection peer.
-Otherwise, again, there is likely a problem with your network
-configuration.
+ @reboot gnunet-arm -s
-You can monitor your own current peer identity with:
+Checking that it works
+----------------------
-::
+Your peer's identity in the network:
- $ gnunet-pils (ctrl-c to quit)
+.. code-block:: console
-You can display the current peer identity once with:
+ $ gnunet-pils --once
+ 49MAZRCX4ZH5QMXTT73GQGNVDK8QHTQXAMP8BEVYKHC1M2RMM1EG
-::
+The peers you are currently connected to (one per line; give it a few
+minutes after starting):
- $ gnunet-pils --once
+.. code-block:: console
+ $ gnunet-core -s
+The estimated size of the network:
+
+.. code-block:: console
+
+ $ gnunet-nse
+
+If ``gnunet-core -s`` lists no peers after several minutes, your peer
+cannot reach the network. The most common reasons are a firewall or NAT
+that blocks incoming and outgoing connections, and a failure to download
+the list of initial peers. See :ref:`Configuring-NAT` and
+:ref:`Configuring-the-hostlist-to-bootstrap`. Names in your own zones keep
+working without a connection; only lookups in other people's zones need the
+network.
+
+Where GNUnet keeps its files
+----------------------------
+
+``~/.config/gnunet.conf``
+ Your configuration, if you have changed any settings. See
+ :doc:`configuration`.
+
+``~/.local/share/gnunet/``
+ Persistent data: the peer's key, your zones' keys, and the databases.
+ See :ref:`gns-backup` for what to back up.
+
+``~/.cache/gnunet/``
+ Log files, one per day. Look here first when something goes wrong.
+
+If the ``XDG_CONFIG_HOME``, ``XDG_DATA_HOME`` or ``XDG_CACHE_HOME``
+environment variables are set, these directories are used instead of
+``~/.config``, ``~/.local/share`` and ``~/.cache``.
+
+System-wide installations
+-------------------------
+
+Some distribution packages, and administrators who follow
+:ref:`The-Multi_002dUser-Setup`, run part of GNUnet as a system service:
+the *system services* run once per machine as the user ``gnunet`` and use
+``/etc/gnunet.conf``, while every user who wants to use GNUnet runs their
+own *user services* with ``gnunet-arm -s`` as above. For this to work,
+your user must be a member of the ``gnunet`` group. The system services
+are started by the system's init system, not by you.
+
+Getting help
+------------
+
+- Read the log files in ``~/.cache/gnunet/``.
+- Check the :doc:`/faq` and the `bug tracker <https://bugs.gnunet.org/>`__
+ for known problems, and report new ones there.
+- Ask on the `help-gnunet@gnu.org
+ <https://lists.gnu.org/mailman/listinfo/help-gnunet>`__ mailing list.
diff --git a/users/subsystems.rst b/users/subsystems.rst
@@ -1,2116 +1,7 @@
-.. _subsystems:
+:orphan:
Subsystems
-**********
-
-This section consists brief description of the subsystems that make up
-GNUnet.
-This image is giving an overview over system dependencies and interactions.
-
-.. image:: /images/gnunet-arch-full.svg
-
-CADET - Decentralized End-to-end Transport
-==========================================
-
-The Confidential Ad-hoc Decentralized End-to-end Transport (CADET) subsystem
-in GNUnet is responsible for secure end-to-end
-communications between nodes in the GNUnet overlay network. CADET builds
-on the CORE subsystem, which provides for the link-layer communication,
-by adding routing, forwarding, and additional security to the
-connections. CADET offers the same cryptographic services as CORE, but
-on an end-to-end level. This is done so peers retransmitting traffic on
-behalf of other peers cannot access the payload data.
-
-- CADET provides confidentiality with so-called perfect forward
- secrecy; we use ECDHE powered by Curve25519 for the key exchange and
- then use symmetric encryption, encrypting with both AES-256 and
- Twofish
-
-- authentication is achieved by signing the ephemeral keys using
- Ed25519, a deterministic variant of ECDSA
-
-- integrity protection (using SHA-512 to do encrypt-then-MAC, although
- only 256 bits are sent to reduce overhead)
-
-- replay protection (using nonces, timestamps, challenge-response,
- message counters and ephemeral keys)
-
-- liveness (keep-alive messages, timeout)
-
-Additional to the CORE-like security benefits, CADET offers other
-properties that make it a more universal service than CORE.
-
-- CADET can establish channels to arbitrary peers in GNUnet. If a peer
- is not immediately reachable, CADET will find a path through the
- network and ask other peers to retransmit the traffic on its behalf.
-
-- CADET offers (optional) reliability mechanisms. In a reliable channel
- traffic is guaranteed to arrive complete, unchanged and in-order.
-
-- CADET takes care of flow and congestion control mechanisms, not
- allowing the sender to send more traffic than the receiver or the
- network are able to process.
-
-.. _CORE-Subsystem:
-
-.. index::
- double: CORE; subsystem
-
-CORE - GNUnet link layer
-========================
-
-The CORE subsystem in GNUnet is responsible for securing link-layer
-communications between nodes in the GNUnet overlay network. CORE builds
-on the TRANSPORT subsystem which provides for the actual, insecure,
-unreliable link-layer communication (for example, via UDP or WLAN), and
-then adds fundamental security to the connections:
-
-- confidentiality with so-called perfect forward secrecy; we use ECDHE
- (`Elliptic-curve
- Diffie—Hellman <http://en.wikipedia.org/wiki/Elliptic_curve_Diffie%E2%80%93Hellman>`__)
- powered by Curve25519 (`Curve25519 <http://cr.yp.to/ecdh.html>`__)
- for the key exchange and then use symmetric encryption, encrypting
- with both AES-256
- (`AES-256 <http://en.wikipedia.org/wiki/Rijndael>`__) and Twofish
- (`Twofish <http://en.wikipedia.org/wiki/Twofish>`__)
-
-- `authentication <http://en.wikipedia.org/wiki/Authentication>`__ is
- achieved by signing the ephemeral keys using Ed25519
- (`Ed25519 <http://ed25519.cr.yp.to/>`__), a deterministic variant of
- ECDSA (`ECDSA <http://en.wikipedia.org/wiki/ECDSA>`__)
-
-- integrity protection (using SHA-512
- (`SHA-512 <http://en.wikipedia.org/wiki/SHA-2>`__) to do
- encrypt-then-MAC
- (`encrypt-then-MAC <http://en.wikipedia.org/wiki/Authenticated_encryption>`__))
-
-- Replay (`replay <http://en.wikipedia.org/wiki/Replay_attack>`__)
- protection (using nonces, timestamps, challenge-response, message
- counters and ephemeral keys)
-
-- liveness (keep-alive messages, timeout)
-
-.. _Limitations:
-
-:index:`Limitations <CORE; limitations>`
-Limitations
------------
-
-CORE does not perform
-`routing <http://en.wikipedia.org/wiki/Routing>`__; using CORE it is
-only possible to communicate with peers that happen to already be
-\"directly\" connected with each other. CORE also does not have an API
-to allow applications to establish such \"direct\" connections --- for
-this, applications can ask TRANSPORT, but TRANSPORT might not be able to
-establish a \"direct\" connection. The TOPOLOGY subsystem is responsible
-for trying to keep a few \"direct\" connections open at all times.
-Applications that need to talk to particular peers should use the CADET
-subsystem, as it can establish arbitrary \"indirect\" connections.
-
-Because CORE does not perform routing, CORE must only be used directly
-by applications that either perform their own routing logic (such as
-anonymous file-sharing) or that do not require routing, for example
-because they are based on flooding the network. CORE communication is
-unreliable and delivery is possibly out-of-order. Applications that
-require reliable communication should use the CADET service. Each
-application can only queue one message per target peer with the CORE
-service at any time; messages cannot be larger than approximately 63
-kilobytes. If messages are small, CORE may group multiple messages
-(possibly from different applications) prior to encryption. If permitted
-by the application (using the `cork <http://baus.net/on-tcp_cork/>`__
-option), CORE may delay transmissions to facilitate grouping of multiple
-small messages. If cork is not enabled, CORE will transmit the message
-as soon as TRANSPORT allows it (TRANSPORT is responsible for limiting
-bandwidth and congestion control). CORE does not allow flow control;
-applications are expected to process messages at line-speed. If flow
-control is needed, applications should use the CADET service.
-
-.. when is a peer connected
-.. _When-is-a-peer-_0022connected_0022_003f:
-
-When is a peer \"connected\"?
------------------------------
-
-In addition to the security features mentioned above, CORE also provides
-one additional key feature to applications using it, and that is a
-limited form of protocol-compatibility checking. CORE distinguishes
-between TRANSPORT-level connections (which enable communication with
-other peers) and application-level connections. Applications using the
-CORE API will (typically) learn about application-level connections from
-CORE, and not about TRANSPORT-level connections. When a typical
-application uses CORE, it will specify a set of message types (from
-``gnunet_protocols.h``) that it understands. CORE will then notify the
-application about connections it has with other peers if and only if
-those applications registered an intersecting set of message types with
-their CORE service. Thus, it is quite possible that CORE only exposes a
-subset of the established direct connections to a particular application
---- and different applications running above CORE might see different
-sets of connections at the same time.
-
-A special case are applications that do not register a handler for any
-message type. CORE assumes that these applications merely want to
-monitor connections (or \"all\" messages via other callbacks) and will
-notify those applications about all connections. This is used, for
-example, by the ``gnunet-core`` command-line tool to display the active
-connections. Note that it is also possible that the TRANSPORT service
-has more active connections than the CORE service, as the CORE service
-first has to perform a key exchange with connecting peers before
-exchanging information about supported message types and notifying
-applications about the new connection.
-.. _Distributed-Hash-Table-_0028DHT_0029:
-
-.. index::
- double: Distributed hash table; subsystem
- see: DHT; Distributed hash table
-
-DHT - Distributed Hash Table
-============================
-
-GNUnet includes a generic distributed hash table that can be used by
-developers building P2P applications in the framework. This section
-documents high-level features and how developers are expected to use the
-DHT. We have a research paper detailing how the DHT works. Also, Nate's
-thesis includes a detailed description and performance analysis (in
-chapter 6). [R5N2011]_ [EVANS2011]_
-
-Key features of GNUnet's DHT include:
-
-- stores key-value pairs with values up to (approximately) 63k in size
-
-- works with many underlay network topologies (small-world, random
- graph), underlay does not need to be a full mesh / clique
-
-- support for extended queries (more than just a simple 'key'),
- filtering duplicate replies within the network (bloomfilter) and
- content validation (for details, please read the subsection on the
- block library)
-
-- can (optionally) return paths taken by the PUT and GET operations to
- the application
-
-- provides content replication to handle churn
-
-GNUnet's DHT is randomized and unreliable. Unreliable means that there
-is no strict guarantee that a value stored in the DHT is always found
-— values are only found with high probability. While this is somewhat
-true in all P2P DHTs, GNUnet developers should be particularly wary of
-this fact (this will help you write secure, fault-tolerant code). Thus,
-when writing any application using the DHT, you should always consider
-the possibility that a value stored in the DHT by you or some other peer
-might simply not be returned, or returned with a significant delay. Your
-application logic must be written to tolerate this (naturally, some loss
-of performance or quality of service is expected in this case).
-
-.. _Block-library-and-plugins:
-
-Block library and plugins
--------------------------
-
-.. _What-is-a-Block_003f:
-
-What is a Block?
-^^^^^^^^^^^^^^^^
-
-Blocks are small (< 63k) pieces of data stored under a key (struct
-GNUNET_HashCode). Blocks have a type (enum GNUNET_BlockType) which
-defines their data format. Blocks are used in GNUnet as units of static
-data exchanged between peers and stored (or cached) locally. Uses of
-blocks include file-sharing (the files are broken up into blocks), the
-VPN (DNS information is stored in blocks) and the DHT (all information
-in the DHT and meta-information for the maintenance of the DHT are both
-stored using blocks). The block subsystem provides a few common
-functions that must be available for any type of block.
-
-
-.. [R5N2011] https://bib.gnunet.org/date.html#R5N
-.. [EVANS2011] https://d-nb.info/1015129951
-.. index::
- double: File sharing; subsystem
- see: FS; File sharing
-
-.. _File_002dsharing-_0028FS_0029-Subsystem:
-
-FS — File sharing over GNUnet
-=============================
-
-This chapter describes the details of how the file-sharing service
-works. As with all services, it is split into an API (libgnunetfs), the
-service process (gnunet-service-fs) and user interface(s). The
-file-sharing service uses the datastore service to store blocks and the
-DHT (and indirectly datacache) for lookups for non-anonymous
-file-sharing. Furthermore, the file-sharing service uses the block
-library (and the block fs plugin) for validation of DHT operations.
-
-In contrast to many other services, libgnunetfs is rather complex since
-the client library includes a large number of high-level abstractions;
-this is necessary since the FS service itself largely only operates on
-the block level. The FS library is responsible for providing a
-file-based abstraction to applications, including directories, meta
-data, keyword search, verification, and so on.
-
-The method used by GNUnet to break large files into blocks and to use
-keyword search is called the \"Encoding for Censorship Resistant
-Sharing\" (ECRS). ECRS is largely implemented in the fs library; block
-validation is also reflected in the block FS plugin and the FS service.
-ECRS on-demand encoding is implemented in the FS service.
-
-.. note:: The documentation in this chapter is quite incomplete.
-
-.. _Encoding-for-Censorship_002dResistant-Sharing-_0028ECRS_0029:
-
-.. index::
- see: Encoding for Censorship-Resistant Sharing; ECRS
-
-:index:`ECRS — Encoding for Censorship-Resistant Sharing <single: ECRS>`
-ECRS — Encoding for Censorship-Resistant Sharing
-------------------------------------------------
-
-When GNUnet shares files, it uses a content encoding that is called
-ECRS, the Encoding for Censorship-Resistant Sharing. Most of ECRS is
-described in the (so far unpublished) research paper attached to this
-page. ECRS obsoletes the previous ESED and ESED II encodings which were
-used in GNUnet before version 0.7.0. The rest of this page assumes that
-the reader is familiar with the attached paper. What follows is a
-description of some minor extensions that GNUnet makes over what is
-described in the paper. The reason why these extensions are not in the
-paper is that we felt that they were obvious or trivial extensions to
-the original scheme and thus did not warrant space in the research
-report.
-
-.. todo:: Find missing link to file system paper.
-
-.. index::
- double: GNU Name System; subsystem
- see: GNS; GNU Name System
-
-.. _GNU-Name-System-_0028GNS_0029:
-
-GNS - the GNU Name system
--------------------------
-
-The GNU Name System (GNS) is a decentralized database that enables users
-to securely resolve names to values. Names can be used to identify other
-users (for example, in social networking), or network services (for
-example, VPN services running at a peer in GNUnet, or purely IP-based
-services on the Internet). Users interact with GNS by typing in a
-hostname that ends in a top-level domain that is configured in the "GNS"
-section, matches an identity of the user or ends in a Base32-encoded
-public key.
-
-Videos giving an overview of most of the GNS and the motivations behind
-it is available here and here. The remainder of this chapter targets
-developers that are familiar with high level concepts of GNS as
-presented in these talks.
-
-.. todo:: Link to videos and GNS talks?
-
-GNS-aware applications should use the GNS resolver to obtain the
-respective records that are stored under that name in GNS. Each record
-consists of a type, value, expiration time and flags.
-
-The type specifies the format of the value. Types below 65536 correspond
-to DNS record types, larger values are used for GNS-specific records.
-Applications can define new GNS record types by reserving a number and
-implementing a plugin (which mostly needs to convert the binary value
-representation to a human-readable text format and vice-versa). The
-expiration time specifies how long the record is to be valid. The GNS
-API ensures that applications are only given non-expired values. The
-flags are typically irrelevant for applications, as GNS uses them
-internally to control visibility and validity of records.
-
-Records are stored along with a signature. The signature is generated
-using the private key of the authoritative zone. This allows any GNS
-resolver to verify the correctness of a name-value mapping.
-
-Internally, GNS uses the NAMECACHE to cache information obtained from
-other users, the NAMESTORE to store information specific to the local
-users, and the DHT to exchange data between users. A plugin API is used
-to enable applications to define new GNS record types.
-
-.. index::
- single: GNS; name cache
- double: subsystem; NAMECACHE
-
-.. _GNS-Namecache:
-
-NAMECACHE — DHT caching of GNS results
-======================================
-
-The NAMECACHE subsystem is responsible for caching (encrypted)
-resolution results of the GNU Name System (GNS). GNS makes zone
-information available to other users via the DHT. However, as accessing
-the DHT for every lookup is expensive (and as the DHT's local cache is
-lost whenever the peer is restarted), GNS uses the NAMECACHE as a more
-persistent cache for DHT lookups. Thus, instead of always looking up
-every name in the DHT, GNS first checks if the result is already
-available locally in the NAMECACHE. Only if there is no result in the
-NAMECACHE, GNS queries the DHT. The NAMECACHE stores data in the same
-(encrypted) format as the DHT. It thus makes no sense to iterate over
-all items in the NAMECACHE – the NAMECACHE does not have a way to
-provide the keys required to decrypt the entries.
-
-Blocks in the NAMECACHE share the same expiration mechanism as blocks in
-the DHT – the block expires wheneever any of the records in the
-(encrypted) block expires. The expiration time of the block is the only
-information stored in plaintext. The NAMECACHE service internally
-performs all of the required work to expire blocks, clients do not have
-to worry about this. Also, given that NAMECACHE stores only GNS blocks
-that local users requested, there is no configuration option to limit
-the size of the NAMECACHE. It is assumed to be always small enough (a
-few MB) to fit on the drive.
-
-The NAMECACHE supports the use of different database backends via a
-plugin API.
-
-.. index::
- double: subsystem; NAMESTORE
-
-.. _NAMESTORE-Subsystem:
-
-NAMESTORE — Storage of local GNS zones
-======================================
-
-The NAMESTORE subsystem provides persistent storage for local GNS zone
-information. All local GNS zone information are managed by NAMESTORE. It
-provides both the functionality to administer local GNS information
-(e.g. delete and add records) as well as to retrieve GNS information
-(e.g to list name information in a client). NAMESTORE does only manage
-the persistent storage of zone information belonging to the user running
-the service: GNS information from other users obtained from the DHT are
-stored by the NAMECACHE subsystem.
-
-NAMESTORE uses a plugin-based database backend to store GNS information
-with good performance. Here sqlite and PostgreSQL are supported
-database backends. NAMESTORE clients interact with the IDENTITY
-subsystem to obtain cryptographic information about zones based on egos
-as described with the IDENTITY subsystem, but internally NAMESTORE
-refers to zones using the respective private key.
-
-NAMESTORE is queried and monitored by the ZONEMASTER service which periodically
-publishes public records of GNS zones. ZONEMASTER also
-collaborates with the NAMECACHE subsystem and stores zone information
-when local information are modified in the NAMECACHE cache to increase look-up
-performance for local information and to enable local access to private records
-in zones through GNS.
-
-NAMESTORE provides functionality to look-up and store records, to
-iterate over a specific or all zones and to monitor zones for changes.
-NAMESTORE functionality can be accessed using the NAMESTORE C API, the NAMESTORE
-REST API, or the NAMESTORE command line tool.
-
-.. index::
- double: HOSTLIST; subsystem
-
-.. _HOSTLIST-Subsystem:
-
-HOSTLIST — HELLO bootstrapping and gossip
-=========================================
-
-Peers in the GNUnet overlay network need address information so that
-they can connect with other peers. GNUnet uses so called HELLO messages
-to store and exchange peer addresses. GNUnet provides several methods
-for peers to obtain this information:
-
-- out-of-band exchange of HELLO messages (manually, using for example
- gnunet-core)
-
-- HELLO messages shipped with GNUnet (automatic with distribution)
-
-- UDP neighbor discovery in LAN (IPv4 broadcast, IPv6 multicast)
-
-- topology gossiping (learning from other peers we already connected
- to), and
-
-- the HOSTLIST daemon covered in this section, which is particularly
- relevant for bootstrapping new peers.
-
-New peers have no existing connections (and thus cannot learn from
-gossip among peers), may not have other peers in their LAN and might be
-started with an outdated set of HELLO messages from the distribution. In
-this case, getting new peers to connect to the network requires either
-manual effort or the use of a HOSTLIST to obtain HELLOs.
-
-.. _HELLOs:
-
-HELLOs
-------
-
-The basic information peers require to connect to other peers are
-contained in so called HELLO messages you can think of as a business
-card. Besides the identity of the peer (based on the cryptographic
-public key) a HELLO message may contain address information that
-specifies ways to contact a peer. By obtaining HELLO messages, a peer
-can learn how to contact other peers.
-
-.. _Overview-for-the-HOSTLIST-subsystem:
-
-Overview for the HOSTLIST subsystem
------------------------------------
-
-The HOSTLIST subsystem provides a way to distribute and obtain contact
-information to connect to other peers using a simple HTTP GET request.
-Its implementation is split in three parts, the main file for the
-daemon itself (``gnunet-daemon-hostlist.c``), the HTTP client used to
-download peer information (``hostlist-client.c``) and the server
-component used to provide this information to other peers
-(``hostlist-server.c``). The server is basically a small HTTP web server
-(based on GNU libmicrohttpd) which provides a list of HELLOs known to
-the local peer for download. The client component is basically a HTTP
-client (based on libcurl) which can download hostlists from one or more
-websites. The hostlist format is a binary blob containing a sequence of
-HELLO messages. Note that any HTTP server can theoretically serve a
-hostlist, the built-in hostlist server makes it simply convenient to
-offer this service.
-
-.. _Features:
-
-Features
-^^^^^^^^
-
-The HOSTLIST daemon can:
-
-- provide HELLO messages with validated addresses obtained from
- PEERINFO to download for other peers
-
-- download HELLO messages and forward these message to the TRANSPORT
- subsystem for validation
-
-- advertises the URL of this peer's hostlist address to other peers via
- gossip
-
-- automatically learn about hostlist servers from the gossip of other
- peers
-
-.. _HOSTLIST-_002d-Limitations:
-
-HOSTLIST - Limitations
-^^^^^^^^^^^^^^^^^^^^^^
-
-The HOSTLIST daemon does not:
-
-- verify the cryptographic information in the HELLO messages
-
-- verify the address information in the HELLO messages
-
-.. _Interacting-with-the-HOSTLIST-daemon:
-
-Interacting with the HOSTLIST daemon
-------------------------------------
-
-The HOSTLIST subsystem is currently implemented as a daemon, so there is
-no need for the user to interact with it and therefore there is no
-command line tool and no API to communicate with the daemon. In the
-future, we can envision changing this to allow users to manually trigger
-the download of a hostlist.
-
-Since there is no command line interface to interact with HOSTLIST, the
-only way to interact with the hostlist is to use STATISTICS to obtain or
-modify information about the status of HOSTLIST:
-
-::
-
- $ gnunet-statistics -s hostlist
-
-In particular, HOSTLIST includes a **persistent** value in statistics
-that specifies when the hostlist server might be queried next. As this
-value is exponentially increasing during runtime, developers may want to
-reset or manually adjust it. Note that HOSTLIST (but not STATISTICS)
-needs to be shutdown if changes to this value are to have any effect on
-the daemon (as HOSTLIST does not monitor STATISTICS for changes to the
-download frequency).
-
-.. _Hostlist-security-address-validation:
-
-Hostlist security address validation
-------------------------------------
-
-Since information obtained from other parties cannot be trusted without
-validation, we have to distinguish between *validated* and *not
-validated* addresses. Before using (and so trusting) information from
-other parties, this information has to be double-checked (validated).
-Address validation is not done by HOSTLIST but by the TRANSPORT service.
-
-The HOSTLIST component is functionally located between the PEERINFO and
-the TRANSPORT subsystem. When acting as a server, the daemon obtains
-valid (*validated*) peer information (HELLO messages) from the PEERINFO
-service and provides it to other peers. When acting as a client, it
-contacts the HOSTLIST servers specified in the configuration, downloads
-the (unvalidated) list of HELLO messages and forwards these information
-to the TRANSPORT server to validate the addresses.
-
-.. _The-HOSTLIST-daemon:
-
-:index:`The HOSTLIST daemon <double: daemon; HOSTLIST>`
-The HOSTLIST daemon
--------------------
-
-The hostlist daemon is the main component of the HOSTLIST subsystem. It
-is started by the ARM service and (if configured) starts the HOSTLIST
-client and server components.
-
-GNUNET_MESSAGE_TYPE_HOSTLIST_ADVERTISEMENT
-If the daemon provides a hostlist itself it can advertise it's own
-hostlist to other peers. To do so it sends a
-``GNUNET_MESSAGE_TYPE_HOSTLIST_ADVERTISEMENT`` message to other peers
-when they connect to this peer on the CORE level. This hostlist
-advertisement message contains the URL to access the HOSTLIST HTTP
-server of the sender. The daemon may also subscribe to this type of
-message from CORE service, and then forward these kind of message to the
-HOSTLIST client. The client then uses all available URLs to download
-peer information when necessary.
-
-When starting, the HOSTLIST daemon first connects to the CORE subsystem
-and if hostlist learning is enabled, registers a CORE handler to receive
-this kind of messages. Next it starts (if configured) the client and
-server. It passes pointers to CORE connect and disconnect and receive
-handlers where the client and server store their functions, so the
-daemon can notify them about CORE events.
-
-To clean up on shutdown, the daemon has a cleaning task, shutting down
-all subsystems and disconnecting from CORE.
-
-.. _The-HOSTLIST-server:
-
-:index:`The HOSTLIST server <single: HOSTLIST; server>`
-The HOSTLIST server
--------------------
-
-The server provides a way for other peers to obtain HELLOs. Basically it
-is a small web server other peers can connect to and download a list of
-HELLOs using standard HTTP; it may also advertise the URL of the
-hostlist to other peers connecting on CORE level.
-
-.. _The-HTTP-Server:
-
-The HTTP Server
-^^^^^^^^^^^^^^^
-
-During startup, the server starts a web server listening on the port
-specified with the HTTPPORT value (default 8080). In addition it
-connects to the PEERINFO service to obtain peer information. The
-HOSTLIST server uses the GNUNET_PEERINFO_iterate function to request
-HELLO information for all peers and adds their information to a new
-hostlist if they are suitable (expired addresses and HELLOs without
-addresses are both not suitable) and the maximum size for a hostlist is
-not exceeded (MAX_BYTES_PER_HOSTLISTS = 500000). When PEERINFO finishes
-(with a last NULL callback), the server destroys the previous hostlist
-response available for download on the web server and replaces it with
-the updated hostlist. The hostlist format is basically a sequence of
-HELLO messages (as obtained from PEERINFO) without any special
-tokenization. Since each HELLO message contains a size field, the
-response can easily be split into separate HELLO messages by the client.
-
-A HOSTLIST client connecting to the HOSTLIST server will receive the
-hostlist as an HTTP response and the server will terminate the
-connection with the result code ``HTTP 200 OK``. The connection will be
-closed immediately if no hostlist is available.
-
-.. _Advertising-the-URL:
-
-Advertising the URL
-^^^^^^^^^^^^^^^^^^^
-
-The server also advertises the URL to download the hostlist to other
-peers if hostlist advertisement is enabled. When a new peer connects and
-has hostlist learning enabled, the server sends a
-``GNUNET_MESSAGE_TYPE_HOSTLIST_ADVERTISEMENT`` message to this peer
-using the CORE service.
-
-HOSTLIST client
-.. _The-HOSTLIST-client:
-
-The HOSTLIST client
--------------------
-
-The client provides the functionality to download the list of HELLOs
-from a set of URLs. It performs a standard HTTP request to the URLs
-configured and learned from advertisement messages received from other
-peers. When a HELLO is downloaded, the HOSTLIST client forwards the
-HELLO to the TRANSPORT service for validation.
-
-The client supports two modes of operation:
-
-- download of HELLOs (bootstrapping)
-
-- learning of URLs
-
-.. _Bootstrapping:
-
-Bootstrapping
-^^^^^^^^^^^^^
-
-For bootstrapping, it schedules a task to download the hostlist from the
-set of known URLs. The downloads are only performed if the number of
-current connections is smaller than a minimum number of connections (at
-the moment 4). The interval between downloads increases exponentially;
-however, the exponential growth is limited if it becomes longer than an
-hour. At that point, the frequency growth is capped at (#number of
-connections \* 1h).
-
-Once the decision has been taken to download HELLOs, the daemon chooses
-a random URL from the list of known URLs. URLs can be configured in the
-configuration or be learned from advertisement messages. The client uses
-a HTTP client library (libcurl) to initiate the download using the
-libcurl multi interface. Libcurl passes the data to the
-callback_download function which stores the data in a buffer if space is
-available and the maximum size for a hostlist download is not exceeded
-(MAX_BYTES_PER_HOSTLISTS = 500000). When a full HELLO was downloaded,
-the HOSTLIST client offers this HELLO message to the TRANSPORT service
-for validation. When the download is finished or failed, statistical
-information about the quality of this URL is updated.
-
-.. _Learning:
-
-:index:`Learning <single: HOSTLIST; learning>`
-Learning
-^^^^^^^^
-
-The client also manages hostlist advertisements from other peers. The
-HOSTLIST daemon forwards ``GNUNET_MESSAGE_TYPE_HOSTLIST_ADVERTISEMENT``
-messages to the client subsystem, which extracts the URL from the
-message. Next, a test of the newly obtained URL is performed by
-triggering a download from the new URL. If the URL works correctly, it
-is added to the list of working URLs.
-
-The size of the list of URLs is restricted, so if an additional server
-is added and the list is full, the URL with the worst quality ranking
-(determined through successful downloads and number of HELLOs e.g.) is
-discarded. During shutdown the list of URLs is saved to a file for
-persistence and loaded on startup. URLs from the configuration file are
-never discarded.
-
-.. _Usage:
-
-Usage
------
-
-To start HOSTLIST by default, it has to be added to the DEFAULTSERVICES
-section for the ARM services. This is done in the default configuration.
-
-For more information on how to configure the HOSTLIST subsystem see the
-installation handbook: Configuring the hostlist to bootstrap Configuring
-your peer to provide a hostlist
-
-.. index::
- double: IDENTITY; subsystem
-
-.. _IDENTITY-Subsystem:
-
-IDENTITY — Ego management
-=========================
-
-Identities of \"users\" in GNUnet are called egos. Egos can be used as
-pseudonyms (\"fake names\") or be tied to an organization (for example,
-\"GNU\") or even the actual identity of a human. GNUnet users are
-expected to have many egos. They might have one tied to their real
-identity, some for organizations they manage, and more for different
-domains where they want to operate under a pseudonym.
-
-The IDENTITY service allows users to manage their egos. The identity
-service manages the private keys egos of the local user; it does not
-manage identities of other users (public keys). Public keys for other
-users need names to become manageable. GNUnet uses the GNU Name System
-(GNS) to give names to other users and manage their public keys
-securely. This chapter is about the IDENTITY service, which is about the
-management of private keys.
-
-On the network, an ego corresponds to an ECDSA key (over Curve25519,
-using RFC 6979, as required by GNS). Thus, users can perform actions
-under a particular ego by using (signing with) a particular private key.
-Other users can then confirm that the action was really performed by
-that ego by checking the signature against the respective public key.
-
-The IDENTITY service allows users to associate a human-readable name
-with each ego. This way, users can use names that will remind them of
-the purpose of a particular ego. The IDENTITY service will store the
-respective private keys and allows applications to access key
-information by name. Users can change the name that is locally (!)
-associated with an ego. Egos can also be deleted, which means that the
-private key will be removed and it thus will not be possible to perform
-actions with that ego in the future.
-
-Additionally, the IDENTITY subsystem can associate service functions
-with egos. For example, GNS requires the ego that should be used for the
-shorten zone. GNS will ask IDENTITY for an ego for the \"gns-short\"
-service. The IDENTITY service has a mapping of such service strings to
-the name of the ego that the user wants to use for this service, for
-example \"my-short-zone-ego\".
-
-Finally, the IDENTITY API provides access to a special ego, the
-anonymous ego. The anonymous ego is special in that its private key is
-not really private, but fixed and known to everyone. Thus, anyone can
-perform actions as anonymous. This can be useful as with this trick,
-code does not have to contain a special case to distinguish between
-anonymous and pseudonymous egos.
-
-.. index::
- double: subsystem; MESSENGER
-
-.. _MESSENGER-Subsystem:
-
-MESSENGER — Room-based end-to-end messaging
-===========================================
-
-The MESSENGER subsystem is responsible for secure end-to-end
-communication in groups of nodes in the GNUnet overlay network.
-MESSENGER builds on the CADET subsystem which provides a reliable and
-secure end-to-end communication between the nodes inside of these
-groups.
-
-Additionally to the CADET security benefits, MESSENGER provides
-following properties designed for application level usage:
-
-- MESSENGER provides integrity by signing the messages with the users
- provided ego
-
-- MESSENGER adds (optional) forward secrecy by replacing the key pair
- of the used ego and signing the propagation of the new one with old
- one (chaining egos)
-
-- MESSENGER provides verification of a original sender by checking
- against all used egos from a member which are currently in active use
- (active use depends on the state of a member session)
-
-- MESSENGER offsers (optional) decentralized message forwarding between
- all nodes in a group to improve availability and prevent MITM-attacks
-
-- MESSENGER handles new connections and disconnections from nodes in
- the group by reconnecting them preserving an efficient structure for
- message distribution (ensuring availability and accountablity)
-
-- MESSENGER provides replay protection (messages can be uniquely
- identified via SHA-512, include a timestamp and the hash of the last
- message)
-
-- MESSENGER allows detection for dropped messages by chaining them
- (messages refer to the last message by their hash) improving
- accountability
-
-- MESSENGER allows requesting messages from other peers explicitly to
- ensure availability
-
-- MESSENGER provides confidentiality by padding messages to few
- different sizes (512 bytes, 4096 bytes, 32768 bytes and maximal
- message size from CADET)
-
-- MESSENGER adds (optional) confidentiality with ECDHE to exchange and
- use symmetric encryption, encrypting with both AES-256 and Twofish
- but allowing only selected members to decrypt (using the receivers
- ego for ECDHE)
-
-Also MESSENGER provides multiple features with privacy in mind:
-
-- MESSENGER allows deleting messages from all peers in the group by the
- original sender (uses the MESSENGER provided verification)
-
-- MESSENGER allows using the publicly known anonymous ego instead of
- any unique identifying ego
-
-- MESSENGER allows your node to decide between acting as host of the
- used messaging room (sharing your peer's identity with all nodes in
- the group) or acting as guest (sharing your peer's identity only with
- the nodes you explicitly open a connection to)
-
-- MESSENGER handles members independently of the peer's identity making
- forwarded messages indistinguishable from directly received ones (
- complicating the tracking of messages and identifying its origin)
-
-- MESSENGER allows names of members being not unique (also names are
- optional)
-
-- MESSENGER does not include information about the selected receiver of
- an explicitly encrypted message in its header, complicating it for
- other members to draw conclusions from communication partners
-
-
-
-.. index::
- single: subsystem; Network size estimation
- see: NSE; Network size estimation
-
-.. _NSE-Subsystem:
-
-NSE — Network size estimation
-=============================
-
-NSE stands for Network Size Estimation. The NSE subsystem provides other
-subsystems and users with a rough estimate of the number of peers
-currently participating in the GNUnet overlay. The computed value is not
-a precise number as producing a precise number in a decentralized,
-efficient and secure way is impossible. While NSE's estimate is
-inherently imprecise, NSE also gives the expected range. For a peer that
-has been running in a stable network for a while, the real network size
-will typically (99.7% of the time) be in the range of [2/3 estimate, 3/2
-estimate]. We will now give an overview of the algorithm used to
-calculate the estimate; all of the details can be found in this
-technical report.
-
-.. todo:: link to the report.
-
-.. _Motivation:
-
-Motivation
----------
-Some subsystems, like DHT, need to know the size of the GNUnet network
-to optimize some parameters of their own protocol. The decentralized
-nature of GNUnet makes efficient and securely counting the exact number
-of peers infeasible. Although there are several decentralized algorithms
-to count the number of peers in a system, so far there is none to do so
-securely. Other protocols may allow any malicious peer to manipulate the
-final result or to take advantage of the system to perform Denial of
-Service (DoS) attacks against the network. GNUnet's NSE protocol avoids
-these drawbacks.
-
-NSE security
-.. _Security:
-
-:index:`Security <single: NSE; security>`
-Security
-^^^^^^^^
-
-The NSE subsystem is designed to be resilient against these attacks. It
-uses `proofs of
-work <http://en.wikipedia.org/wiki/Proof-of-work_system>`__ to prevent
-one peer from impersonating a large number of participants, which would
-otherwise allow an adversary to artificially inflate the estimate. The
-DoS protection comes from the time-based nature of the protocol: the
-estimates are calculated periodically and out-of-time traffic is either
-ignored or stored for later retransmission by benign peers. In
-particular, peers cannot trigger global network communication at will.
-
-.. _Principle:
-
-:index:`Principle <single: NSE; principle of operation>`
-Principle
----------
-
-The algorithm calculates the estimate by finding the globally closest
-peer ID to a random, time-based value.
-
-The idea is that the closer the ID is to the random value, the more
-\"densely packed\" the ID space is, and therefore, more peers are in the
-network.
-
-.. _Example:
-
-Example
-^^^^^^^
-
-Suppose all peers have IDs between 0 and 100 (our ID space), and the
-random value is 42. If the closest peer has the ID 70 we can imagine
-that the average \"distance\" between peers is around 30 and therefore
-the are around 3 peers in the whole ID space. On the other hand, if the
-closest peer has the ID 44, we can imagine that the space is rather
-packed with peers, maybe as much as 50 of them. Naturally, we could have
-been rather unlucky, and there is only one peer and happens to have the
-ID 44. Thus, the current estimate is calculated as the average over
-multiple rounds, and not just a single sample.
-
-.. _Algorithm:
-
-Algorithm
-^^^^^^^^^
-
-Given that example, one can imagine that the job of the subsystem is to
-efficiently communicate the ID of the closest peer to the target value
-to all the other peers, who will calculate the estimate from it.
-
-.. _Target-value:
-
-Target value
-^^^^^^^^^^^^
-
-The target value itself is generated by hashing the current time,
-rounded down to an agreed value. If the rounding amount is 1h (default)
-and the time is 12:34:56, the time to hash would be 12:00:00. The
-process is repeated each rounding amount (in this example would be every
-hour). Every repetition is called a round.
-
-.. _Timing:
-
-Timing
-^^^^^^
-
-The NSE subsystem has some timing control to avoid everybody
-broadcasting its ID all at one. Once each peer has the target random
-value, it compares its own ID to the target and calculates the
-hypothetical size of the network if that peer were to be the closest.
-Then it compares the hypothetical size with the estimate from the
-previous rounds. For each value there is an associated point in the
-period, let's call it \"broadcast time\". If its own hypothetical
-estimate is the same as the previous global estimate, its \"broadcast
-time\" will be in the middle of the round. If its bigger it will be
-earlier and if its smaller (the most likely case) it will be later. This
-ensures that the peers closest to the target value start broadcasting
-their ID the first.
-
-.. _Controlled-Flooding:
-
-Controlled Flooding
-^^^^^^^^^^^^^^^^^^^
-
-When a peer receives a value, first it verifies that it is closer than
-the closest value it had so far, otherwise it answers the incoming
-message with a message containing the better value. Then it checks a
-proof of work that must be included in the incoming message, to ensure
-that the other peer's ID is not made up (otherwise a malicious peer
-could claim to have an ID of exactly the target value every round). Once
-validated, it compares the broadcast time of the received value with the
-current time and if it's not too early, sends the received value to its
-neighbors. Otherwise it stores the value until the correct broadcast
-time comes. This prevents unnecessary traffic of sub-optimal values,
-since a better value can come before the broadcast time, rendering the
-previous one obsolete and saving the traffic that would have been used
-to broadcast it to the neighbors.
-
-.. _Calculating-the-estimate:
-
-Calculating the estimate
-^^^^^^^^^^^^^^^^^^^^^^^^
-
-Once the closest ID has been spread across the network each peer gets
-the exact distance between this ID and the target value of the round and
-calculates the estimate with a mathematical formula described in the
-tech report. The estimate generated with this method for a single round
-is not very precise. Remember the case of the example, where the only
-peer is the ID 44 and we happen to generate the target value 42,
-thinking there are 50 peers in the network. Therefore, the NSE subsystem
-remembers the last 64 estimates and calculates an average over them,
-giving a result of which usually has one bit of uncertainty (the real
-size could be half of the estimate or twice as much). Note that the
-actual network size is calculated in powers of two of the raw input,
-thus one bit of uncertainty means a factor of two in the size estimate.
-
-.. index::
- double: subsystem; PEERINFO
-
-.. _PEERINFO-Subsystem:
-
-PEERINFO — Persistent HELLO storage
-===================================
-
-The PEERINFO subsystem is used to store verified (validated) information
-about known peers in a persistent way. It obtains these addresses for
-example from TRANSPORT service which is in charge of address validation.
-Validation means that the information in the HELLO message are checked
-by connecting to the addresses and performing a cryptographic handshake
-to authenticate the peer instance stating to be reachable with these
-addresses. Peerinfo does not validate the HELLO messages itself but only
-stores them and gives them to interested clients.
-
-As future work, we think about moving from storing just HELLO messages
-to providing a generic persistent per-peer information store. More and
-more subsystems tend to need to store per-peer information in persistent
-way. To not duplicate this functionality we plan to provide a PEERSTORE
-service providing this functionality.
-
-.. _PEERINFO-_002d-Features:
-
-PEERINFO - Features
--------------------
-
-- Persistent storage
-
-- Client notification mechanism on update
-
-- Periodic clean up for expired information
-
-- Differentiation between public and friend-only HELLO
-
-.. _PEERINFO-_002d-Limitations:
-
-PEERINFO - Limitations
-----------------------
-
-- Does not perform HELLO validation
-
-.. _DeveloperPeer-Information:
-
-DeveloperPeer Information
--------------------------
-
-The PEERINFO subsystem stores these information in the form of HELLO
-messages you can think of as business cards. These HELLO messages
-contain the public key of a peer and the addresses a peer can be reached
-under. The addresses include an expiration date describing how long they
-are valid. This information is updated regularly by the TRANSPORT
-service by revalidating the address. If an address is expired and not
-renewed, it can be removed from the HELLO message.
-
-Some peer do not want to have their HELLO messages distributed to other
-peers, especially when GNUnet's friend-to-friend modus is enabled. To
-prevent this undesired distribution. PEERINFO distinguishes between
-*public* and *friend-only* HELLO messages. Public HELLO messages can be
-freely distributed to other (possibly unknown) peers (for example using
-the hostlist, gossiping, broadcasting), whereas friend-only HELLO
-messages may not be distributed to other peers. Friend-only HELLO
-messages have an additional flag ``friend_only`` set internally. For
-public HELLO message this flag is not set. PEERINFO does and cannot not
-check if a client is allowed to obtain a specific HELLO type.
-
-The HELLO messages can be managed using the GNUnet HELLO library. Other
-GNUnet systems can obtain these information from PEERINFO and use it for
-their purposes. Clients are for example the HOSTLIST component providing
-these information to other peers in form of a hostlist or the TRANSPORT
-subsystem using these information to maintain connections to other
-peers.
-
-.. _Startup:
-
-Startup
--------
-
-During startup the PEERINFO services loads persistent HELLOs from disk.
-First PEERINFO parses the directory configured in the HOSTS value of the
-``PEERINFO`` configuration section to store PEERINFO information. For
-all files found in this directory valid HELLO messages are extracted. In
-addition it loads HELLO messages shipped with the GNUnet distribution.
-These HELLOs are used to simplify network bootstrapping by providing
-valid peer information with the distribution. The use of these HELLOs
-can be prevented by setting the ``USE_INCLUDED_HELLOS`` in the
-``PEERINFO`` configuration section to ``NO``. Files containing invalid
-information are removed.
-
-.. _Managing-Information:
-
-Managing Information
---------------------
-
-The PEERINFO services stores information about known PEERS and a single
-HELLO message for every peer. A peer does not need to have a HELLO if no
-information are available. HELLO information from different sources, for
-example a HELLO obtained from a remote HOSTLIST and a second HELLO
-stored on disk, are combined and merged into one single HELLO message
-per peer which will be given to clients. During this merge process the
-HELLO is immediately written to disk to ensure persistence.
-
-PEERINFO in addition periodically scans the directory where information
-are stored for empty HELLO messages with expired TRANSPORT addresses.
-This periodic task scans all files in the directory and recreates the
-HELLO messages it finds. Expired TRANSPORT addresses are removed from
-the HELLO and if the HELLO does not contain any valid addresses, it is
-discarded and removed from the disk.
-
-.. _Obtaining-Information:
-
-Obtaining Information
----------------------
-
-When a client requests information from PEERINFO, PEERINFO performs a
-lookup for the respective peer or all peers if desired and transmits
-this information to the client. The client can specify if friend-only
-HELLOs have to be included or not and PEERINFO filters the respective
-HELLO messages before transmitting information.
-
-To notify clients about changes to PEERINFO information, PEERINFO
-maintains a list of clients interested in this notifications. Such a
-notification occurs if a HELLO for a peer was updated (due to a merge
-for example) or a new peer was added.
-
-.. index::
- double: subsystem; PEERSTORE
-
-.. _PEERSTORE-Subsystem:
-
-PEERSTORE — Extensible local persistent data storage
-====================================================
-
-GNUnet's PEERSTORE subsystem offers persistent per-peer storage for
-other GNUnet subsystems. GNUnet subsystems can use PEERSTORE to
-persistently store and retrieve arbitrary data. Each data record stored
-with PEERSTORE contains the following fields:
-
-- subsystem: Name of the subsystem responsible for the record.
-
-- peerid: Identity of the peer this record is related to.
-
-- key: a key string identifying the record.
-
-- value: binary record value.
-
-- expiry: record expiry date.
-
-.. _Functionality:
-
-Functionality
--------------
-
-Subsystems can store any type of value under a (subsystem, peerid, key)
-combination. A \"replace\" flag set during store operations forces the
-PEERSTORE to replace any old values stored under the same (subsystem,
-peerid, key) combination with the new value. Additionally, an expiry
-date is set after which the record is \*possibly\* deleted by PEERSTORE.
-
-Subsystems can iterate over all values stored under any of the following
-combination of fields:
-
-- (subsystem)
-
-- (subsystem, peerid)
-
-- (subsystem, key)
-
-- (subsystem, peerid, key)
-
-Subsystems can also request to be notified about any new values stored
-under a (subsystem, peerid, key) combination by sending a \"watch\"
-request to PEERSTORE.
-
-.. _Architecture:
-
-Architecture
-------------
-
-PEERSTORE implements the following components:
-
-- PEERSTORE service: Handles store, iterate and watch operations.
-
-- PEERSTORE API: API to be used by other subsystems to communicate and
- issue commands to the PEERSTORE service.
-
-- PEERSTORE plugins: Handles the persistent storage. At the moment,
- only an \"sqlite\" plugin is implemented.
-
-.. index::
- double: subsystem; REGEX
-
-.. _REGEX-Subsystem:
-
-REGEX — Service discovery using regular expressions
-===================================================
-
-Using the REGEX subsystem, you can discover peers that offer a
-particular service using regular expressions. The peers that offer a
-service specify it using a regular expressions. Peers that want to
-patronize a service search using a string. The REGEX subsystem will then
-use the DHT to return a set of matching offerers to the patrons.
-
-For the technical details, we have Max's defense talk and Max's Master's
-thesis.
-
-.. note:: An additional publication is under preparation and available
- to team members (in Git).
-
-.. todo:: Missing links to Max's talk and Master's thesis
-
-.. _How-to-run-the-regex-profiler:
-
-How to run the regex profiler
------------------------------
-
-The gnunet-regex-profiler can be used to profile the usage of mesh/regex
-for a given set of regular expressions and strings. Mesh/regex allows
-you to announce your peer ID under a certain regex and search for peers
-matching a particular regex using a string. See
-`szengel2012ms <https://bib.gnunet.org/full/date.html#2012_5f2>`__ for a
-full introduction.
-
-First of all, the regex profiler uses GNUnet testbed, thus all the
-implications for testbed also apply to the regex profiler (for example
-you need password-less ssh login to the machines listed in your hosts
-file).
-
-**Configuration**
-
-Moreover, an appropriate configuration file is needed. In the following
-paragraph the important details are highlighted.
-
-Announcing of the regular expressions is done by the
-gnunet-daemon-regexprofiler, therefore you have to make sure it is
-started, by adding it to the START_ON_DEMAND set of ARM:
-
-::
-
- [regexprofiler]
- START_ON_DEMAND = YES
-
-Furthermore you have to specify the location of the binary:
-
-::
-
- [regexprofiler]
- # Location of the gnunet-daemon-regexprofiler binary.
- BINARY = /home/szengel/gnunet/src/mesh/.libs/gnunet-daemon-regexprofiler
- # Regex prefix that will be applied to all regular expressions and
- # search string.
- REGEX_PREFIX = "GNVPN-0001-PAD"
-
-When running the profiler with a large scale deployment, you probably
-want to reduce the workload of each peer. Use the following options to
-do this.
-
-::
-
- [dht]
- # Force network size estimation
- FORCE_NSE = 1
-
- [dhtcache]
- DATABASE = heap
- # Disable RC-file for Bloom filter? (for benchmarking with limited IO
- # availability)
- DISABLE_BF_RC = YES
- # Disable Bloom filter entirely
- DISABLE_BF = YES
-
- [nse]
- # Minimize proof-of-work CPU consumption by NSE
- WORKBITS = 1
-
-**Options**
-
-To finally run the profiler some options and the input data need to be
-specified on the command line.
-
-::
-
- gnunet-regex-profiler -c config-file -d log-file -n num-links \
- -p path-compression-length -s search-delay -t matching-timeout \
- -a num-search-strings hosts-file policy-dir search-strings-file
-
-Where\...
-
-- \... ``config-file`` means the configuration file created earlier.
-
-- \... ``log-file`` is the file where to write statistics output.
-
-- \... ``num-links`` indicates the number of random links between
- started peers.
-
-- \... ``path-compression-length`` is the maximum path compression
- length in the DFA.
-
-- \... ``search-delay`` time to wait between peers finished linking and
- starting to match strings.
-
-- \... ``matching-timeout`` timeout after which to cancel the
- searching.
-
-- \... ``num-search-strings`` number of strings in the
- search-strings-file.
-
-- \... the ``hosts-file`` should contain a list of hosts for the
- testbed, one per line in the following format:
-
- - ``user@host_ip:port``
-
-- \... the ``policy-dir`` is a folder containing text files containing
- one or more regular expressions. A peer is started for each file in
- that folder and the regular expressions in the corresponding file are
- announced by this peer.
-
-- \... the ``search-strings-file`` is a text file containing search
- strings, one in each line.
-
-You can create regular expressions and search strings for every AS in
-the Internet using the attached scripts. You need one of the `CAIDA
-routeviews
-prefix2as <http://data.caida.org/datasets/routing/routeviews-prefix2as/>`__
-data files for this. Run
-
-::
-
- create_regex.py <filename> <output path>
-
-to create the regular expressions and
-
-::
-
- create_strings.py <input path> <outfile>
-
-to create a search strings file from the previously created regular
-expressions.
-
-
-
-.. index::
- double: subsystem; REST
-
-.. _REST-Subsystem:
-
-REST — RESTful GNUnet Web APIs
-==============================
-
-.. todo:: Define REST
-
-Using the REST subsystem, you can expose REST-based APIs or services.
-The REST service is designed as a pluggable architecture.
-
-**Configuration**
-
-The REST service can be configured in various ways. The reference config
-file can be found in ``src/rest/rest.conf``:
-
-::
-
- [rest]
- REST_PORT=7776
- REST_ALLOW_HEADERS=Authorization,Accept,Content-Type
- REST_ALLOW_ORIGIN=*
- REST_ALLOW_CREDENTIALS=true
-
-The port as well as CORS (cross-origin resource sharing) headers
-that are supposed to be advertised by the rest service are configurable.
-
-.. index::
- double: subsystem; REVOCATION
-
-.. _REVOCATION-Subsystem:
-
-REVOCATION — Ego key revocation
-===============================
-
-The REVOCATION subsystem is responsible for key revocation of Egos. If a
-user learns that their private key has been compromised or has lost it,
-they can use the REVOCATION system to inform all of the other users that
-their private key is no longer valid. The subsystem thus includes ways
-to query for the validity of keys and to propagate revocation messages.
-
-.. _Dissemination:
-
-Dissemination
--------------
-
-When a revocation is performed, the revocation is first of all
-disseminated by flooding the overlay network. The goal is to reach every
-peer, so that when a peer needs to check if a key has been revoked, this
-will be purely a local operation where the peer looks at its local
-revocation list. Flooding the network is also the most robust form of
-key revocation --- an adversary would have to control a separator of the
-overlay graph to restrict the propagation of the revocation message.
-Flooding is also very easy to implement --- peers that receive a
-revocation message for a key that they have never seen before simply
-pass the message to all of their neighbours.
-
-Flooding can only distribute the revocation message to peers that are
-online. In order to notify peers that join the network later, the
-revocation service performs efficient set reconciliation over the sets
-of known revocation messages whenever two peers (that both support
-REVOCATION dissemination) connect. The SET service is used to perform
-this operation efficiently.
-
-.. _Revocation-Message-Design-Requirements:
-
-Revocation Message Design Requirements
---------------------------------------
-
-However, flooding is also quite costly, creating O(\|E\|) messages on a
-network with \|E\| edges. Thus, revocation messages are required to
-contain a proof-of-work, the result of an expensive computation (which,
-however, is cheap to verify). Only peers that have expended the CPU time
-necessary to provide this proof will be able to flood the network with
-the revocation message. This ensures that an attacker cannot simply
-flood the network with millions of revocation messages. The
-proof-of-work required by GNUnet is set to take days on a typical PC to
-compute; if the ability to quickly revoke a key is needed, users have
-the option to pre-compute revocation messages to store off-line and use
-instantly after their key has expired.
-
-Revocation messages must also be signed by the private key that is being
-revoked. Thus, they can only be created while the private key is in the
-possession of the respective user. This is another reason to create a
-revocation message ahead of time and store it in a secure location.
-
-.. index::
- double: subsystems; Random peer sampling
- see: RPS; Random peer sampling
-
-.. _RPS-Subsystem:
-
-RPS — Random peer sampling
-==========================
-
-In literature, Random Peer Sampling (RPS) refers to the problem of
-reliably [1]_ drawing random samples from an unstructured p2p network.
-
-Doing so in a reliable manner is not only hard because of inherent
-problems but also because of possible malicious peers that could try to
-bias the selection.
-
-It is useful for all kind of gossip protocols that require the selection
-of random peers in the whole network like gathering statistics,
-spreading and aggregating information in the network, load balancing and
-overlay topology management.
-
-The approach chosen in the RPS service implementation in GNUnet follows
-the `Brahms <https://bib.gnunet.org/full/date.html\#2009_5f0>`__ design.
-
-The current state is \"work in progress\". There are a lot of things
-that need to be done, primarily finishing the experimental evaluation
-and a re-design of the API.
-
-The abstract idea is to subscribe to connect to/start the RPS service
-and request random peers that will be returned when they represent a
-random selection from the whole network with high probability.
-
-An additional feature to the original Brahms-design is the selection of
-sub-groups: The GNUnet implementation of RPS enables clients to ask for
-random peers from a group that is defined by a common shared secret.
-(The secret could of course also be public, depending on the use-case.)
-
-Another addition to the original protocol was made: The sampler
-mechanism that was introduced in Brahms was slightly adapted and used to
-actually sample the peers and returned to the client. This is necessary
-as the original design only keeps peers connected to random other peers
-in the network. In order to return random peers to client requests
-independently random, they cannot be drawn from the connected peers. The
-adapted sampler makes sure that each request for random peers is
-independent from the others.
-
-.. _Brahms:
-
-Brahms
-------
-
-The high-level concept of Brahms is two-fold: Combining push-pull gossip
-with locally fixing a assumed bias using cryptographic min-wise
-permutations. The central data structure is the view - a peer's current
-local sample. This view is used to select peers to push to and pull
-from. This simple mechanism can be biased easily. For this reason Brahms
-'fixes' the bias by using the so-called sampler. A data structure that
-takes a list of elements as input and outputs a random one of them
-independently of the frequency in the input set. Both an element that
-was put into the sampler a single time and an element that was put into
-it a million times have the same probability of being the output. This
-is achieved with exploiting min-wise independent permutations. In the
-RPS service we use HMACs: On the initialisation of a sampler element, a
-key is chosen at random. On each input the HMAC with the random key is
-computed. The sampler element keeps the element with the minimal HMAC.
-
-In order to fix the bias in the view, a fraction of the elements in the
-view are sampled through the sampler from the random stream of peer IDs.
-
-According to the theoretical analysis of Bortnikov et al. this suffices
-to keep the network connected and having random peers in the view.
-
-.. [1]
- \"Reliable\" in this context means having no bias, neither spatial,
- nor temporal, nor through malicious activity.
-
-.. index::
- double: STATISTICS; subsystem
-
-.. _STATISTICS-Subsystem:
-
-STATISTICS — Runtime statistics publication
-===========================================
-
-In GNUnet, the STATISTICS subsystem offers a central place for all
-subsystems to publish unsigned 64-bit integer run-time statistics.
-Keeping this information centrally means that there is a unified way for
-the user to obtain data on all subsystems, and individual subsystems do
-not have to always include a custom data export method for performance
-metrics and other statistics. For example, the TRANSPORT system uses
-STATISTICS to update information about the number of directly connected
-peers and the bandwidth that has been consumed by the various plugins.
-This information is valuable for diagnosing connectivity and performance
-issues.
-
-Following the GNUnet service architecture, the STATISTICS subsystem is
-divided into an API which is exposed through the header
-**gnunet_statistics_service.h** and the STATISTICS service
-**gnunet-service-statistics**. The **gnunet-statistics** command-line
-tool can be used to obtain (and change) information about the values
-stored by the STATISTICS service. The STATISTICS service does not
-communicate with other peers.
-
-Data is stored in the STATISTICS service in the form of tuples
-**(subsystem, name, value, persistence)**. The subsystem determines to
-which other GNUnet's subsystem the data belongs. name is the name
-through which value is associated. It uniquely identifies the record
-from among other records belonging to the same subsystem. In some parts
-of the code, the pair **(subsystem, name)** is called a **statistic** as
-it identifies the values stored in the STATISTCS service.The persistence
-flag determines if the record has to be preserved across service
-restarts. A record is said to be persistent if this flag is set for it;
-if not, the record is treated as a non-persistent record and it is lost
-after service restart. Persistent records are written to and read from
-the file **statistics.data** before shutdown and upon startup. The file
-is located in the HOME directory of the peer.
-
-An anomaly of the STATISTICS service is that it does not terminate
-immediately upon receiving a shutdown signal if it has any clients
-connected to it. It waits for all the clients that are not monitors to
-close their connections before terminating itself. This is to prevent
-the loss of data during peer shutdown — delaying the STATISTICS
-service shutdown helps other services to store important data to
-STATISTICS during shutdown.
-
-.. index::
- double: TRANSPORT Next Generation; subsystem
-
-.. _TRANSPORT_002dNG-Subsystem:
-
-TRANSPORT-NG — Next-generation transport management
-===================================================
-
-The current GNUnet TRANSPORT architecture is rooted in the GNUnet 0.4
-design of using plugins for the actual transmission operations and the
-ATS subsystem to select a plugin and allocate bandwidth. The following
-key issues have been identified with this design:
-
-- Bugs in one plugin can affect the TRANSPORT service and other
- plugins. There is at least one open bug that affects sockets, where
- the origin is difficult to pinpoint due to the large code base.
-
-- Relevant operating system default configurations often impose a limit
- of 1024 file descriptors per process. Thus, one plugin may impact
- other plugin's connectivity choices.
-
-- Plugins are required to offer bi-directional connectivity. However,
- firewalls (incl. NAT boxes) and physical environments sometimes only
- allow uni-directional connectivity, which then currently cannot be
- utilized at all.
-
-- Distance vector routing was implemented in 209 but shortly afterwards
- broken and due to the complexity of implementing it as a plugin and
- dealing with the resource allocation consequences was never useful.
-
-- Most existing plugins communicate completely using cleartext,
- exposing metad data (message size) and making it easy to fingerprint
- and possibly block GNUnet traffic.
-
-- Various NAT traversal methods are not supported.
-
-- The service logic is cluttered with \"manipulation\" support code for
- TESTBED to enable faking network characteristics like lossy
- connections or firewewalls.
-
-- Bandwidth allocation is done in ATS, requiring the duplication of
- state and resulting in much delayed allocation decisions. As a
- result, often available bandwidth goes unused. Users are expected to
- manually configure bandwidth limits, instead of TRANSPORT using
- congestion control to adapt automatically.
-
-- TRANSPORT is difficult to test and has bad test coverage.
-
-- HELLOs include an absolute expiration time. Nodes with unsynchronized
- clocks cannot connect.
-
-- Displaying the contents of a HELLO requires the respective plugin as
- the plugin-specific data is encoded in binary. This also complicates
- logging.
-
-.. _Design-goals-of-TNG:
-
-Design goals of TNG
--------------------
-
-In order to address the above issues, we want to:
-
-- Move plugins into separate processes which we shall call
- *communicators*. Communicators connect as clients to the transport
- service.
-
-- TRANSPORT should be able to utilize any number of communicators to the
- same peer at the same time.
-
-- TRANSPORT should be responsible for fragmentation, retransmission,
- flow- and congestion-control. Users should no longer have to
- configure bandwidth limits: TRANSPORT should detect what is available
- and use it.
-
-- Communicators should be allowed to be uni-directional and
- unreliable. TRANSPORT shall create bi-directional channels from this
- whenever possible.
-
-- DV should no longer be a plugin, but part of TRANSPORT.
-
-- TRANSPORT should provide communicators help communicating, for
- example in the case of uni-directional communicators or the need for
- out-of-band signalling for NAT traversal. We call this functionality
- *backchannels*.
-
-- Transport manipulation should be signalled to CORE on a per-message
- basis instead of an approximate bandwidth.
-
-- CORE should signal performance requirements (reliability, latency,
- etc.) on a per-message basis to TRANSPORT. If possible, TRANSPORT
- should consider those options when scheduling messages for
- transmission.
-
-- HELLOs should be in a human-readable format with monotonic time
- expirations.
-
-The new architecture is planned as follows:
-
-.. image:: /images/tng.png
-
-TRANSPORT's main objective is to establish bi-directional virtual links
-using a variety of possibly uni-directional communicators. Links undergo
-the following steps:
-
-1. Communicator informs TRANSPORT A that a queue (direct neighbour) is
- available, or equivalently TRANSPORT A discovers a (DV) path to a
- target B.
-
-2. TRANSPORT A sends a challenge to the target peer, trying to confirm
- that the peer can receive. FIXME: This is not implemented properly
- for DV. Here we should really take a validated DVH and send a
- challenge exactly down that path!
-
-3. The other TRANSPORT, TRANSPORT B, receives the challenge, and sends
- back a response, possibly using a dierent path. If TRANSPORT B does
- not yet have a virtual link to A, it must try to establish a virtual
- link.
-
-4. Upon receiving the response, TRANSPORT A creates the virtual link. If
- the response included a challenge, TRANSPORT A must respond to this
- challenge as well, eectively re-creating the TCP 3-way handshake
- (just with longer challenge values).
-
-.. _HELLO_002dNG:
-
-HELLO-NG
---------
-
-HELLOs change in three ways. First of all, communicators encode the
-respective addresses in a human-readable URL-like string. This way, we
-do no longer require the communicator to print the contents of a HELLO.
-Second, HELLOs no longer contain an expiration time, only a creation
-time. The receiver must only compare the respective absolute values. So
-given a HELLO from the same sender with a larger creation time, then the
-old one is no longer valid. This also obsoletes the need for the
-gnunet-hello binary to set HELLO expiration times to never. Third, a
-peer no longer generates one big HELLO that always contains all of the
-addresses. Instead, each address is signed individually and shared only
-over the address scopes where it makes sense to share the address. In
-particular, care should be taken to not share MACs across the Internet
-and confine their use to the LAN. As each address is signed separately,
-having multiple addresses valid at the same time (given the new creation
-time expiration logic) requires that those addresses must have exactly
-the same creation time. Whenever that monotonic time is increased, all
-addresses must be re-signed and re-distributed.
-
-.. _Priorities-and-preferences:
-
-Priorities and preferences
---------------------------
-
-In the new design, TRANSPORT adopts a feature (which was previously
-already available in CORE) of the MQ API to allow applications to
-specify priorities and preferences per message (or rather, per MQ
-envelope). The (updated) MQ API allows applications to specify one of
-four priority levels as well as desired preferences for transmission by
-setting options on an envelope. These preferences currently are:
-
-- GNUNET_MQ_PREF_UNRELIABLE: Disables TRANSPORT waiting for ACKS on
- unreliable channels like UDP. Now it is fire and forget. These
- messages then cannot be used for RTT estimates either.
-
-- GNUNET_MQ_PREF_LOW_LATENCY: Directs TRANSPORT to select the
- lowest-latency transmission choices possible.
-
-- GNUNET_MQ_PREF_CORK_ALLOWED: Allows TRANSPORT to delay transmission
- to group the message with other messages into a larger batch to
- reduce the number of packets sent.
-
-- GNUNET_MQ_PREF_GOODPUT: Directs TRANSPORT to select the highest
- goodput channel available.
-
-- GNUNET_MQ_PREF_OUT_OF_ORDER: Allows TRANSPORT to reorder the messages
- as it sees fit, otherwise TRANSPORT should attempt to preserve
- transmission order.
-
-Each MQ envelope is always able to store those options (and the
-priority), and in the future this uniform API will be used by TRANSPORT,
-CORE, CADET and possibly other subsystems that send messages (like
-LAKE). When CORE sets preferences and priorities, it is supposed to
-respect the preferences and priorities it is given from higher layers.
-Similarly, CADET also simply passes on the preferences and priorities of
-the layer above CADET. When a layer combines multiple smaller messages
-into one larger transmission, the ``GNUNET_MQ_env_combine_options()``
-should be used to calculate options for the combined message. We note
-that the exact semantics of the options may differ by layer. For
-example, CADET will always strictly implement reliable and in-order
-delivery of messages, while the same options are only advisory for
-TRANSPORT and CORE: they should try (using ACKs on unreliable
-communicators, not changing the message order themselves), but if
-messages are lost anyway (e.g. because a TCP is dropped in the middle),
-or if messages are reordered (e.g. because they took different paths
-over the network and arrived in a different order) TRANSPORT and CORE do
-not have to correct this. Whether a preference is strict or loose is
-thus dened by the respective layer.
-
-.. _Communicators:
-
-Communicators
--------------
-
-The API for communicators is defined in
-``gnunet_transport_communication_service.h``. Each communicator must
-specify its (global) communication characteristics, which for now only
-say whether the communication is reliable (e.g. TCP, HTTPS) or
-unreliable (e.g. UDP, WLAN). Each communicator must specify a unique
-address prex, or NULL if the communicator cannot establish outgoing
-connections (for example because it is only acting as a TCP server). A
-communicator must tell TRANSPORT which addresses it is reachable under.
-Addresses may be added or removed at any time. A communicator may have
-zero addresses (transmission only). Addresses do not have to match the
-address prefix.
-
-TRANSPORT may ask a communicator to try to connect to another address.
-TRANSPORT will only ask for connections where the address matches the
-communicator's address prefix that was provided when the connection was
-established. Communicators should then attempt to establish a
-connection.
-It is under the discretion of the communicator whether to honor this request.
-Reasons for not honoring such a request may be that an existing connection exists
-or resource limitations.
-No response is provided to TRANSPORT service on failure.
-The TRANSPORT service has to ask the communicator explicitly to retry.
-
-If a communicator succeeds in establishing an outgoing connection for
-transmission, or if a communicator receives an incoming bi-directional
-connection, the communicator must inform the TRANSPORT service that a
-message queue (MQ) for transmission is now available.
-For that MQ, the communicator must provide the peer identity claimed by the other end.
-It must also provide a human-readable address (for debugging) and a maximum transfer unit
-(MTU). A MTU of zero means sending is not supported, SIZE_MAX should be
-used for no MTU. The communicator should also tell TRANSPORT what
-network type is used for the queue. The communicator may tell TRANSPORT
-anytime that the queue was deleted and is no longer available.
-
-The communicator API also provides for flow control. First,
-communicators exhibit back-pressure on TRANSPORT: the number of messages
-TRANSPORT may add to a queue for transmission will be limited. So by not
-draining the transmission queue, back-pressure is provided to TRANSPORT.
-In the other direction, communicators may allow TRANSPORT to give
-back-pressure towards the communicator by providing a non-NULL
-``GNUNET_TRANSPORT_MessageCompletedCallback`` argument to the
-``GNUNET_TRANSPORT_communicator_receive`` function. In this case,
-TRANSPORT will only invoke this function once it has processed the
-message and is ready to receive more. Communicators should then limit
-how much traffic they receive based on this backpressure. Note that
-communicators do not have to provide a
-``GNUNET_TRANSPORT_MessageCompletedCallback``; for example, UDP cannot
-support back-pressure due to the nature of the UDP protocol. In this
-case, TRANSPORT will implement its own TRANSPORT-to-TRANSPORT flow
-control to reduce the sender's data rate to acceptable levels.
-
-TRANSPORT may notify a communicator about backchannel messages TRANSPORT
-received from other peers for this communicator. Similarly,
-communicators can ask TRANSPORT to try to send a backchannel message to
-other communicators of other peers. The semantics of the backchannel
-message are up to the communicators which use them. TRANSPORT may fail
-transmitting backchannel messages, and TRANSPORT will not attempt to
-retransmit them.
-
-UDP communicator
-^^^^^^^^^^^^^^^^
-
-The UDP communicator implements a basic encryption layer to protect from
-metadata leakage.
-The layer tries to establish a shared secret using an Elliptic-Curve Diffie-Hellman
-key exchange in which the initiator of a packet creates an ephemeral key pair
-to encrypt a message for the target peer identity.
-The communicator always offers this kind of transmission queue to a (reachable)
-peer in which messages are encrypted with dedicated keys.
-The performance of this queue is not suitable for high volume data transfer.
-
-If the UDP connection is bi-directional, or the TRANSPORT is able to offer a
-backchannel connection, the resulting key can be re-used if the recieving peer
-is able to ACK the reception.
-This will cause the communicator to offer a new queue (with a higher priority
-than the default queue) to TRANSPORT with a limited capacity.
-The capacity is increased whenever the communicator receives an ACK for a
-transmission.
-This queue is suitable for high-volume data transfer and TRANSPORT will likely
-prioritize this queue (if available).
-
-Communicators that try to establish a connection to a target peer authenticate
-their peer ID (public key) in the first packets by signing a monotonic time
-stamp, its peer ID, and the target peerID and send this data as well as the signature
-in one of the first packets.
-Receivers should keep track (persist) of the monotonic time stamps for each
-peer ID to reject possible replay attacks.
-
-FIXME: Handshake wire format? KX, Flow.
-
-TCP communicator
-^^^^^^^^^^^^^^^^
-
-FIXME: Handshake wire format? KX, Flow.
-
-QUIC communicator
-^^^^^^^^^^^^^^^^^
-The QUIC communicator runs over a bi-directional UDP connection.
-TLS layer with self-signed certificates (binding/signed with peer ID?).
-Single, bi-directional stream?
-FIXME: Handshake wire format? KX, Flow.
-
-.. index::
- double: TRANSPORT; subsystem
-
-.. _TRANSPORT-Subsystem:
-
-TRANSPORT — Overlay transport management
-========================================
-
-This chapter documents how the GNUnet transport subsystem works. The
-GNUnet transport subsystem consists of three main components: the
-transport API (the interface used by the rest of the system to access
-the transport service), the transport service itself (most of the
-interesting functions, such as choosing transports, happens here) and
-the transport plugins. A transport plugin is a concrete implementation
-for how two GNUnet peers communicate; many plugins exist, for example
-for communication via TCP, UDP, HTTP, HTTPS and others. Finally, the
-transport subsystem uses supporting code, especially the NAT/UPnP
-library to help with tasks such as NAT traversal.
-
-Key tasks of the transport service include:
-
-- Create our HELLO message, notify clients and neighbours if our HELLO
- changes (using NAT library as necessary)
-
-- Validate HELLOs from other peers (send PING), allow other peers to
- validate our HELLO's addresses (send PONG)
-
-- Upon request, establish connections to other peers (using address
- selection from ATS subsystem) and maintain them (again using PINGs
- and PONGs) as long as desired
-
-- Accept incoming connections, give ATS service the opportunity to
- switch communication channels
-
-- Notify clients about peers that have connected to us or that have
- been disconnected from us
-
-- If a (stateful) connection goes down unexpectedly (without explicit
- DISCONNECT), quickly attempt to recover (without notifying clients)
- but do notify clients quickly if reconnecting fails
-
-- Send (payload) messages arriving from clients to other peers via
- transport plugins and receive messages from other peers, forwarding
- those to clients
-
-- Enforce inbound traffic limits (using flow-control if it is
- applicable); outbound traffic limits are enforced by CORE, not by us
- (!)
-
-- Enforce restrictions on P2P connection as specified by the blacklist
- configuration and blacklisting clients
-
-Note that the term \"clients\" in the list above really refers to the
-GNUnet-CORE service, as CORE is typically the only client of the
-transport service.
-
-.. index::
- double: subsystem; SET
-
-.. _SET-Subsystem:
-
-SET — Peer to peer set operations (Deprecated)
-==============================================
-
-.. note::
-
- The SET subsystem is in process of being replaced by the SETU and SETI
- subsystems, which provide basically the same functionality, just using
- two different subsystems. SETI and SETU should be used for new code.
-
-The SET service implements efficient set operations between two peers
-over a CADET tunnel. Currently, set union and set intersection are the
-only supported operations. Elements of a set consist of an *element
-type* and arbitrary binary *data*. The size of an element's data is
-limited to around 62 KB.
-
-.. _Local-Sets:
-
-Local Sets
-----------
-
-Sets created by a local client can be modified and reused for multiple
-operations. As each set operation requires potentially expensive special
-auxiliary data to be computed for each element of a set, a set can only
-participate in one type of set operation (either union or intersection).
-The type of a set is determined upon its creation. If a the elements of
-a set are needed for an operation of a different type, all of the set's
-element must be copied to a new set of appropriate type.
-
-.. _Set-Modifications:
-
-Set Modifications
------------------
-
-Even when set operations are active, one can add to and remove elements
-from a set. However, these changes will only be visible to operations
-that have been created after the changes have taken place. That is,
-every set operation only sees a snapshot of the set from the time the
-operation was started. This mechanism is *not* implemented by copying
-the whole set, but by attaching *generation information* to each element
-and operation.
-
-.. _Set-Operations:
-
-Set Operations
---------------
-
-Set operations can be started in two ways: Either by accepting an
-operation request from a remote peer, or by requesting a set operation
-from a remote peer. Set operations are uniquely identified by the
-involved *peers*, an *application id* and the *operation type*.
-
-The client is notified of incoming set operations by *set listeners*. A
-set listener listens for incoming operations of a specific operation
-type and application id. Once notified of an incoming set request, the
-client can accept the set request (providing a local set for the
-operation) or reject it.
-
-.. _Result-Elements:
-
-Result Elements
----------------
-
-The SET service has three *result modes* that determine how an
-operation's result set is delivered to the client:
-
-- **Full Result Set.** All elements of set resulting from the set
- operation are returned to the client.
-
-- **Added Elements.** Only elements that result from the operation and
- are not already in the local peer's set are returned. Note that for
- some operations (like set intersection) this result mode will never
- return any elements. This can be useful if only the remove peer is
- actually interested in the result of the set operation.
-
-- **Removed Elements.** Only elements that are in the local peer's
- initial set but not in the operation's result set are returned. Note
- that for some operations (like set union) this result mode will never
- return any elements. This can be useful if only the remove peer is
- actually interested in the result of the set operation.
-
-.. index::
- double: subsystem; SETI
-
-.. _SETI-Subsystem:
-
-SETI — Peer to peer set intersections
-=====================================
-
-The SETI service implements efficient set intersection between two peers
-over a CADET tunnel. Elements of a set consist of an *element type* and
-arbitrary binary *data*. The size of an element's data is limited to
-around 62 KB.
-
-.. _Intersection-Sets:
-
-Intersection Sets
------------------
-
-Sets created by a local client can be modified (by adding additional
-elements) and reused for multiple operations. If elements are to be
-removed, a fresh set must be created by the client.
-
-.. _Set-Intersection-Modifications:
-
-Set Intersection Modifications
-------------------------------
-
-Even when set operations are active, one can add elements to a set.
-However, these changes will only be visible to operations that have been
-created after the changes have taken place. That is, every set operation
-only sees a snapshot of the set from the time the operation was started.
-This mechanism is *not* implemented by copying the whole set, but by
-attaching *generation information* to each element and operation.
-
-.. _Set-Intersection-Operations:
-
-Set Intersection Operations
----------------------------
-
-Set operations can be started in two ways: Either by accepting an
-operation request from a remote peer, or by requesting a set operation
-from a remote peer. Set operations are uniquely identified by the
-involved *peers*, an *application id* and the *operation type*.
-
-The client is notified of incoming set operations by *set listeners*. A
-set listener listens for incoming operations of a specific operation
-type and application id. Once notified of an incoming set request, the
-client can accept the set request (providing a local set for the
-operation) or reject it.
-
-.. _Intersection-Result-Elements:
-
-Intersection Result Elements
-----------------------------
-
-The SET service has two *result modes* that determine how an operation's
-result set is delivered to the client:
-
-- **Return intersection.** All elements of set resulting from the set
- intersection are returned to the client.
-
-- **Removed Elements.** Only elements that are in the local peer's
- initial set but not in the intersection are returned.
-
-
-
-
-.. index::
- double: SETU; subsystem
-
-.. _SETU-Subsystem:
-
-SETU — Peer to peer set unions
-==============================
-
-The SETU service implements efficient set union operations between two
-peers over a CADET tunnel. Elements of a set consist of an *element
-type* and arbitrary binary *data*. The size of an element's data is
-limited to around 62 KB.
-
-.. _Union-Sets:
-
-Union Sets
-----------
-
-Sets created by a local client can be modified (by adding additional
-elements) and reused for multiple operations. If elements are to be
-removed, a fresh set must be created by the client.
-
-.. _Set-Union-Modifications:
-
-Set Union Modifications
------------------------
-
-Even when set operations are active, one can add elements to a set.
-However, these changes will only be visible to operations that have been
-created after the changes have taken place. That is, every set operation
-only sees a snapshot of the set from the time the operation was started.
-This mechanism is *not* implemented by copying the whole set, but by
-attaching *generation information* to each element and operation.
-
-.. _Set-Union-Operations:
-
-Set Union Operations
---------------------
-
-Set operations can be started in two ways: Either by accepting an
-operation request from a remote peer, or by requesting a set operation
-from a remote peer. Set operations are uniquely identified by the
-involved *peers*, an *application id* and the *operation type*.
-
-The client is notified of incoming set operations by *set listeners*. A
-set listener listens for incoming operations of a specific operation
-type and application id. Once notified of an incoming set request, the
-client can accept the set request (providing a local set for the
-operation) or reject it.
-
-.. _Union-Result-Elements:
-
-Union Result Elements
----------------------
-
-The SET service has three *result modes* that determine how an
-operation's result set is delivered to the client:
-
-- **Locally added Elements.** Elements that are in the union but not
- already in the local peer's set are returned.
-
-- **Remote added Elements.** Additionally, notify the client if the
- remote peer lacked some elements and thus also return to the local
- client those elements that we are sending to the remote peer to be
- added to its union. Obtaining these elements requires setting the
- ``GNUNET_SETU_OPTION_SYMMETRIC`` option.
+The description of GNUnet's subsystems has moved to the developer
+handbook: see :ref:`subsystems`.
diff --git a/users/vpn.rst b/users/vpn.rst
@@ -3,6 +3,11 @@
Virtual Public Network
----------------------
+.. note::
+
+ The VPN is under development; expect bugs and incompatible changes. See
+ :doc:`/status`.
+
Using the GNUnet Virtual Public Network (VPN) application you can tunnel
IP traffic over GNUnet. Moreover, the VPN comes with built-in protocol
translation and DNS-ALG support, enabling IPv4-to-IPv6 protocol