path: root/doc/onboarding.texi

\input texinfo @c -*-texinfo-*-
@c %**start of header
@setfilename onboarding.info
@include version.texi
@settitle Notes for taler.net admins and developers @value{VERSION}

@c Define a new index for options.
@defcodeindex op
@c Combine everything into one index (arbitrarily chosen to be the
@c concept index).
@syncodeindex op cp
@c %**end of header

@copying
Howtos for taler.net admins and developers (version @value{VERSION}, @value{UPDATED}),
Copyright @copyright{} 2017 INRIA

@quotation
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.3 or
any later version published by the Free Software Foundation; with no
Invariant Sections, with no Front-Cover Texts, and with no Back-Cover
Texts.  A copy of the license is included in the section entitled
``GNU Free Documentation License''.
@end quotation
@end copying
@c If your manual is published on paper by the FSF, it should include
@c The standard FSF Front-Cover and Back-Cover Texts, as given in
@c maintain.texi.
@c
@c Titlepage
@c
@titlepage
@title Notes for taler.net admins and developers
@subtitle Version @value{VERSION}
@subtitle @value{UPDATED}
@author Marcello Stanisci (@email{marcello@@taler.net})
@page
@vskip 0pt plus 1filll
@insertcopying
@end titlepage

@c @summarycontents
@contents

@ifnottex
@node Top
@top Notes for taler.net admins and developers
@insertcopying
@end ifnottex

@menu
* demo/test deployment::                Deploy Taler for "demo" and "test" setups.
* Standalone deployment::               Deploy Taler in your homepage @ taler.net
* Testing components::                  How to make and run tests
* Releases::                            Releases patterns
* Code::                                Where to find Taler code
* Bugtracking::                         How to track Taler bugs
* Continuous integration::              How CI is currently performed
* Code coverage::                       Where to find coverage reports
@end menu

@node demo/test deployment
@chapter demo/test deployment

The following steps are valid if run as one of the following users:

@itemize
@item demo-blue
@item demo-green
@item test-blue
@item test-green
@end itemize

@section Compile and switch color.

If the setup is already bootstrapped, then it should only be needed
to login as 'demo-X' (with X being the inactive color); and then:

@example
$ source activate
$ taler-deployment-build
@end example

and then switch the color by logging in as the @emph{demo} user,
and switch the color with the following command:

@example
$ taler-deployment-switch-demo-X
@end example

@section Full bootstrap.

In order to bootstrap a Taler installation under a empty home
directory, do:

@example
$ cd $HOME 
$ git clone git://git.taler.net/deployment
@end example

Then run the bootstrap script that
will download all the repositories.

@example
$ ./deployment/bootstrap-bluegreen demo X # with X == 'green' / 'blue'
@end example

If successful, then activate the new environment with:

@example
source activate
@end example

Compile and install all the components.
@example
$ taler-deployment-build
@end example

Create the global configuration file.

@example
$ taler-deployment-config-generate
@end example

Create the folder to share data between the two colors.

@example
$ taler-deployment-hier
@end example

Create all the keys.

@example
$ taler-deployment-keyup
@end example

Sign the @code{/wire} response for the exchange.

@example
$ taler-deployment-sign
@end example

@cartouche
@quotation Note
If the DB schema of merchant or exchange changed, at this point
it MIGHT be necessary to reset all the tables.  To this regard,
consider running one of the following commands:

@example
# To reset the merchant DB.
$ taler-merchant-dbinit -r

# To reset the exchange DB.
$ taler-exchange-dbinit -r
@end example

@end quotation
@end cartouche

If all the steps succeeded, then it should be possible
to launch all the services.  Give:

@example
$ taler-deployment-start
@end example

Verify that all services are up and running:
@example
$ taler-deployment-arm -I
$ tail logs/<component>-<date>.log
@end example

Switch now to the color just built:

@example
$ taler-deployment-switch-<demo/test>-<blue/green>
@end example

@section How to upgrade the code.

Some repositories, especially the ones from the released components,
have a @emph{stable} branch, that keeps older and more stable code.
Therefore,  upon each release we must rebase those stable branches
on the master.

The following commands do that:

@example
$ cd $REPO

$ git pull origin master stable
$ git checkout stable

# option a: resolve conflicts resulting from hotfixes
$ git rebase master
$ ...

# option b: force stable to master
$ git update-ref refs/heads/stable master

$ git push # possibly with --force

# continue development
$ git checkout master
@end example

@node Standalone deployment
@chapter Standalone deployment

This installation assumes that all the steps are run with @code{$HOME}
as @code{$CWD}.

NOTE: this documents does not explain how to get all the dependencies
installed, but just how to use the helper scripts hosted in this repository.
Any component comes with a "doc" directory, or a README file where building
instructions are reported.  As a last resort, those instructions are also
available at https://docs.taler.net.

The first step is to fetch the @cite{deployment} repository, which hosts all
the needed scripts.

@example
# Adapt the repository's URL to your needs.
$ git clone git://git.taler.net/deployment
@end example

The next step is to fetch all the codebases from all the components.

@example
$ ./deployment/bootstrap-standalone
@end example

If the previous step succeeded, a file named @code{activate} should be now
in the @code{$CWD}.  It contains environmental definitions for @code{$PATH} and
database names.

@cartouche
@quotation Note
Please @emph{ignore} the output from the previous script when it succeeds,
which is

@quotation

@smallexample
WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

/usr/lib/postgresql/9.5/bin/pg_ctl -D talerdb -l logfile start
@end smallexample

The reason is that this message is generated by Postgresql's utilities and
you never need to start your database manually; it will be started by the
init script that launches all the Taler processes.
@end quotation
@end quotation
@end cartouche

Now we need to compile and install all the downloaded codebases.

@example
# We first update `@w{`}$PATH`@w{`}, in order to make all the compilation
# and configuration utilities available.
$ source activate

# Double check if the previous step worked: $PATH should
# contain $HOME/local/bin.
$ echo $PATH

# Then we need to install GNUnet, as it provides the 'ARM'
# utility that is used to start the database service.  The
# database service is needed to let the compilation run its
# tests.

$ cd deployment/taler-build/
$ make gnunet-stamp

# Now we can start the database
$ taler-deployment-arm -s
$ taler-deployment-arm -i taler-postgres-standalone

# If the previous commands succeeded, then we can install all the remaining
# components and run checks for them.  Issue:
$ taler-deployment-build
@end example

Now make the configuration file @code{$@{HOME@}/.config/taler.conf}:
@example
$ taler-deployment-config-generate
@end example

The next step is to generate @cite{signkeys} and @cite{denomkeys}.
Note that it will also get the denomkeys signed by the (local mock)
auditor.

@example
$ taler-deployment-keyup
@end example

The following one will place signatures inside wireformat JSON files.

@example
$ taler-deployment-config-sign
@end example

@c An error of "invalid currency name" might be related to the current
@c policy of 12-chars limit for currency names; which is likely going to
@c be changed.

It may be necessary to define database tables for the exchange.  The
following command does that.  Note that you have to manually start the
database, with the following command.

@example
taler-deployment-arm -s
taler-deployment-arm -i taler-postrges-standalone
@end example

@example
# Note that this command _also_ erases any previous
# data in the database.

$ taler-exchange-dbinit -r
$ taler-merchant-dbinit -r
@end example

If all previous steps succeeded, it is now possible to launch all
the processes:

@example
$ taler-deployment-start
@end example

All the services should be reachable at the following URL:
@code{https://env.taler.net/<username>/<service>[/endopoint[?arg0=x&arg1=y]]}

@node Testing components
@chapter Testing components

@c CMDs
@c Traits
@c Twister setup

This chapter is a VERY ABSTRACT description of how testing
is implemented in Taler, and in NO WAY wants to substitute
the reading of the actual source code by the user.

In Taler, a test case is a array of @code{struct TALER_TESTING_Command},
informally referred to as @code{CMD}, that is iteratively executed by
the testing interpreter.  This latter is transparently initiated
by the testing library.

However, the developer does not have to defined CMDs manually,
but rather call the proper constructor provided by the library.
For example, if a CMD is supposed to test feature @code{x},
then the library would provide the @code{TALER_TESTING_cmd_x ()}
constructor for it.  Obviously, each constructor has its own
particular arguments that make sense to test @code{x}, and all
constructor are thoroughly commented within the source code.

Internally, each CMD has two methods: @code{run ()} and @code{cleanup ()}.
The former contains the main logic to test feature @code{x},
whereas the latter cleans the memory up after execution.

In a test life, each CMD needs some internal state, made by
values it keeps in memory. Often, the test has to @emph{share}
those values with other CMDs: for example, CMD1 may create some
key material and CMD2 needs this key material to encrypt data.

The offering of internal values from CMD1 to CMD2 is made
by @emph{traits}.  A trait is a @code{struct TALER_TESTING_Trait},
and each CMD contains a array of traits, that it offers
via the public trait interface to other commands.  The
definition and filling of such array happens transparently
to the test developer.

For example, the following example shows how CMD2 takes
an amount object offered by CMD1 via the trait interface.

Note: the main interpreter and the most part of CMDs and traits
are hosted inside the exchange codebase, but nothing prevents
the developer from implementing new CMDs and traits within
other codebases.

@example
/* Withouth loss of generality, let's consider the
 * following logic to exist inside the run() method of CMD1 */
..

struct TALER_Amount *a;
/**
 * the second argument (0) points to the first amount object offered,
 * in case multiple are available.
 */
if (GNUNET_OK != TALER_TESTING_get_trait_amount_obj (cmd2, 0, &a))
  return GNUNET_SYSERR;
...

use(a); /* 'a' points straight into the internal state of CMD2 */
@end example

In the Taler realm, there is also the possibility to alter the
behaviour of supposedly well-behaved components.  This is needed
when, for example, we want the exchange to return some corrupted
signature in order to check if the merchant backend detects it.

This alteration is accomplished by another service called @emph{twister}.
The twister acts as a proxy between service A and B, and can be
programmed to tamper with the data exchanged by A and B.

Please refer to the Twister codebase (under the @code{test} directory)
in order to see how to configure it.

@node Releases
@chapter Releases

@section Release Process and Checklists

This document describes the process for releasing a new version of the various
Taler components to the official GNU mirrors.

The following components are published on the GNU mirrors

@itemize
@item taler-exchange (exchange.git)
@item taler-merchant (merchant.git)
@item talerdonations (donations.git)
@item talerblog (blog.git)
@item taler-bank (bank.git)
@item taler-wallet-webex (wallet-webex.git)
@end itemize

@section Tagging

Tag releases with an @b{annotated} commit, like

@example
git tag -a v0.1.0 -m "Official release v0.1.0"
git push origin v0.1.0
@end example

@section Database for tests

For tests in the exchange and merchant to run, make sure that
a database @emph{talercheck} is accessible by @emph{$USER}.  Otherwise tests
involving the database logic are skipped.

@section Exchange, merchant

Set the version in @code{configure.ac}.  The commit being tagged
should be the change of the version.

For the exchange test cases to pass, @code{make install} must be run first.
Without it, test cases will fail because plugins can't be located.

@example
./bootstrap
./configure # add required options for your system
make dist
tar -xf taler-$COMPONENT-$VERSION.tar.gz
cd taler-$COMPONENT-$VERSION
make install check
@end example

@section Wallet WebExtension

The version of the wallet is in @emph{manifest.json}.  The @code{version_name} should be
adjusted, and @emph{version} should be increased independently on every upload to
the WebStore.

@example
./configure
make dist
@end example

@c FIXME: selenium test cases

@section Upload to GNU mirrors

See @emph{https://www.gnu.org/prep/maintain/maintain.html#Automated-FTP-Uploads}

Directive file:

@example
version: 1.2
directory: taler
filename: taler-exchange-0.1.0.tar.gz
@end example

Upload the files in @b{binary mode} to the ftp servers.

@node Code
@chapter Code
Taler code is versioned via Git.  For those users without write access,
all the codebases are found at the following URL:

@example
git://git.taler.net/<repository>
@end example

A complete list of all the existing repositories is currently found at
@code{https://git.taler.net/}.  Note: @code{<repository>} must NOT have
the @code{.git} extension.

@node Bugtracking
@chapter Bugtracking
Bug tracking is done with Mantis (https://www.mantisbt.org/).  All the
bugs are then showed and managed at @code{https://bugs.gnunet.org/}, under
the "Taler" project.  A registration on the Web site is needed in order to
use the bug tracker.

@node Continuous integration
@chapter Continuous integration
CI is done with Buildbot (https://buildbot.net/), and builds are triggered
by the means of Git hooks.  The results are published at @code{https://buildbot.wild.gv.taler.net/}.

In order to avoid downtimes, CI uses a "blue/green" deployment technique.
In detail, there are two users building code on the system, the "green" and
the "blue" user; and at any given time, one is running Taler services and
the other one is either building the code or waiting for that.

There is also the possibility to trigger builds manually, but this is only
reserved to "admin" users.

@node Code coverage
@chapter Code coverage
Code coverage is done with the Gcov / Lcov (http://ltp.sourceforge.net/coverage/lcov.php)
combo, and it is run *nightly* (once a day) by a Buildbot worker.  The
coverage results are then published at @code{https://lcov.taler.net/}.

@bye