testrunner.md (5272B)
1 # Test runner design 2 3 ## Goals 4 5 The unit and package test workflow has three guarantees: 6 7 1. A successful repository run is easy to scan: it shows package progress and 8 one summary instead of a TAP entry for every passing test. 9 2. All packages are attempted even when an earlier package fails, and the 10 final process status reports whether any package failed. 11 3. A test command runs only artifacts produced from the current source tree. 12 Deleted or renamed test sources must not survive as stale JavaScript and run 13 again. 14 15 These guarantees apply to the Node-based unit and package tests in this 16 repository. The integration test runner described in the root README is a 17 separate workflow. 18 19 ## Commands 20 21 The normal repository command is: 22 23 ```shell 24 make check 25 ``` 26 27 It installs the locked dependencies, cleans generated workspace files, builds 28 the workspace, and invokes the root package test orchestrator. The explicit 29 workspace clean is a defense in depth for all generated files, not the only 30 protection against stale tests. 31 32 After a successful workspace build, rerun only the root test phase with: 33 34 ```shell 35 pnpm check 36 ``` 37 38 or run one package directly: 39 40 ```shell 41 pnpm --filter @gnu-taler/taler-util test 42 ``` 43 44 The root summary prints the direct package rerun command for every failure. 45 46 ## Why packages remain separate 47 48 The root command does not put every JavaScript file into one Node test runner 49 process. Packages use different compilation and bundling configurations, have 50 different dependency graphs, and sometimes need package-local setup. Combining 51 all files would weaken those package boundaries and make the root runner aware 52 of implementation details that belong in each package. 53 54 Instead, `packages/qa-tooling/bin/test-all.mjs` discovers workspace packages 55 with a `test` script, orders them after their tested workspace dependencies, 56 and executes those scripts sequentially. It does not stop at the first failed 57 package. A custom Node reporter suppresses successful per-test TAP output, 58 prints failure details immediately, and records counts that the orchestrator 59 combines into one final table. 60 61 This is one repository-level execution and one consistent report, while each 62 package still owns its compiler, test glob, environment, and exit status. 63 Packages whose `test` script only checks compilation appear as `compile only` 64 because they do not produce Node test events. 65 66 The concise reporter is supplied to child Node processes through 67 `NODE_OPTIONS`. A caller-provided `--test-reporter` option is rejected because 68 two reporters cannot reliably produce the repository summary. Run a package 69 directly when another Node reporter is needed for debugging. 70 71 ## Generated test output 72 73 Every package test command must be hermetic with respect to its generated test 74 files. Before compiling or bundling tests, it removes the complete directory 75 that it owns and then recreates that directory from the current sources. 76 77 The ownership conventions are: 78 79 - TypeScript package tests that emit to `lib/` run `test:clean` first. That 80 removes `lib/` and the package TypeScript build-info file before invoking the 81 compiler. If a package's normal build produces additional module formats, 82 the test command recreates those formats before returning so dependents do 83 not see a partial package build. Coverage commands that compile tests follow 84 the same cleanup rule. 85 - Web UI test builds emit beneath `dist/test/`, never `dist/prod/`. The shared 86 `@gnu-taler/web-util/build` helper empties the configured test destination 87 before every build and refuses to remove the package root or a path outside 88 it. 89 - A package with a different generated-test directory must give that directory 90 a single clear owner and empty it at the start of every `test` command. 91 - Source files and hand-maintained fixtures must never be stored in a directory 92 owned by generated test output. 93 94 Cleaning must happen inside the package test command, rather than relying only 95 on `make check`, because developers and CI jobs often rerun an individual 96 package. The root clean remains useful for catching unrelated stale build 97 artifacts before the production build. 98 99 The QA tooling tests enforce the common TypeScript cleanup prefix and reject UI 100 test builders that target `dist/prod`. The shared UI cleanup helper also has 101 tests for deletion and path-safety behavior. 102 103 ## Adding or changing tests 104 105 When adding a package test command: 106 107 1. Put the complete build-and-run workflow in the package's `test` script. 108 2. Choose a generated directory owned only by that workflow and empty it before 109 compiling. Prefer `lib/` for the existing TypeScript emitter pattern and 110 `dist/test/` for UI bundles. 111 3. Use Node's test runner for the produced test files so the repository 112 reporter can collect consistent results. 113 4. Make the package command return nonzero for compilation failures, test 114 failures, and setup failures. 115 5. Verify both a direct package run and `pnpm check`. To exercise deletion, 116 rename or remove a test source and confirm its old generated file is absent 117 after the next direct package run. 118 119 Keep informational output from successful tests small. Failure diagnostics can 120 be detailed; they are preserved by the concise reporter and followed by the 121 repository summary.