runtests.md (10366B)
1 --- 2 c: Copyright (C) Daniel Stenberg, <daniel@haxx.se>, et al. 3 SPDX-License-Identifier: curl 4 Title: runtests.pl 5 Section: 1 6 Source: runtests 7 See-also: 8 - runtests.pl 9 Added-in: 7.5 10 --- 11 12 # NAME 13 14 runtests.pl - run one or more test cases 15 16 # SYNOPSIS 17 18 **runtests.pl [options] [tests]** 19 20 # DESCRIPTION 21 22 *runtests.pl* runs one, several or all the existing test cases in curl's 23 test suite. It is often called from the root Makefile of the curl package with 24 'make test'. 25 26 # TESTS 27 28 Specify which test(s) to run by specifying test numbers or keywords. 29 30 If no test number or keyword is given, all existing tests that the script can 31 find are considered for running. You can specify single test cases to run by 32 specifying test numbers space-separated, like `1 3 5 7 11`, and you can 33 specify a range of tests like `45 to 67`. 34 35 Specify tests to not run with a leading exclamation point, like `!66`, which 36 runs all available tests except number 66. 37 38 Prefix a test number with a tilde (~) to still run it, but ignore the results. 39 40 It is also possible to specify tests based on a keyword describing the test(s) 41 to run, like `FTPS`. The keywords are strings used in the individual tests. 42 43 Features are included as keywords with the `feat:` prefix (e.g., `feat:debug`). 44 Specify a feature to run only tests requiring it, or exclude tests using 45 `!feat:<feature>`, like `!feat:proxy`, to disable tests which depend on that 46 feature. 47 48 You can also specify keywords with a leading exclamation point and the keyword 49 or phrase, like "!HTTP NTLM auth" to run all tests **except** those using this 50 keyword. Remember that the exclamation marks and spaces need to be quoted 51 somehow when entered at many command shells. 52 53 Prefix a keyword with a tilde (~) to still run it, but ignore the results. 54 55 # OUTPUT 56 57 When running without `-s` (short output), for instance when running 58 runtests.pl directly rather than via make, each test emits a pair of lines 59 like this: 60 61 Test 0045...[simple HTTP Location: without protocol in initial URL] 62 --pd---e-v- OK (45 out of 1427, remaining: 16:08, took 6.188s, duration: 00:31) 63 64 the first line contains the test number and a description. On the second line, 65 the characters at the beginning are flags indicating which aspects of curl's 66 behavior were checked by the test: 67 68 s stdout 69 r stderr 70 p protocol 71 d data 72 u upload 73 P proxy 74 o output 75 e exit code 76 m memory 77 v valgrind 78 E the test was run event-based 79 80 The remainder of the second line contains the test result, current test sequence, 81 total number of tests to be run and an estimated amount of time to complete the 82 test run. 83 84 # OPTIONS 85 86 ## `-a` 87 88 Continue running the rest of the test cases even if one test fails. By 89 default, the test script stops as soon as an error is detected. 90 91 ## `-ac \<curl\>` 92 93 Provide a path to a curl binary to talk to APIs (currently only CI test APIs). 94 95 ## `-am` 96 97 Display test results in automake style output (`PASS/FAIL: [number] [name]`). 98 99 ## `-c \<curl\>` 100 101 Provide a path to a custom curl binary to run the tests with. Default is the 102 curl executable in the build tree. 103 104 ## `-d` 105 106 Enable protocol debug: have the servers display protocol output. If used in 107 conjunction with parallel testing, it is difficult to associate the logs with 108 the specific test being run. 109 110 ## `-E \<exclude_file\>` 111 112 Load the **exclude_file** with additional reasons why certain tests should be 113 skipped. Useful when testing with external HTTP proxies in which case some of 114 the tests are not appropriate. 115 116 The file contains colon-delimited lines. The first field contains the type of 117 exclusion, the second field contains a pattern and the final field contains 118 the reason why matching tests should be skipped. The exclusion types are 119 *keyword*, *test*, and *tool*. 120 121 ## `-e` or `--test-event` 122 123 Run the test event-based (if possible). This makes runtests invoke curl with 124 --test-event option. This option only works if both curl and libcurl were 125 built debug-enabled. 126 127 ## `-f` 128 129 Force the test to run even if mentioned in DISABLED. 130 131 ## `-g` 132 133 Run the given test(s) with gdb. This is best used on a single test case and 134 curl built --disable-shared. This then fires up gdb with command line set to 135 run the specified test case. Simply (set a break-point and) type 'run' to 136 start. 137 138 ## `-gl` 139 140 Run the given test(s) with lldb. This is best used on a single test case and 141 curl built --disable-shared. This then fires up lldb with command line set to 142 run the specified test case. Simply (set a break-point and) type 'run' to 143 start. 144 145 ## `-gw` 146 147 Run the given test(s) with gdb as a windowed application. 148 149 ## `-h, --help` 150 151 Displays a help text about this program's command line options. 152 153 ## `-j[num]` 154 155 Spawn the given number of processes to run tests in. This defaults to 0 to run 156 tests serially within a single process. Using a number greater than one allows 157 multiple tests to run in parallel, speeding up a test run. The optimum number 158 is dependent on the system and set of tests to run, but 7 times the number of 159 CPU cores is a good figure to start with, or 1.3 times if Valgrind is in use, 160 or 5 times for torture tests. Enabling parallel tests is not recommended in 161 conjunction with the -g option. 162 163 ## `-k` 164 165 Keep output and log files in log/ after a test run, even if no error was 166 detected. Useful for debugging. 167 168 ## `-L \<file\>` 169 170 Load and execute the specified file which should contain perl code. This 171 option allows one to change *runtests.pl* behavior by overwriting functions 172 and variables and is useful when testing external proxies using curl's 173 regression test suite. 174 175 ## `-l` 176 177 Lists all test case names. 178 179 ## `-n` 180 181 Disable the check for and use of valgrind. 182 183 ## `--no-debuginfod` 184 185 Delete the `DEBUGINFOD_URLS` variable if that is defined. Makes valgrind, gdb 186 etc not able to use this functionality. 187 188 ## `-o \<variablename=value\>` 189 190 Overwrite the specified internal **variable** with **value**. Useful to change 191 variables that did not get a dedicated flag to change them. Check the source to 192 see which variables are available. 193 194 ## `-P \<proxy\>` 195 196 Use the specified HTTP proxy when executing tests, even if the tests 197 themselves do not specify a proxy. This option allows one to test external 198 proxies using curl's regression test suite. 199 200 ## `-p` 201 202 Prints out all files in the log directory to stdout when a test case fails. 203 Practical when used in the automated and distributed tests since then the 204 people checking the failures and the reasons for them might not have physical 205 access to the machine and logs. 206 207 ## `-R` 208 209 Run the tests in a scrambled, or randomized, order instead of sequentially. 210 211 The random seed initially set for this is fixed per month and can be set with 212 *--seed*. 213 214 ## `-r` 215 216 Display run time statistics. (Requires the `Perl Time::HiRes` module) 217 218 ## `-rf` 219 220 Display full run time statistics. (Requires the `Perl Time::HiRes` module) 221 222 ## `--repeat=[num]` 223 224 This repeats the given set of test numbers this many times. If no test numbers 225 are given, it repeats ALL tests this many times. It adds the new repeated 226 sequence at the end of the initially given one. 227 228 If **-R** option is also used, the scrambling is done after the repeats have 229 extended the test sequence. 230 231 ## `--retry=[num]` 232 233 Number of attempts for the whole test run to retry failed tests. 234 235 ## `-s` 236 237 Shorter output. Speaks less than default. 238 239 ## `--seed=[num]` 240 241 When using *--shallow* or *-R* that randomize certain aspects of the behavior, 242 this option can set the initial seed. If not set, the random seed is set based 243 on the currently set local year and month and the first line of the "curl -V" 244 output. 245 246 ## `--shallow=[num]` 247 248 Used together with **-t**. This limits the number of tests to fail in torture 249 mode to no more than **num** per test case. If this reduces the amount, the 250 script randomly discards entries to fail until the amount is **num**. 251 252 The random seed initially set for this is fixed per month and can be set with 253 *--seed*. 254 255 ## `-t[num]` 256 257 Selects a **torture** test for the given tests. This makes runtests.pl first 258 run the tests once and count the number of memory allocations made. It then 259 reruns the test that number of times, each time forcing one of the allocations 260 to fail until all allocations have been tested. By setting *num* you can force 261 the allocation with that number to be set to fail at once instead of looping 262 through everyone, which is handy when debugging and then often in combination 263 with *-g*. 264 265 ## `--test-duphandle` 266 267 Passes the `--test-duphandle` option to curl when invoked. This command line 268 option only exists in debug builds and runs curl normally, but duplicates the 269 easy handle before the transfer and use the duplicate instead of the original 270 handle. This verifies that the duplicate works exactly as good as the original 271 handle. 272 273 Because of how the curl tool uses a share object to store and keep some data, 274 not everything is however perfectly copied in the duplicate. In particular 275 HSTS data is not. A specific test case can be set to avoid using 276 `--test-duphandle` by disabling it on a per test basis. 277 278 ## `-u` 279 280 Error instead of warning on server unexpectedly alive. 281 282 ## `-v` 283 284 Enable verbose output. Speaks more than by default. If used in conjunction 285 with parallel testing, it is difficult to associate the logs with the specific 286 test being run. 287 288 ## `-vc \<curl\>` 289 290 Provide a path to a custom curl binary to run when verifying that the servers 291 running are indeed our test servers. Default is the curl executable in the 292 build tree. 293 294 # RUNNING TESTS 295 296 Many tests have conditions that must be met before the test case can run fine. 297 They could depend on built-in features in libcurl or features present in the 298 operating system or even in third-party libraries that curl may or may not 299 use. 300 301 The test script checks most of these by itself to determine when it is safe to 302 attempt to run each test. Those which cannot be run due to failed requirements 303 are simply skipped and listed at the completion of all test cases. In some 304 unusual configurations, the test script cannot make the correct determination 305 for all tests. In these cases, the problematic tests can be skipped using the 306 "!keyword" skip feature documented earlier. 307 308 # WRITING TESTS 309 310 The simplest way to write test cases is to start with a similar existing test, 311 save it with a new number and then adjust it to fit. There is an attempt to 312 document the test case file format in **tests/FILEFORMAT.md**.