BUFQ.md (6408B)
1 <!-- 2 Copyright (C) Daniel Stenberg, <daniel@haxx.se>, et al. 3 4 SPDX-License-Identifier: curl 5 --> 6 7 # bufq 8 9 This is an internal module for managing I/O buffers. A `bufq` can be written 10 to and read from. It manages read and write positions and has a maximum size. 11 12 ## read/write 13 14 Its basic read/write functions have a similar signature and return code 15 handling as many internal curl read and write ones. 16 17 18 ``` 19 ssize_t Curl_bufq_write(struct bufq *q, const unsigned char *buf, size_t len, CURLcode *err); 20 21 - returns the length written into `q` or -1 on error. 22 - writing to a full `q` returns -1 and set *err to CURLE_AGAIN 23 24 ssize_t Curl_bufq_read(struct bufq *q, unsigned char *buf, size_t len, CURLcode *err); 25 26 - returns the length read from `q` or -1 on error. 27 - reading from an empty `q` returns -1 and set *err to CURLE_AGAIN 28 29 ``` 30 31 To pass data into a `bufq` without an extra copy, read callbacks can be used. 32 33 ``` 34 typedef ssize_t Curl_bufq_reader(void *reader_ctx, unsigned char *buf, size_t len, 35 CURLcode *err); 36 37 ssize_t Curl_bufq_slurp(struct bufq *q, Curl_bufq_reader *reader, void *reader_ctx, 38 CURLcode *err); 39 ``` 40 41 `Curl_bufq_slurp()` invokes the given `reader` callback, passing it its own 42 internal buffer memory to write to. It may invoke the `reader` several times, 43 as long as it has space and while the `reader` always returns the length that 44 was requested. There are variations of `slurp` that call the `reader` at most 45 once or only read in a maximum amount of bytes. 46 47 The analog mechanism for write out buffer data is: 48 49 ``` 50 typedef ssize_t Curl_bufq_writer(void *writer_ctx, const unsigned char *buf, size_t len, 51 CURLcode *err); 52 53 ssize_t Curl_bufq_pass(struct bufq *q, Curl_bufq_writer *writer, void *writer_ctx, 54 CURLcode *err); 55 ``` 56 57 `Curl_bufq_pass()` invokes the `writer`, passing its internal memory and 58 remove the amount that `writer` reports. 59 60 ## peek and skip 61 62 It is possible to get access to the memory of data stored in a `bufq` with: 63 64 ``` 65 bool Curl_bufq_peek(const struct bufq *q, const unsigned char **pbuf, size_t *plen); 66 ``` 67 68 On returning TRUE, `pbuf` points to internal memory with `plen` bytes that one 69 may read. This is only valid until another operation on `bufq` is performed. 70 71 Instead of reading `bufq` data, one may simply skip it: 72 73 ``` 74 void Curl_bufq_skip(struct bufq *q, size_t amount); 75 ``` 76 77 This removes `amount` number of bytes from the `bufq`. 78 79 ## lifetime 80 81 `bufq` is initialized and freed similar to the `dynbuf` module. Code using 82 `bufq` holds a `struct bufq` somewhere. Before it uses it, it invokes: 83 84 ``` 85 void Curl_bufq_init(struct bufq *q, size_t chunk_size, size_t max_chunks); 86 ``` 87 88 The `bufq` is told how many "chunks" of data it shall hold at maximum and how 89 large those "chunks" should be. There are some variants of this, allowing for 90 more options. How "chunks" are handled in a `bufq` is presented in the section 91 about memory management. 92 93 The user of the `bufq` has the responsibility to call: 94 95 ``` 96 void Curl_bufq_free(struct bufq *q); 97 ``` 98 to free all resources held by `q`. It is possible to reset a `bufq` to empty via: 99 100 ``` 101 void Curl_bufq_reset(struct bufq *q); 102 ``` 103 104 ## memory management 105 106 Internally, a `bufq` uses allocation of fixed size, e.g. the "chunk_size", up 107 to a maximum number, e.g. "max_chunks". These chunks are allocated on demand, 108 therefore writing to a `bufq` may return `CURLE_OUT_OF_MEMORY`. Once the max 109 number of chunks are used, the `bufq` reports that it is "full". 110 111 Each chunks has a `read` and `write` index. A `bufq` keeps its chunks in a 112 list. Reading happens always at the head chunk, writing always goes to the 113 tail chunk. When the head chunk becomes empty, it is removed. When the tail 114 chunk becomes full, another chunk is added to the end of the list, becoming 115 the new tail. 116 117 Chunks that are no longer used are returned to a `spare` list by default. If 118 the `bufq` is created with option `BUFQ_OPT_NO_SPARES` those chunks are freed 119 right away. 120 121 If a `bufq` is created with a `bufc_pool`, the no longer used chunks are 122 returned to the pool. Also `bufq` asks the pool for a chunk when it needs one. 123 More in section "pools". 124 125 ## empty, full and overflow 126 127 One can ask about the state of a `bufq` with methods such as 128 `Curl_bufq_is_empty(q)`, `Curl_bufq_is_full(q)`, etc. The amount of data held 129 by a `bufq` is the sum of the data in all its chunks. This is what is reported 130 by `Curl_bufq_len(q)`. 131 132 Note that a `bufq` length and it being "full" are only loosely related. A 133 simple example: 134 135 * create a `bufq` with chunk_size=1000 and max_chunks=4. 136 * write 4000 bytes to it, it reports "full" 137 * read 1 bytes from it, it still reports "full" 138 * read 999 more bytes from it, and it is no longer "full" 139 140 The reason for this is that full really means: *bufq uses max_chunks and the 141 last one cannot be written to*. 142 143 When you read 1 byte from the head chunk in the example above, the head still 144 hold 999 unread bytes. Only when those are also read, can the head chunk be 145 removed and a new tail be added. 146 147 There is another variation to this. If you initialized a `bufq` with option 148 `BUFQ_OPT_SOFT_LIMIT`, it allows writes **beyond** the `max_chunks`. It 149 reports **full**, but one can **still** write. This option is necessary, if 150 partial writes need to be avoided. It means that you need other checks to keep 151 the `bufq` from growing ever larger and larger. 152 153 154 ## pools 155 156 A `struct bufc_pool` may be used to create chunks for a `bufq` and keep spare 157 ones around. It is initialized and used via: 158 159 ``` 160 void Curl_bufcp_init(struct bufc_pool *pool, size_t chunk_size, size_t spare_max); 161 162 void Curl_bufq_initp(struct bufq *q, struct bufc_pool *pool, size_t max_chunks, int opts); 163 ``` 164 165 The pool gets the size and the mount of spares to keep. The `bufq` gets the 166 pool and the `max_chunks`. It no longer needs to know the chunk sizes, as 167 those are managed by the pool. 168 169 A pool can be shared between many `bufq`s, as long as all of them operate in 170 the same thread. In curl that would be true for all transfers using the same 171 multi handle. The advantages of a pool are: 172 173 * when all `bufq`s are empty, only memory for `max_spare` chunks in the pool 174 is used. Empty `bufq`s holds no memory. 175 * the latest spare chunk is the first to be handed out again, no matter which 176 `bufq` needs it. This keeps the footprint of "recently used" memory smaller.