qr-camera-ignores-the-plane-strides.md (14550B)
1 <!-- 2 This file is part of GNU Taler 3 (C) 2026 Taler Systems S.A. 4 5 GNU Taler is free software; you can redistribute it and/or modify it under the 6 terms of the GNU General Public License as published by the Free Software 7 Foundation; either version 3, or (at your option) any later version. 8 9 GNU Taler is distributed in the hope that it will be useful, but WITHOUT ANY 10 WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR 11 A PARTICULAR PURPOSE. See the GNU General Public License for more details. 12 13 You should have received a copy of the GNU General Public License along with 14 GNU Taler; see the file COPYING. If not, see <http://www.gnu.org/licenses/> 15 --> 16 17 # Scanning a QR code reads the camera frame wrong, and one bad frame ends scanning 18 19 Investigated after a report that scanning works on Android 15 but not on 20 Android 14. The camera cannot be driven from an emulator — the virtual scene's 21 camera pose cannot be aimed at a code — so the pipeline was taken apart 22 instead: a camera frame is a luminance plane plus two numbers saying how it is 23 laid out, which can be built on a workstation and put through the app's own 24 arithmetic. Everything below is reproducible with a JDK and the zxing jar the 25 app already depends on; no device, no emulator, no camera. 26 27 **Severity: high.** Two independent defects, each of which stops scanning 28 outright rather than degrading it. Neither produces an error message: the 29 viewfinder keeps running, the user waves the phone at a valid code, and the 30 wallet never reacts. 31 32 ## Defect 1 — the frame's layout is ignored, so nothing decodes 33 34 `QrCameraAnalyzer` converts each frame by hand and never looks at how it is 35 laid out: 36 37 ```kotlin 38 // taler-kotlin-android/src/main/java/net/taler/lib/android/qr/QrCameraAnalyzer.kt:150 39 private fun yuv420888ToNv21(image: Image): ByteArray { 40 val yPlane = image.planes[0].buffer 41 … 42 val ySize = yPlane.remaining() 43 yPlane.get(nv21, 0, ySize) // one bulk copy 44 … 45 } 46 ``` 47 48 ```console 49 $ grep -c 'rowStride\|pixelStride' QrCameraAnalyzer.kt 50 0 51 ``` 52 53 A `YUV_420_888` plane is not a tightly packed image. `rowStride` is the 54 distance between the starts of two rows and the hardware is free to pad, so 55 `rowStride >= width`; `pixelStride` is the distance between two samples. 56 (`ImageFormat.YUV_420_888` documents the Y plane's pixel stride as always one, 57 so row padding is the part that varies in practice — the patch handles both 58 because handling both costs nothing.) A single `get()` is correct only when the 59 plane happens to be tight. With padded rows every row after the first lands 60 shifted by the padding, the image shears a little further on each line, the 61 finder patterns stop being squares, and zxing returns `NotFoundException` for 62 every frame — which the analyzer swallows, so a systematically broken decoder 63 looks exactly like an empty viewfinder. 64 65 Run against the current code, `repro_qr_plane_strides.java`: 66 67 ``` 68 == 1. control: tightly packed plane, rotation 0 69 app pipeline, rowStride == width decoded 70 71 == 2. the same frame with padded rows (rowStride > width) 72 app pipeline, rowStride = 656 (pad 16) FAIL: NotFoundException 73 patched extractLuma, same frame decoded 74 zxing dataWidth = rowStride, no copy at all decoded 75 app pipeline, rowStride = 672 (pad 32) FAIL: NotFoundException 76 … (same for 64, 128) 77 78 == 3. interleaved luma (pixelStride 2) 79 app pipeline FAIL: NotFoundException 80 patched extractLuma decoded 81 ``` 82 83 Not "degraded": no padding value decodes a single frame, and the same frame 84 decodes the moment the stride is honoured. The code is in the frame, unchanged; 85 only the arithmetic differs. 86 87 ## Defect 2 — a frame that throws stops the camera for good 88 89 `proxy.close()` sits in the `finally` of the *decode* `try`, which is entered 90 after the conversion and rotation have already run: 91 92 ```kotlin 93 val nv21 = yuv420888ToNv21(mediaImage) // throws here … 94 val (rotatedNv21, …) = rotateNV21(nv21, …) // … or here 95 … 96 try { 97 onQrDetected(qrReader.decodeWithState(bitmap).text) 98 } catch (_: NotFoundException) { 99 } finally { 100 proxy.close() // … and this never runs 101 } 102 ``` 103 104 The analysis use case is built with `STRATEGY_KEEP_ONLY_LATEST`, under which 105 CameraX holds one frame and acquires the next only once the current proxy is 106 closed. So an exception anywhere above that `try` does not cost one frame: it 107 ends frame delivery for the rest of the screen's life, while the preview — a 108 separate use case — carries on looking perfectly healthy. 109 110 That is not hypothetical, because the two defects meet. `rotateNV21` sizes its 111 output from `width * height` while the array it was handed was sized from the 112 *padded* `remaining()`, and its 180° branch copies the chroma tail out to 113 `data.size`: 114 115 ``` 116 == 4. the app's rotation on a padded plane 117 rotation 90 degrees, rowStride = 672 FAIL: NotFoundException 118 rotation 180 degrees, rowStride = 672 THREW ArrayIndexOutOfBoundsException: Index 460800 out of bounds for length 460800 119 rotation 270 degrees, rowStride = 672 FAIL: NotFoundException 120 ``` 121 122 `rotationDegrees == 180` is reached when the device is held in the orientation 123 opposite to the sensor's, and nothing in the wallet constrains that: 124 125 ```console 126 $ grep -rn screenOrientation wallet/src/main/AndroidManifest.xml 127 $ # no output -- the scan screen rotates freely 128 ``` 129 130 So on a device whose camera pads its rows, turning the phone around during a 131 scan kills scanning until the screen is left and re-entered — with no error, 132 no log line, and a live preview. 133 134 ## Defect 3 — the analysis runs on the UI thread 135 136 ```kotlin 137 useCase.setAnalyzer(mainExecutor) { proxy -> … } 138 ``` 139 140 Every frame costs a copy of the luminance plane plus a zxing decode, and both 141 happen on the main thread, at frame rate. It is not why scanning fails, but it 142 is why the scan screen janks, and it makes a slow decode cost dropped frames. 143 144 ## What is *not* wrong 145 146 Worth recording, because each of these is a plausible place to go looking: 147 148 * **The rotation arithmetic is correct.** On a tightly packed plane all three 149 branches produce a decodable image (section 6 of the reproducer). The 270° 150 branch's odd-looking indexing is in the chroma half, which nothing reads. 151 * **The rotation is not needed at all.** zxing locates a QR code by its three 152 finder patterns and derives the orientation from them: 153 154 ``` 155 == 5. is the rotation needed at all? (tight plane, kept as it arrives) 156 frame arrives rotated 0, decoded as-is decoded 157 frame arrives rotated 90, decoded as-is decoded 158 frame arrives rotated 180, decoded as-is decoded 159 frame arrives rotated 270, decoded as-is decoded 160 ``` 161 162 So `rotateNV21` buys nothing, and it is where the crash lives. Deleting it is 163 the fix, not repairing it. 164 * **Asking zxing to rotate instead would be worse.** The obvious replacement, 165 `PlanarYUVLuminanceSource.rotateCounterClockwise()`, is not implemented: 166 zxing 3.5.3 *and* 3.5.4 inherit the base class, where 167 `isRotateSupported()` returns false and the method throws 168 `UnsupportedOperationException`. The merchant terminal's copy of this 169 pipeline does exactly that — see 170 `pos-config-scanner-asks-for-a-rotation-zxing-refuses.md`. 171 * **The decoder and the hints are fine**, and so is the gallery path. Importing 172 a photo of a code decodes 10 of 10 on API 34 and on API 36, with identical 173 screens; that route goes through `decodeQrFromBitmap`, which builds an 174 `RGBLuminanceSource` from `Bitmap.getPixels` and so never touches any of the 175 arithmetic above. A working gallery import is therefore not evidence that the 176 camera path works — it is the control that isolates it. 177 178 ## Why this can differ between Android versions 179 180 The layout is the camera's choice, not the app's: it comes from the HAL, 181 together with the resolution CameraX selects from the device's advertised 182 stream configurations. Both can change when a device is upgraded — the same 183 hardware with a new vendor camera stack may pad where it did not before, or 184 offer a different analysis resolution — and both differ between devices. 185 "Works on 15, not on 14" is precisely the shape of a stride assumption, which 186 is why the report points here. It is *not* proof: the two numbers that decide 187 it can be printed in one log line, which is the first manual test below. 188 189 ## Reproduce 190 191 Without a device, against an unpatched checkout: 192 193 ```bash 194 ZX=$(find ~/.gradle ~/.m2 -name 'core-3.5.*.jar' | head -1) 195 javac -cp "$ZX" -d /tmp/qrrepro simulator/findings/repro_qr_plane_strides.java 196 java -cp "$ZX:/tmp/qrrepro" repro_qr_plane_strides 197 ``` 198 199 With the patch applied, the same ground is covered by unit tests that stay in 200 the tree: 201 202 ```bash 203 ./gradlew :taler-kotlin-android:testDebugUnitTest \ 204 --tests 'net.taler.lib.android.qr.*' 205 ``` 206 207 `ExtractLumaTest` (4 cases) pins the layout arithmetic; 208 `QrFrameDecodeTest` (5 cases) encodes a real QR code, lays it out tight, 209 padded by 16/32/64/128 bytes and interleaved, and asserts it decodes — plus 210 one case asserting that the *old* bulk copy does not decode, and one that the 211 code reads at all four orientations. All nine pass with the patch; the four 212 `ExtractLumaTest` cases were written first and three of them failed against the 213 original arithmetic. 214 215 By hand on an affected device: open **Scan QR code**, point it at any valid 216 `taler://` code, and nothing happens — no error, no reaction, and `adb logcat` 217 shows nothing, because the per-frame `NotFoundException` is caught and dropped. 218 219 ## Fix 220 221 `qr-camera-ignores-the-plane-strides.diff` does all of it in one file: 222 223 * `extractLuma(buffer, rowStride, pixelStride, width, height)` — a pure 224 function, so it is testable without an emulator — copies the luminance plane 225 row by row, skips the padding, steps by `pixelStride`, keeps the single 226 `get()` fast path for the tight case, and drops the chroma copy that zxing 227 never read. 228 * `rotateNV21` is deleted (see above). 229 * The whole frame body is wrapped in `try { … } catch (e: Exception) { Log.w } 230 finally { proxy.close() }`, so no frame can stop the camera. 231 * Analysis moves to a single-thread executor, with `onQrDetected` posted back 232 to the main thread. 233 234 Verified: `:taler-kotlin-android:compileDebugKotlin`, `:taler-kotlin-android:lintDebug` 235 (0 errors) and the nine unit tests above all pass. Not verified on a real 236 camera — that is what the manual tests are for. 237 238 ### Reference implementations, none of them Google Play 239 240 The arithmetic above is the minimal change. Two alternatives are worth knowing, 241 because both move the problem out of the app entirely, and neither needs Play 242 Services (no ML Kit anywhere): 243 244 1. **Let zxing take the stride.** `PlanarYUVLuminanceSource`'s constructor 245 already separates the buffer's layout from the image: 246 `(yuvData, dataWidth, dataHeight, left, top, width, height, reverseHoriz)`. 247 Passing `rowStride` as `dataWidth` and `width`/`height` as the crop decodes 248 a padded frame with no stride arithmetic in the app at all — the 249 "no copy at all" line in section 2 of the reproducer. This is what the 250 merchant terminal's patch uses, being a four-line change there. 251 2. **Let CameraX do the conversion.** `camera-core` 1.5.3 (already a 252 dependency) carries, verified by `javap` on the resolved artifact: 253 `ImageAnalysis.Builder.setOutputImageFormat(OUTPUT_IMAGE_FORMAT_NV21)`, 254 `setOutputImageRotationEnabled(boolean)`, and internally 255 `ImageProcessingUtil.rotateYUVAndConvertToNV21` over a libyuv 256 `nativeRotateYUV`. That delivers a tightly packed, already-rotated NV21 257 buffer from native code, which is exactly what `PlanarYUVLuminanceSource` 258 wants, and deletes both hand-written functions. CameraX's own stride-aware 259 conversion, `androidx.camera.core.internal.utils.ImageUtil.yuv_420_888toNv21`, 260 is worth reading as the reference arithmetic but is a restricted API — read 261 it, do not call it. 262 `OUTPUT_IMAGE_FORMAT_RGBA_8888` plus `ImageProxy.toBitmap()` would go 263 further and let the camera and gallery paths share the one 264 `decodeQrFromBitmap` decoder, at the cost of an RGBA conversion per frame 265 where zxing only ever reads luminance. 266 267 Worth doing alongside, in any variant: count consecutive failed decodes and log 268 one debug line when the count crosses a threshold. Both defects above were 269 silent, and that single counter would have named either of them in a minute. 270 271 ## Attended tests the developer should run 272 273 These need a human holding a phone; each one is chosen to separate a cause 274 rather than to confirm a guess. 275 276 1. **Print the layout, once per session.** `rowStride`, `pixelStride`, 277 `width`, `height` of `planes[0]`, plus `rotationDegrees` and the resolution 278 CameraX chose, on every device and OS version in the report. This is the 279 decisive test: if `rowStride == width` on Android 15 and `rowStride > width` 280 on Android 14 for the same hardware, defect 1 is the report, and nothing 281 else needs proving. Ideally on the same device before and after the upgrade. 282 2. **Scan in all four orientations**, portrait, both landscapes and upside 283 down, on a device whose rows are padded. Reverse orientation is where 284 `rotationDegrees == 180`, the branch that throws. 285 3. **After a failed scan, check whether frames still arrive** — a temporary 286 per-frame counter in the analyzer is enough. "No frames" means defect 2; 287 "frames but no decode" means defect 1. Then leave the screen and come back: 288 if scanning works again until the next failure, that is defect 2 confirmed. 289 4. **Import a photo of the same code from the gallery.** It goes through a 290 different decoder. Gallery decodes, camera does not ⇒ the fault is in the 291 frame handling, not in the code, the decoder or the lighting. 292 5. **Vary the code's size in the frame and the light**: filling the frame, 293 then small and at arm's length, then dim. A layout fault fails all of them 294 identically; focus and resolution problems do not. 295 6. **Scan a non-Taler QR code** (any URL). It exercises the same pipeline 296 without the app's URI handling, so a decode that the wallet ignores 297 separates "could not read the code" from "read it and did nothing with it".