taler-android

Android apps for GNU Taler (wallet, PoS, cashier)
Log | Files | Refs | README | LICENSE

qr-camera-ignores-the-plane-strides.md (14550B)


      1 <!--
      2   This file is part of GNU Taler
      3   (C) 2026 Taler Systems S.A.
      4 
      5   GNU Taler is free software; you can redistribute it and/or modify it under the
      6   terms of the GNU General Public License as published by the Free Software
      7   Foundation; either version 3, or (at your option) any later version.
      8 
      9   GNU Taler is distributed in the hope that it will be useful, but WITHOUT ANY
     10   WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
     11   A PARTICULAR PURPOSE.  See the GNU General Public License for more details.
     12 
     13   You should have received a copy of the GNU General Public License along with
     14   GNU Taler; see the file COPYING.  If not, see <http://www.gnu.org/licenses/>
     15 -->
     16 
     17 # Scanning a QR code reads the camera frame wrong, and one bad frame ends scanning
     18 
     19 Investigated after a report that scanning works on Android 15 but not on
     20 Android 14. The camera cannot be driven from an emulator — the virtual scene's
     21 camera pose cannot be aimed at a code — so the pipeline was taken apart
     22 instead: a camera frame is a luminance plane plus two numbers saying how it is
     23 laid out, which can be built on a workstation and put through the app's own
     24 arithmetic. Everything below is reproducible with a JDK and the zxing jar the
     25 app already depends on; no device, no emulator, no camera.
     26 
     27 **Severity: high.** Two independent defects, each of which stops scanning
     28 outright rather than degrading it. Neither produces an error message: the
     29 viewfinder keeps running, the user waves the phone at a valid code, and the
     30 wallet never reacts.
     31 
     32 ## Defect 1 — the frame's layout is ignored, so nothing decodes
     33 
     34 `QrCameraAnalyzer` converts each frame by hand and never looks at how it is
     35 laid out:
     36 
     37 ```kotlin
     38 // taler-kotlin-android/src/main/java/net/taler/lib/android/qr/QrCameraAnalyzer.kt:150
     39 private fun yuv420888ToNv21(image: Image): ByteArray {
     40     val yPlane = image.planes[0].buffer
     41     …
     42     val ySize = yPlane.remaining()
     43     yPlane.get(nv21, 0, ySize)       // one bulk copy
     44     …
     45 }
     46 ```
     47 
     48 ```console
     49 $ grep -c 'rowStride\|pixelStride' QrCameraAnalyzer.kt
     50 0
     51 ```
     52 
     53 A `YUV_420_888` plane is not a tightly packed image. `rowStride` is the
     54 distance between the starts of two rows and the hardware is free to pad, so
     55 `rowStride >= width`; `pixelStride` is the distance between two samples.
     56 (`ImageFormat.YUV_420_888` documents the Y plane's pixel stride as always one,
     57 so row padding is the part that varies in practice — the patch handles both
     58 because handling both costs nothing.) A single `get()` is correct only when the
     59 plane happens to be tight. With padded rows every row after the first lands
     60 shifted by the padding, the image shears a little further on each line, the
     61 finder patterns stop being squares, and zxing returns `NotFoundException` for
     62 every frame — which the analyzer swallows, so a systematically broken decoder
     63 looks exactly like an empty viewfinder.
     64 
     65 Run against the current code, `repro_qr_plane_strides.java`:
     66 
     67 ```
     68 == 1. control: tightly packed plane, rotation 0
     69 app pipeline, rowStride == width                           decoded
     70 
     71 == 2. the same frame with padded rows (rowStride > width)
     72 app pipeline, rowStride = 656 (pad 16)                     FAIL: NotFoundException
     73   patched extractLuma, same frame                          decoded
     74   zxing dataWidth = rowStride, no copy at all              decoded
     75 app pipeline, rowStride = 672 (pad 32)                     FAIL: NotFoundException
     76   …                                                        (same for 64, 128)
     77 
     78 == 3. interleaved luma (pixelStride 2)
     79 app pipeline                                               FAIL: NotFoundException
     80   patched extractLuma                                      decoded
     81 ```
     82 
     83 Not "degraded": no padding value decodes a single frame, and the same frame
     84 decodes the moment the stride is honoured. The code is in the frame, unchanged;
     85 only the arithmetic differs.
     86 
     87 ## Defect 2 — a frame that throws stops the camera for good
     88 
     89 `proxy.close()` sits in the `finally` of the *decode* `try`, which is entered
     90 after the conversion and rotation have already run:
     91 
     92 ```kotlin
     93 val nv21 = yuv420888ToNv21(mediaImage)          // throws here …
     94 val (rotatedNv21, …) = rotateNV21(nv21, …)      // … or here
     95 …
     96 try {
     97     onQrDetected(qrReader.decodeWithState(bitmap).text)
     98 } catch (_: NotFoundException) {
     99 } finally {
    100     proxy.close()                                // … and this never runs
    101 }
    102 ```
    103 
    104 The analysis use case is built with `STRATEGY_KEEP_ONLY_LATEST`, under which
    105 CameraX holds one frame and acquires the next only once the current proxy is
    106 closed. So an exception anywhere above that `try` does not cost one frame: it
    107 ends frame delivery for the rest of the screen's life, while the preview — a
    108 separate use case — carries on looking perfectly healthy.
    109 
    110 That is not hypothetical, because the two defects meet. `rotateNV21` sizes its
    111 output from `width * height` while the array it was handed was sized from the
    112 *padded* `remaining()`, and its 180° branch copies the chroma tail out to
    113 `data.size`:
    114 
    115 ```
    116 == 4. the app's rotation on a padded plane
    117 rotation 90 degrees, rowStride = 672                       FAIL: NotFoundException
    118 rotation 180 degrees, rowStride = 672                      THREW ArrayIndexOutOfBoundsException: Index 460800 out of bounds for length 460800
    119 rotation 270 degrees, rowStride = 672                      FAIL: NotFoundException
    120 ```
    121 
    122 `rotationDegrees == 180` is reached when the device is held in the orientation
    123 opposite to the sensor's, and nothing in the wallet constrains that:
    124 
    125 ```console
    126 $ grep -rn screenOrientation wallet/src/main/AndroidManifest.xml
    127 $               # no output -- the scan screen rotates freely
    128 ```
    129 
    130 So on a device whose camera pads its rows, turning the phone around during a
    131 scan kills scanning until the screen is left and re-entered — with no error,
    132 no log line, and a live preview.
    133 
    134 ## Defect 3 — the analysis runs on the UI thread
    135 
    136 ```kotlin
    137 useCase.setAnalyzer(mainExecutor) { proxy -> … }
    138 ```
    139 
    140 Every frame costs a copy of the luminance plane plus a zxing decode, and both
    141 happen on the main thread, at frame rate. It is not why scanning fails, but it
    142 is why the scan screen janks, and it makes a slow decode cost dropped frames.
    143 
    144 ## What is *not* wrong
    145 
    146 Worth recording, because each of these is a plausible place to go looking:
    147 
    148 * **The rotation arithmetic is correct.** On a tightly packed plane all three
    149   branches produce a decodable image (section 6 of the reproducer). The 270°
    150   branch's odd-looking indexing is in the chroma half, which nothing reads.
    151 * **The rotation is not needed at all.** zxing locates a QR code by its three
    152   finder patterns and derives the orientation from them:
    153 
    154   ```
    155   == 5. is the rotation needed at all?  (tight plane, kept as it arrives)
    156   frame arrives rotated 0, decoded as-is                     decoded
    157   frame arrives rotated 90, decoded as-is                    decoded
    158   frame arrives rotated 180, decoded as-is                   decoded
    159   frame arrives rotated 270, decoded as-is                   decoded
    160   ```
    161 
    162   So `rotateNV21` buys nothing, and it is where the crash lives. Deleting it is
    163   the fix, not repairing it.
    164 * **Asking zxing to rotate instead would be worse.** The obvious replacement,
    165   `PlanarYUVLuminanceSource.rotateCounterClockwise()`, is not implemented:
    166   zxing 3.5.3 *and* 3.5.4 inherit the base class, where
    167   `isRotateSupported()` returns false and the method throws
    168   `UnsupportedOperationException`. The merchant terminal's copy of this
    169   pipeline does exactly that — see
    170   `pos-config-scanner-asks-for-a-rotation-zxing-refuses.md`.
    171 * **The decoder and the hints are fine**, and so is the gallery path. Importing
    172   a photo of a code decodes 10 of 10 on API 34 and on API 36, with identical
    173   screens; that route goes through `decodeQrFromBitmap`, which builds an
    174   `RGBLuminanceSource` from `Bitmap.getPixels` and so never touches any of the
    175   arithmetic above. A working gallery import is therefore not evidence that the
    176   camera path works — it is the control that isolates it.
    177 
    178 ## Why this can differ between Android versions
    179 
    180 The layout is the camera's choice, not the app's: it comes from the HAL,
    181 together with the resolution CameraX selects from the device's advertised
    182 stream configurations. Both can change when a device is upgraded — the same
    183 hardware with a new vendor camera stack may pad where it did not before, or
    184 offer a different analysis resolution — and both differ between devices.
    185 "Works on 15, not on 14" is precisely the shape of a stride assumption, which
    186 is why the report points here. It is *not* proof: the two numbers that decide
    187 it can be printed in one log line, which is the first manual test below.
    188 
    189 ## Reproduce
    190 
    191 Without a device, against an unpatched checkout:
    192 
    193 ```bash
    194 ZX=$(find ~/.gradle ~/.m2 -name 'core-3.5.*.jar' | head -1)
    195 javac -cp "$ZX" -d /tmp/qrrepro simulator/findings/repro_qr_plane_strides.java
    196 java  -cp "$ZX:/tmp/qrrepro" repro_qr_plane_strides
    197 ```
    198 
    199 With the patch applied, the same ground is covered by unit tests that stay in
    200 the tree:
    201 
    202 ```bash
    203 ./gradlew :taler-kotlin-android:testDebugUnitTest \
    204     --tests 'net.taler.lib.android.qr.*'
    205 ```
    206 
    207 `ExtractLumaTest` (4 cases) pins the layout arithmetic;
    208 `QrFrameDecodeTest` (5 cases) encodes a real QR code, lays it out tight,
    209 padded by 16/32/64/128 bytes and interleaved, and asserts it decodes — plus
    210 one case asserting that the *old* bulk copy does not decode, and one that the
    211 code reads at all four orientations. All nine pass with the patch; the four
    212 `ExtractLumaTest` cases were written first and three of them failed against the
    213 original arithmetic.
    214 
    215 By hand on an affected device: open **Scan QR code**, point it at any valid
    216 `taler://` code, and nothing happens — no error, no reaction, and `adb logcat`
    217 shows nothing, because the per-frame `NotFoundException` is caught and dropped.
    218 
    219 ## Fix
    220 
    221 `qr-camera-ignores-the-plane-strides.diff` does all of it in one file:
    222 
    223 * `extractLuma(buffer, rowStride, pixelStride, width, height)` — a pure
    224   function, so it is testable without an emulator — copies the luminance plane
    225   row by row, skips the padding, steps by `pixelStride`, keeps the single
    226   `get()` fast path for the tight case, and drops the chroma copy that zxing
    227   never read.
    228 * `rotateNV21` is deleted (see above).
    229 * The whole frame body is wrapped in `try { … } catch (e: Exception) { Log.w }
    230   finally { proxy.close() }`, so no frame can stop the camera.
    231 * Analysis moves to a single-thread executor, with `onQrDetected` posted back
    232   to the main thread.
    233 
    234 Verified: `:taler-kotlin-android:compileDebugKotlin`, `:taler-kotlin-android:lintDebug`
    235 (0 errors) and the nine unit tests above all pass. Not verified on a real
    236 camera — that is what the manual tests are for.
    237 
    238 ### Reference implementations, none of them Google Play
    239 
    240 The arithmetic above is the minimal change. Two alternatives are worth knowing,
    241 because both move the problem out of the app entirely, and neither needs Play
    242 Services (no ML Kit anywhere):
    243 
    244 1. **Let zxing take the stride.** `PlanarYUVLuminanceSource`'s constructor
    245    already separates the buffer's layout from the image:
    246    `(yuvData, dataWidth, dataHeight, left, top, width, height, reverseHoriz)`.
    247    Passing `rowStride` as `dataWidth` and `width`/`height` as the crop decodes
    248    a padded frame with no stride arithmetic in the app at all — the
    249    "no copy at all" line in section 2 of the reproducer. This is what the
    250    merchant terminal's patch uses, being a four-line change there.
    251 2. **Let CameraX do the conversion.** `camera-core` 1.5.3 (already a
    252    dependency) carries, verified by `javap` on the resolved artifact:
    253    `ImageAnalysis.Builder.setOutputImageFormat(OUTPUT_IMAGE_FORMAT_NV21)`,
    254    `setOutputImageRotationEnabled(boolean)`, and internally
    255    `ImageProcessingUtil.rotateYUVAndConvertToNV21` over a libyuv
    256    `nativeRotateYUV`. That delivers a tightly packed, already-rotated NV21
    257    buffer from native code, which is exactly what `PlanarYUVLuminanceSource`
    258    wants, and deletes both hand-written functions. CameraX's own stride-aware
    259    conversion, `androidx.camera.core.internal.utils.ImageUtil.yuv_420_888toNv21`,
    260    is worth reading as the reference arithmetic but is a restricted API — read
    261    it, do not call it.
    262    `OUTPUT_IMAGE_FORMAT_RGBA_8888` plus `ImageProxy.toBitmap()` would go
    263    further and let the camera and gallery paths share the one
    264    `decodeQrFromBitmap` decoder, at the cost of an RGBA conversion per frame
    265    where zxing only ever reads luminance.
    266 
    267 Worth doing alongside, in any variant: count consecutive failed decodes and log
    268 one debug line when the count crosses a threshold. Both defects above were
    269 silent, and that single counter would have named either of them in a minute.
    270 
    271 ## Attended tests the developer should run
    272 
    273 These need a human holding a phone; each one is chosen to separate a cause
    274 rather than to confirm a guess.
    275 
    276 1. **Print the layout, once per session.** `rowStride`, `pixelStride`,
    277    `width`, `height` of `planes[0]`, plus `rotationDegrees` and the resolution
    278    CameraX chose, on every device and OS version in the report. This is the
    279    decisive test: if `rowStride == width` on Android 15 and `rowStride > width`
    280    on Android 14 for the same hardware, defect 1 is the report, and nothing
    281    else needs proving. Ideally on the same device before and after the upgrade.
    282 2. **Scan in all four orientations**, portrait, both landscapes and upside
    283    down, on a device whose rows are padded. Reverse orientation is where
    284    `rotationDegrees == 180`, the branch that throws.
    285 3. **After a failed scan, check whether frames still arrive** — a temporary
    286    per-frame counter in the analyzer is enough. "No frames" means defect 2;
    287    "frames but no decode" means defect 1. Then leave the screen and come back:
    288    if scanning works again until the next failure, that is defect 2 confirmed.
    289 4. **Import a photo of the same code from the gallery.** It goes through a
    290    different decoder. Gallery decodes, camera does not ⇒ the fault is in the
    291    frame handling, not in the code, the decoder or the lighting.
    292 5. **Vary the code's size in the frame and the light**: filling the frame,
    293    then small and at arm's length, then dim. A layout fault fails all of them
    294    identically; focus and resolution problems do not.
    295 6. **Scan a non-Taler QR code** (any URL). It exercises the same pipeline
    296    without the app's URI handling, so a decode that the wallet ignores
    297    separates "could not read the code" from "read it and did nothing with it".