Skip to content

Add codec-level scaled image decoding#3934

Open
PradyumnaShome wants to merge 1 commit into
Shopify:mainfrom
PradyumnaShome:feat/scaled-image-decode
Open

Add codec-level scaled image decoding#3934
PradyumnaShome wants to merge 1 commit into
Shopify:mainfrom
PradyumnaShome:feat/scaled-image-decode

Conversation

@PradyumnaShome

Copy link
Copy Markdown

Summary

  • add Skia.Image.MakeImageFromEncodedScaled(data, targetWidth, targetHeight?)
  • decode through SkCodec::getScaledDimensions() and getImage() on native platforms instead of always creating a full-resolution decoded image
  • preserve aspect ratio, account for EXIF orientation, reject invalid dimensions, and never request upscaling
  • provide equivalent output behavior on Web by resizing after CanvasKit's full-resolution decode
  • document codec approximation and fallback behavior, with targeted tests for reduced dimensions and invalid inputs

Motivation

Callers often need a thumbnail or an image for a smaller canvas/compositing target. Decoding every source at full resolution first adds avoidable CPU work and peak memory pressure, especially for large encoded images. This API lets native codecs decode at their closest supported scale. A codec without scaled-decode support falls back to source resolution; CanvasKit does not expose codec-level scaled decoding, so Web falls back to full decode followed by resize.

No performance numbers are claimed here. A follow-up benchmark can compare wall time and peak resident memory for full decode versus scaled decode across representative large JPEG, PNG, and WebP inputs and several target scales.

Semantics

  • targetWidth is required and must be a positive finite number.
  • targetHeight, when provided, must also be positive and finite.
  • Both dimensions form an aspect-preserving fit box.
  • Native output uses the codec's closest supported scale, so it can be slightly larger or smaller than the requested box.
  • Requests at or above source size do not upscale.
  • Invalid data, invalid dimensions, or decode failure return null.

Validation

  • yarn workspace @shopify/react-native-skia tsc
  • targeted ESLint for changed TypeScript/test files
  • yarn workspace @shopify/react-native-skia test src/renderer/__tests__/e2e/Image.spec.tsx --runInBand (5 tests)
  • pod install including React Native codegen
  • iOS Simulator Debug build of ReactTestApp
  • Android build (local environment has no Java runtime)

This is a draft until Android CI or a local JDK-backed build validates the shared C++ binding through the Android toolchain.

Avoid full-resolution CPU and memory costs when callers only need a smaller target, while documenting codec and web fallbacks.
@PradyumnaShome

Copy link
Copy Markdown
Author

I have signed the CLA!

@PradyumnaShome PradyumnaShome marked this pull request as ready for review July 11, 2026 08:25
@wcandillon wcandillon self-requested a review July 13, 2026 06:21

@wcandillon wcandillon left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank You for making this PR.

  1. Do you have examples/were you able to run some measurements on this? Currently the image is decode on UI thread (lazy) and this would decode the image on the JS thread, I'm curious to learn more about how you tested/measured this.
  2. I think this should be an option in existing decoding method rather than a new method.
  3. On web, FilterMode.Linear + MipmapMode.None is likely the wrong default

@wcandillon

Copy link
Copy Markdown
Contributor

Thank You for making this PR. Here are my current thoughts on the topic:

  1. On the Web this could potentially be an important improvement to not load the encoded data in the WASM memory. This would be done via createImageBitmap which supports the resize options
  2. We offer createImageBitmap on iOS and Android for WebGPU (this was the only sensible path for WebGPU users' to decode image). createImageBitmap supports these options. So I am wondering if the move is not now to push the createImageBitmap implementation for iOS and Android. That implementation relies on platform codecs instead of Skia codecs, I would need to investigate this a bit further.

@PradyumnaShome

Copy link
Copy Markdown
Author

Hi William! Did some benchmarking and this does not actually help, unfortunately. I do have a patch that lets you do scaled decoding for JPEGs, but it does not work for PNGs and so I'm not sure if that would still be helpful. I wish there was a universal patch.

I guess the long-term solution for our app is to just maintain copies at different aspect ratios, instead of trying to decode lower-quality versions via Skia paths.

Would scaled JPEG decoding help? In our app we do so much decoding that I feel like we're always looking for ways to speed this up. Curious if you have thoughts on good directions on streamlining this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants