Skip to content

fix(ios): bypass CIContext+createCGImage in MLKVisionImage to fix 300+ MiB IOSurface leak#864

Open
Toker38 wants to merge 1 commit into
flutter-ml:developfrom
Toker38:fix/ios-iosurface-leak-cmsamplebuffer
Open

fix(ios): bypass CIContext+createCGImage in MLKVisionImage to fix 300+ MiB IOSurface leak#864
Toker38 wants to merge 1 commit into
flutter-ml:developfrom
Toker38:fix/ios-iosurface-leak-cmsamplebuffer

Conversation

@Toker38
Copy link
Copy Markdown

@Toker38 Toker38 commented May 12, 2026

Summary

pixelBufferToVisionImage instantiates a fresh CIContext per frame and renders via createCGImage(_:from:). Each call inserts a CI::SurfaceCacheEntry that the system never releases under sustained camera streaming — leaking ~3.5 MiB of VM: IOSurface per call (one 720p BGRA frame buffer). On a real device this reaches 300+ MiB in 60 seconds and continues to climb until the OS terminates the app.

This PR replaces the CoreImage round-trip with VisionImage(buffer: CMSampleBuffer) — the same path used by Google's official MLKit iOS sample. CoreImage is no longer involved, so the SurfaceCacheEntry is never created.

Fixes #863. Likely also addresses #708 and #790 on iOS.

Why a shared CIContext is not enough

Per the Apple Developer Forums thread on this exact symptom, @autoreleasepool blocks, sharing a single CIContext, and explicit CGImageRelease are listed as not confirmed working for this leak. The cache entry retention happens at the IOAccelerator layer, beyond ARC's reach. The only effective mitigation is to avoid createCGImage entirely — which is what VisionImage(buffer:) does.

Measurements (Instruments → Allocations on iOS, --release build)

Workload: camera ^0.11.x, ImageFormatGroup.bgra8888, ResolutionPreset.high (720p), FaceDetector.processImage at ~3 FPS, no face in frame, 60 s sustained.

Before

Generation Duration IOSurface growth Allocations
A (warmup) 0–25 s 136 MiB 161
B 25–64 s 341 MiB 97
C 64–86 s 186 MiB 53

Each surface is exactly 3.52 MiB = 1280 × 720 × 4 (BGRA frame buffer). Net: ~8.5 MiB/sec of unbounded IOSurface growth.

Stack trace (largest persistent allocation):

IOSurfaceClientCreateChild
-[IOSurface initWithProperties:]
IOSurfaceCreate
CreateCachedSurface
CI::SurfaceCacheEntry::SurfaceCacheEntry
CI::ProviderNode::surfaceForROI
CI::Context::render
-[CIContext(_createCGImageInternal) _createCGImage:fromRect:format:premultiplied:colorSpace:deferred:renderCallback:]
-[CIContext(createCGImage) createCGImage:fromRect:]
+[MLKVisionImage(FlutterPlugin) pixelBufferToVisionImage:]    <-- plugin entry
+[MLKVisionImage(FlutterPlugin) bytesToVisionImage:]
-[GoogleMlKitFaceDetectionPlugin handleDetection:result:]

After (CMSampleBuffer path)

VM: IOSurface growth in 60 s drops from 341 MiB → near baseline (no observable accumulation per generation). Surfaces that the camera pipeline holds for active frames are still allocated normally; the per-processImage cache entries are gone.

Detection accuracy and latency are unchanged — MLKit's vision detectors consume CMSampleBuffer natively at the same speed as UIImage-backed vision images.

Implementation notes

  • VisionImage(buffer:) requires a CMSampleBuffer, which we build from the CVPixelBuffer via CMVideoFormatDescriptionCreateForImageBuffer + CMSampleBufferCreateForImageBuffer. No CoreImage allocations are made.
  • kCMTimeInvalid / kCMTimeZero timing is appropriate for a single-frame buffer that's not part of a continuous decode session.
  • CMVideoFormatDescription and CMSampleBuffer are managed by Swift ARC for CFTypes — no manual CFRelease needed.
  • Failure paths return nil (matching the existing nil-returning pattern in this file) so callers degrade gracefully instead of crashing.
  • CoreMedia is added as an explicit import.

Test plan

  • Verified on iPhone (iOS 26.2, real device, --release build) under sustained 60 s camera stream with face detection — VM: IOSurface growth eliminated.
  • Verified detection results unchanged (face detection accepts CMSampleBuffer-backed VisionImages).
  • Maintainers' CI on packaged example app.

…er path

Each call to pixelBufferToVisionImage created a fresh CIContext and ran
createCGImage(_:from:) on it. Even after the local CIContext is released
by ARC, the underlying CI::SurfaceCacheEntry retains the IOSurface in
the system's IOAccelerator warm cache. Under sustained camera streaming
this leaks ~3.5 MiB per call (one 720p BGRA frame buffer), reaching
300+ MiB of VM:IOSurface memory in 60 seconds and continuing until the
OS terminates the app.

A shared CIContext + autoreleasepool does not stop the growth (Apple
Developer Forums thread 17142). The only way to bypass the cache is to
avoid createCGImage entirely.

VisionImage(buffer: CMSampleBuffer) accepts the CVPixelBuffer wrapped in
a CMSampleBuffer directly. CoreImage is no longer involved, so no
SurfaceCacheEntry is ever created. This is also the path Google's
official MLKit iOS sample uses
(googlesamples/mlkit/ios/quickstarts/vision/VisionExample/CameraViewController.swift).

Verified on a private fork: VM:IOSurface growth in a 60s no-face camera
stream drops from 341 MiB to near-baseline. Detection accuracy and
latency are unchanged. See flutter-ml#863 for full Instruments traces and
measurements.
@Toker38 Toker38 changed the base branch from master to develop May 12, 2026 20:42
udiedrichsen added a commit to moinsen-dev/google_ml_kit_flutter that referenced this pull request May 16, 2026
…l_kit_flutter

- PR flutter-ml#864 / Issue flutter-ml#863: Fix iOS IOSurface memory leak in pixelBufferToVisionImage.
  Replaces CoreImage CIContext+createCGImage round-trip with VisionImage(buffer:)
  via CMSampleBuffer wrapper, eliminating ~3.5 MiB leak per frame on sustained
  camera streaming.

- PR flutter-ml#862 / Issue flutter-ml#825: Add Apple Silicon iOS 26+ simulator support.
  Ships opt-in Podfile helper (apple_silicon_simulator.rb + patch_arm64_simulator.py)
  that re-labels arm64 device slices as iOS Simulator and strips EXCLUDED_ARCHS.
  Updates README with usage instructions and wires up example app Podfile.

- Issue flutter-ml#857: Fix IllegalStateException 'Reply already submitted' crash in
  DocumentScanner on Oppo/ColorOS devices. Clears pendingResult after calling
  .success() or .error() so duplicate onActivityResult deliveries are safe.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

iOS: 300+ MiB IOSurface leak from CIContext+createCGImage in MLKVisionImage+FlutterPlugin (release builds)

1 participant