fix(ios): bypass CIContext+createCGImage in MLKVisionImage to fix 300+ MiB IOSurface leak#864
Open
Toker38 wants to merge 1 commit into
Open
Conversation
…er path Each call to pixelBufferToVisionImage created a fresh CIContext and ran createCGImage(_:from:) on it. Even after the local CIContext is released by ARC, the underlying CI::SurfaceCacheEntry retains the IOSurface in the system's IOAccelerator warm cache. Under sustained camera streaming this leaks ~3.5 MiB per call (one 720p BGRA frame buffer), reaching 300+ MiB of VM:IOSurface memory in 60 seconds and continuing until the OS terminates the app. A shared CIContext + autoreleasepool does not stop the growth (Apple Developer Forums thread 17142). The only way to bypass the cache is to avoid createCGImage entirely. VisionImage(buffer: CMSampleBuffer) accepts the CVPixelBuffer wrapped in a CMSampleBuffer directly. CoreImage is no longer involved, so no SurfaceCacheEntry is ever created. This is also the path Google's official MLKit iOS sample uses (googlesamples/mlkit/ios/quickstarts/vision/VisionExample/CameraViewController.swift). Verified on a private fork: VM:IOSurface growth in a 60s no-face camera stream drops from 341 MiB to near-baseline. Detection accuracy and latency are unchanged. See flutter-ml#863 for full Instruments traces and measurements.
udiedrichsen
added a commit
to moinsen-dev/google_ml_kit_flutter
that referenced
this pull request
May 16, 2026
…l_kit_flutter - PR flutter-ml#864 / Issue flutter-ml#863: Fix iOS IOSurface memory leak in pixelBufferToVisionImage. Replaces CoreImage CIContext+createCGImage round-trip with VisionImage(buffer:) via CMSampleBuffer wrapper, eliminating ~3.5 MiB leak per frame on sustained camera streaming. - PR flutter-ml#862 / Issue flutter-ml#825: Add Apple Silicon iOS 26+ simulator support. Ships opt-in Podfile helper (apple_silicon_simulator.rb + patch_arm64_simulator.py) that re-labels arm64 device slices as iOS Simulator and strips EXCLUDED_ARCHS. Updates README with usage instructions and wires up example app Podfile. - Issue flutter-ml#857: Fix IllegalStateException 'Reply already submitted' crash in DocumentScanner on Oppo/ColorOS devices. Clears pendingResult after calling .success() or .error() so duplicate onActivityResult deliveries are safe.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
pixelBufferToVisionImageinstantiates a freshCIContextper frame and renders viacreateCGImage(_:from:). Each call inserts aCI::SurfaceCacheEntrythat the system never releases under sustained camera streaming — leaking ~3.5 MiB ofVM: IOSurfaceper call (one 720p BGRA frame buffer). On a real device this reaches 300+ MiB in 60 seconds and continues to climb until the OS terminates the app.This PR replaces the CoreImage round-trip with
VisionImage(buffer: CMSampleBuffer)— the same path used by Google's official MLKit iOS sample. CoreImage is no longer involved, so theSurfaceCacheEntryis never created.Fixes #863. Likely also addresses #708 and #790 on iOS.
Why a shared
CIContextis not enoughPer the Apple Developer Forums thread on this exact symptom,
@autoreleasepoolblocks, sharing a single CIContext, and explicitCGImageReleaseare listed as not confirmed working for this leak. The cache entry retention happens at the IOAccelerator layer, beyond ARC's reach. The only effective mitigation is to avoidcreateCGImageentirely — which is whatVisionImage(buffer:)does.Measurements (Instruments → Allocations on iOS, --release build)
Workload:
camera ^0.11.x,ImageFormatGroup.bgra8888,ResolutionPreset.high(720p),FaceDetector.processImageat ~3 FPS, no face in frame, 60 s sustained.Before
Each surface is exactly 3.52 MiB = 1280 × 720 × 4 (BGRA frame buffer). Net: ~8.5 MiB/sec of unbounded IOSurface growth.
Stack trace (largest persistent allocation):
After (CMSampleBuffer path)
VM: IOSurfacegrowth in 60 s drops from 341 MiB → near baseline (no observable accumulation per generation). Surfaces that the camera pipeline holds for active frames are still allocated normally; the per-processImagecache entries are gone.Detection accuracy and latency are unchanged — MLKit's vision detectors consume
CMSampleBuffernatively at the same speed asUIImage-backed vision images.Implementation notes
VisionImage(buffer:)requires aCMSampleBuffer, which we build from theCVPixelBufferviaCMVideoFormatDescriptionCreateForImageBuffer+CMSampleBufferCreateForImageBuffer. No CoreImage allocations are made.kCMTimeInvalid/kCMTimeZerotiming is appropriate for a single-frame buffer that's not part of a continuous decode session.CMVideoFormatDescriptionandCMSampleBufferare managed by Swift ARC for CFTypes — no manualCFReleaseneeded.nil(matching the existing nil-returning pattern in this file) so callers degrade gracefully instead of crashing.CoreMediais added as an explicit import.Test plan
VM: IOSurfacegrowth eliminated.