improvement(execution): memory usage for aggregated results (#4650)

* improvement(execution): memory usage for aggregated results

* progress

* address comments

* loop/parallel results compaction

* address comment

* remove build files, harden edge cases

* remove hotpath serialiazation

* display change to make use of only preview

* materialize refs before sending in response block

* preserve exact large-value access through workflow materialization

* address comments

* progress

* fix notif error + sync manifest undefined exit

* fix streaming ref materialization

* fix tests

Author: icecrasher321

apps/docs/content/docs/en/blocks/function.mdx

@@ -198,7 +198,7 @@ const file = <readfile.file>;
 const base64 = await sim.files.readBase64(file);
 ```
 
-`sim.files.readBase64(file)`, `sim.files.readText(file)`, `sim.files.readBase64Chunk(file, { offset, length })`, and `sim.files.readTextChunk(file, { offset, length })` read from server-side execution storage under memory caps. `sim.values.read(ref)` can explicitly read a large execution value reference. These helpers are available only in JavaScript functions without imports. JavaScript with imports, Python, and shell do not support these lazy helpers yet.
+`sim.files.readBase64(file)`, `sim.files.readText(file)`, `sim.files.readBase64Chunk(file, { offset, length })`, and `sim.files.readTextChunk(file, { offset, length })` read from server-side execution storage under memory caps. `sim.values.read(ref)` explicitly reads a large execution value reference, and `sim.values.readArray(ref)` reads a manifest-backed large array. These helpers are available only in JavaScript functions without imports. JavaScript with imports, Python, and shell do not support these lazy helpers yet.
 
 Very large full reads can still fail by design; use chunk helpers or return a file when you need to handle more data.
 
@@ -228,7 +228,7 @@ return { name: file.name, chunk: firstMegabyteBase64 };
 
 Chunk `offset` and `length` are byte-based. For Unicode text, a chunk can split a multi-byte character at the boundary; use text chunks for approximate text processing and prefer smaller structured references when exact parsing matters.
 
-Avoid passing a full large object into a Function block when you only need one field. For example, prefer `<api.data.customerId>` over `<api.data>` when the API response is large. If a JavaScript Function without imports references a large execution value, Sim automatically reads it through `sim.values.read(...)` at runtime under memory caps.
+Avoid passing a full large object into a Function block when you only need one field. For example, prefer `<api.data.customerId>` over `<api.data>` when the API response is large. If a JavaScript Function without imports references a whole large execution value, Sim automatically rewrites it to `sim.values.read(...)` at runtime under memory caps. If the value is a manifest-backed array, Sim rewrites it to `sim.values.readArray(...)` so array variables can stay compact between blocks.
 
 For large generated data, write the result to a file or table with `outputPath`, `outputSandboxPath`, or `outputTable` instead of returning the entire payload inline.
 

apps/docs/content/docs/en/execution/api-deployment.mdx

@@ -232,7 +232,7 @@ Workflow execution responses are capped by platform request and response limits.
 }
 ```
 
-The `version` field is part of the external API contract. Treat the reference as an opaque placeholder for a value that could not be safely embedded in the response. `id`, `key`, and `executionId` are not fetch URLs; `key` points to execution-scoped server storage. Use `selectedOutputs` to request a smaller nested field, reduce the data passed between blocks, or return the data from a Response block when your workflow intentionally owns the HTTP response body. File outputs are metadata-first; request `.base64` only when you need inline file content. JavaScript Function blocks can explicitly read large files or value refs with the `sim.files` and `sim.values` helpers under memory caps.
+The `version` field is part of the external API contract. Treat the reference as an opaque placeholder for a value that could not be safely embedded in the response. `id`, `key`, and `executionId` are not fetch URLs; `key` points to execution-scoped server storage. Use `selectedOutputs` to request a smaller nested field, reduce the data passed between blocks, or return the data from a Response block when your workflow intentionally owns the HTTP response body. File outputs are metadata-first; request `.base64` only when you need inline file content. JavaScript Function blocks can explicitly read large files, value refs, and manifest-backed arrays with the `sim.files` and `sim.values` helpers under memory caps.
 
 ### Asynchronous
 

apps/sim/app/api/function/execute/route.test.ts

@@ -12,9 +12,10 @@ import {
 import { NextRequest } from 'next/server'
 import { beforeEach, describe, expect, it, vi } from 'vitest'
 
-const { mockExecuteInE2B, mockExecuteInIsolatedVM } = vi.hoisted(() => ({
+const { mockExecuteInE2B, mockExecuteInIsolatedVM, mockUploadFile } = vi.hoisted(() => ({
   mockExecuteInE2B: vi.fn(),
   mockExecuteInIsolatedVM: vi.fn(),
+  mockUploadFile: vi.fn(),
 }))
 
 vi.mock('@/lib/execution/isolated-vm', () => ({
@@ -42,16 +43,26 @@ vi.mock('@/lib/uploads/contexts/workspace/workspace-file-manager', () => ({
   uploadWorkspaceFile: vi.fn(),
 }))
 
+vi.mock('@/lib/uploads', () => ({
+  StorageService: {
+    uploadFile: mockUploadFile,
+  },
+}))
+
 vi.mock('@/lib/workflows/utils', () => workflowsUtilsMock)
 
 vi.mock('@/lib/core/config/feature-flags', () => featureFlagsMock)
 
 import { validateProxyUrl } from '@/lib/core/security/input-validation'
+import { clearLargeValueCacheForTests } from '@/lib/execution/payloads/cache'
+import { isLargeArrayManifest } from '@/lib/execution/payloads/large-array-manifest-metadata'
+import { isLargeValueRef } from '@/lib/execution/payloads/large-value-ref'
 import { POST } from '@/app/api/function/execute/route'
 
 describe('Function Execute API Route', () => {
   beforeEach(() => {
     vi.clearAllMocks()
+    featureFlagsMock.isE2bEnabled = false
 
     hybridAuthMockFns.mockCheckInternalAuth.mockResolvedValue({
       success: true,
@@ -60,6 +71,8 @@ describe('Function Execute API Route', () => {
     })
 
     mockExecuteInIsolatedVM.mockResolvedValue({ result: 'test', stdout: '' })
+    mockUploadFile.mockImplementation(async ({ customKey }) => ({ key: customKey }))
+    clearLargeValueCacheForTests()
 
     mockExecuteInE2B.mockResolvedValue({
       result: 'e2b success',
@@ -201,6 +214,60 @@ describe('Function Execute API Route', () => {
       expect(data.output).toHaveProperty('executionTime')
     })
 
+    it('compacts large array result fields to manifests when execution context is durable', async () => {
+      mockExecuteInIsolatedVM.mockResolvedValueOnce({
+        result: {
+          rows: Array.from({ length: 120_000 }, (_, index) => ({
+            key: `SIM-${index}`,
+            payload: 'x'.repeat(100),
+          })),
+        },
+        stdout: '',
+      })
+
+      const req = createMockRequest('POST', {
+        code: 'return rows',
+        workflowId: 'workflow-1',
+        workspaceId: 'workspace-1',
+        executionId: 'execution-1',
+      })
+
+      const response = await POST(req)
+      const data = await response.json()
+
+      expect(response.status).toBe(200)
+      expect(data.success).toBe(true)
+      expect(isLargeArrayManifest(data.output.result.rows)).toBe(true)
+      expect(data.output.result.rows).toMatchObject({
+        __simLargeArrayManifest: true,
+        kind: 'array',
+        totalCount: 120_000,
+      })
+    })
+
+    it('keeps large string result fields as generic large value refs', async () => {
+      mockExecuteInIsolatedVM.mockResolvedValueOnce({
+        result: {
+          text: 'x'.repeat(9 * 1024 * 1024),
+        },
+        stdout: '',
+      })
+
+      const req = createMockRequest('POST', {
+        code: 'return text',
+        workflowId: 'workflow-1',
+        workspaceId: 'workspace-1',
+        executionId: 'execution-1',
+      })
+
+      const response = await POST(req)
+      const data = await response.json()
+
+      expect(response.status).toBe(200)
+      expect(data.success).toBe(true)
+      expect(isLargeValueRef(data.output.result.text)).toBe(true)
+    })
+
     it('should return computed result for multi-line code', async () => {
       mockExecuteInIsolatedVM.mockResolvedValueOnce({ result: 10, stdout: '' })
 
@@ -240,6 +307,73 @@ describe('Function Execute API Route', () => {
       expect(response.status).toBe(200)
       expect(data.success).toBe(true)
     })
+
+    it('rejects large refs in runtimes without ref-native helpers', async () => {
+      featureFlagsMock.isE2bEnabled = true
+      const req = createMockRequest('POST', {
+        code: 'echo "$__blockRef_0"',
+        language: 'shell',
+        contextVariables: {
+          __blockRef_0: {
+            __simLargeValueRef: true,
+            version: 1,
+            id: 'lv_ABCDEFGHIJKL',
+            kind: 'array',
+            size: 12 * 1024 * 1024,
+            executionId: 'execution-1',
+          },
+        },
+      })
+
+      const response = await POST(req)
+      const data = await response.json()
+
+      expect(response.status).toBe(500)
+      expect(data.success).toBe(false)
+      expect(data.error).toContain(
+        'Large execution values require the JavaScript isolated-vm runtime'
+      )
+    })
+
+    it('registers manifest array read broker for isolated-vm execution', async () => {
+      const req = createMockRequest('POST', {
+        code: 'return await sim.values.readArray(__blockRef_0)',
+        language: 'javascript',
+        contextVariables: {
+          __blockRef_0: {
+            __simLargeArrayManifest: true,
+            version: 2,
+            kind: 'array',
+            totalCount: 1,
+            chunkCount: 1,
+            byteSize: 16,
+            chunks: [
+              {
+                ref: {
+                  __simLargeValueRef: true,
+                  version: 1,
+                  id: 'lv_ABCDEFGHIJKL',
+                  kind: 'array',
+                  size: 16,
+                  executionId: 'execution-1',
+                },
+                count: 1,
+                byteSize: 16,
+              },
+            ],
+            preview: [{ id: 1 }],
+          },
+        },
+      })
+
+      const response = await POST(req)
+      const data = await response.json()
+      const [, options] = mockExecuteInIsolatedVM.mock.calls.at(-1) ?? []
+
+      expect(response.status).toBe(200)
+      expect(data.success).toBe(true)
+      expect(options?.brokers).toHaveProperty('sim.values.readArray')
+    })
   })
 
   describe('Template Variable Resolution', () => {

apps/sim/app/api/function/execute/route.ts

@@ -14,7 +14,12 @@ import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
 import { executeInE2B, executeShellInE2B } from '@/lib/execution/e2b'
 import { executeInIsolatedVM, type IsolatedVMBrokerHandler } from '@/lib/execution/isolated-vm'
 import { CodeLanguage, DEFAULT_CODE_LANGUAGE, isValidCodeLanguage } from '@/lib/execution/languages'
-import { isLargeValueRef } from '@/lib/execution/payloads/large-value-ref'
+import { recordMaterializedAccessKeys } from '@/lib/execution/payloads/access-keys'
+import {
+  isLargeArrayManifest,
+  materializeLargeArrayManifest,
+} from '@/lib/execution/payloads/large-array-manifest'
+import { containsLargeValueRef, isLargeValueRef } from '@/lib/execution/payloads/large-value-ref'
 import {
   MAX_FUNCTION_INLINE_BYTES,
   MAX_INLINE_MATERIALIZATION_BYTES,
@@ -699,6 +704,8 @@ interface FunctionRouteExecutionContext {
   workspaceId?: string
   executionId?: string
   largeValueExecutionIds?: string[]
+  largeValueKeys?: string[]
+  fileKeys?: string[]
   allowLargeValueWorkflowScope?: boolean
   userId?: string
   requestId: string
@@ -741,17 +748,26 @@ function getBrokerFileArgs(args: unknown): {
 function createFunctionRuntimeBrokers(
   context: FunctionRouteExecutionContext
 ): Record<string, IsolatedVMBrokerHandler> {
+  context.largeValueKeys ??= []
+  context.fileKeys ??= []
+  const largeValueKeys = context.largeValueKeys
+  const fileKeys = context.fileKeys
   const base = {
     requestId: context.requestId,
     workflowId: context.workflowId,
     workspaceId: context.workspaceId,
     executionId: context.executionId,
     largeValueExecutionIds: context.largeValueExecutionIds,
+    largeValueKeys,
+    fileKeys,
     allowLargeValueWorkflowScope: context.allowLargeValueWorkflowScope,
     userId: context.userId,
     logger,
   }
 
+  const recordMaterializedKeys = (value: unknown) =>
+    recordMaterializedAccessKeys({ largeValueKeys, fileKeys }, value)
+
   const readFile = async (args: unknown, encoding: 'base64' | 'text', chunked = false) => {
     const fileArgs = getBrokerFileArgs(args)
     return readUserFileContent(fileArgs.file, {
@@ -786,6 +802,24 @@ function createFunctionRuntimeBrokers(
       if (value === undefined) {
         throw unavailableLargeValueError(ref)
       }
+      recordMaterializedKeys(value)
+      return value
+    },
+    'sim.values.readArray': async (args) => {
+      const record = asRecord(args)
+      const options = asRecord(record.options)
+      const manifest = record.ref
+      if (!isLargeArrayManifest(manifest)) {
+        throw new Error('Expected a large array manifest.')
+      }
+      if (!context.executionId) {
+        throw new Error('Large array manifests require an execution context.')
+      }
+      const value = await materializeLargeArrayManifest(manifest, {
+        ...base,
+        maxBytes: clampInlineBytes(options.maxBytes, MAX_INLINE_MATERIALIZATION_BYTES),
+      })
+      recordMaterializedKeys(value)
       return value
     },
   }
@@ -810,7 +844,17 @@ async function functionJsonResponse<T>(
   context: FunctionRouteExecutionContext,
   init?: ResponseInit
 ) {
-  return NextResponse.json(await compactFunctionRouteBody(body, context), init)
+  return NextResponse.json(
+    await compactFunctionRouteBody(
+      {
+        ...body,
+        largeValueKeys: context.largeValueKeys,
+        fileKeys: context.fileKeys,
+      },
+      context
+    ),
+    init
+  )
 }
 
 async function maybeExportSandboxFileToWorkspace(args: {
@@ -955,6 +999,8 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
       workflowId,
       executionId,
       largeValueExecutionIds,
+      largeValueKeys,
+      fileKeys,
       allowLargeValueWorkflowScope = false,
       workspaceId,
       isCustomTool = false,
@@ -979,6 +1025,8 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
       workspaceId,
       executionId,
       largeValueExecutionIds,
+      largeValueKeys,
+      fileKeys,
       allowLargeValueWorkflowScope,
       userId: auth.userId,
       requestId,
@@ -1013,6 +1061,12 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
       contextVariables = { ...codeResolution.contextVariables, ...preResolvedContextVariables }
     }
 
+    if (lang === CodeLanguage.Shell && containsLargeValueRef(contextVariables)) {
+      throw new Error(
+        'Large execution values require the JavaScript isolated-vm runtime. Select a nested field or read the value in a JavaScript function.'
+      )
+    }
+
     let jsImports = ''
     let jsRemainingCode = resolvedCode
     let hasImports = false
@@ -1124,6 +1178,12 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
       !isCustomTool &&
       (lang === CodeLanguage.Python || (lang === CodeLanguage.JavaScript && hasImports))
 
+    if (useE2B && containsLargeValueRef(contextVariables)) {
+      throw new Error(
+        'Large execution values require the JavaScript isolated-vm runtime. Remove imports, select a nested field, or read the value in a JavaScript function without E2B.'
+      )
+    }
+
     if (useE2B) {
       logger.info(`[${requestId}] E2B status`, {
         enabled: isE2bEnabled,