Add changes for compatibility with WASM components and collocated UDF servers

[kesmit13]

This PR makes several changes to allow the singlestoredb work in WASM environments. Many of these changes benefit standard installations as well such as lazy loading of numpy, pandas, polars, and pyarrow. Others move imports that are only needed in certain environments, but not within WASM.

A new collocated UDF server implementation is also included that uses a high-performance loop in the C extension to parse and call Python functions on each row. This function is used both by standard collocated servers as well as WASM-based UDF handlers.

---

Note

High Risk
High risk because it adds a new Unix-socket UDF execution mode with dynamic exec()-based function registration and significantly expands the C accelerator for ROWDAT_1 parsing/serialization and socket/mmap I/O, which can affect stability and memory-safety/performance on the UDF hot path.

Overview
Adds a new plugin-mode UDF server (singlestoredb.functions.ext.plugin) that serves Python UDFs over a Unix socket with thread or pre-fork process pools, including a CLI entry point (python-udf-server) and control signals like @@health, @@functions, and @@register (dynamic function registration).

Expands the C extension (accel.c) to support ROWDAT_1 DECIMAL/DATE/TIME/DATETIME/TIMESTAMP encode/decode, introduces a combined call_function_accel load→call→dump fast path, and adds low-level helpers (mmap_read, mmap_write, recv_exact) with WASM stubs. Updates JSON/ASGI UDF typing/encoding for datetime/decimal, implements VECTOR() dtype, and shifts numpy/pandas/polars/pyarrow handling toward lazy/type-map access.

Also refactors internal result handling in Connection._iquery to accept multiple dataframe/table return types via duck-typing, tweaks SHOW output key handling, and moves jwt to a local import to reduce baseline imports; docs are updated to describe the new plugin mode and environment variables.

^{Reviewed by Cursor Bugbot for commit 87def99. Bugbot is set up for automated code reviews on this repo. Configure here.}

[Copilot]

Pull request overview

This PR introduces WASM-compatibility improvements (primarily by lazy-loading heavyweight optional dependencies and moving environment-specific imports into call sites) and adds a new collocated Python UDF server implementation, including a new C-extension hot path to accelerate rowdat_1 decode → Python call → rowdat_1 encode.

Changes:

• Added a WIT interface definition and a WASM build helper script for external UDF component workflows.
• Refactored optional dependency handling (numpy/pandas/polars/pyarrow, IPython, JWT) to be more robust in constrained/WASM-like environments.
• Added a new collocated UDF server (socket + mmap protocol, thread/process modes, dynamic registration) and a C-extension accelerator entry point (call_function_accel).

Reviewed changes

Copilot reviewed 25 out of 25 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
wit/udf.wit	Defines the external UDF WIT interface and exported world.
singlestoredb/utils/_lazy_import.py	Adds cached lazy imports for heavy optional deps.
singlestoredb/utils/dtypes.py	Converts dtype maps to lazily-evaluated, cached getters.
singlestoredb/utils/results.py	Switches result formatting to lazy imports + cached type maps.
singlestoredb/utils/events.py	Broadens IPython import failure handling.
singlestoredb/converters.py	Uses lazy numpy import in vector converters.
singlestoredb/connection.py	Adjusts internal result-to-dict conversion to avoid importing pandas.
singlestoredb/mysql/connection.py	Adds WASM-friendly DEFAULT_USER detection (handles OSError).
singlestoredb/auth.py	Moves jwt import into call site.
singlestoredb/management/utils.py	Moves jwt import into call sites for WASM-friendliness.
singlestoredb/management/manager.py	Moves jwt import into is_jwt call site.
singlestoredb/functions/dtypes.py	Updates exports to use dtype-map getter functions.
singlestoredb/functions/ext/rowdat_1.py	Replaces eager dtype maps with lazy getter functions.
singlestoredb/functions/ext/json.py	Replaces eager dtype maps with lazy getter functions.
singlestoredb/functions/ext/collocated/*	Adds collocated server, protocol handling, registry, control signals, and WASM adapter.
singlestoredb/tests/test_connection.py	Makes pandas string dtype assertions version-tolerant.
resources/build_wasm.sh	Adds a build helper for wasm32-wasip2 wheels.
pyproject.toml	Adds python-udf-server CLI entry point.
accel.c	Adds call_function_accel C hot path and exports it from the extension module.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated 11 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 33 out of 33 changed files in this pull request and generated 6 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

These PyDict_SetItem calls pass a freshly-created PyLong key (PyLong_FromUnsignedLongLong(...)) without DECREF'ing it afterward. Since PyDict_SetItem increments references to both key and value, the temporary key object leaks here (and similarly for other inserted objects). Store the key in a local PyObject*, call PyDict_SetItem, then Py_DECREF(key), or use PyDict_SetItemString with a stable key strategy if possible.

[Copilot]

Handshake reads the function name using a single recvmsg(namelen, ...), but recvmsg is not guaranteed to return all requested bytes. If it returns a partial name, msg.decode() will succeed with truncated data and the protocol will desync. Treat short reads as an error (e.g., verify len(msg) == namelen and abort/close), or loop until the full name is received (while ensuring SCM_RIGHTS fds are only accepted once).

Suggested change

	if len(msg) != namelen:
	logger.warning(
	f'Expected function name of {namelen} bytes, got {len(msg)}',
	)
	return

[Copilot]

_bind_socket unconditionally unlinks any existing path. If the socket path is user-configurable (often under /tmp), this can be exploited to delete arbitrary files (e.g., via symlink or replacing the path with a regular file) when the server starts. Before unlinking, validate that the existing path is a Unix domain socket (and ideally owned by the current user), or use a safer strategy like binding to a uniquely generated path and refusing to overwrite non-sockets.

[Copilot]

create_function executes generated code into the 'main' module when it exists (which it typically will for any running process). This can pollute/overwrite the server's own main globals and makes dynamic registrations interfere with unrelated code in the process. Use a dedicated module namespace for dynamic UDFs (e.g., a unique module name per registration batch or a private module like 'singlestoredb.functions.ext.plugin._dynamic'), and avoid reusing the process' real main module.

Suggested change

	name = '__main__'
	compiled = compile(full_code, f'<{name}>', 'exec')
	if name in sys.modules:
	module = sys.modules[name]
	else:
	module = types.ModuleType(name)
	module.__file__ = f'<{name}>'
	sys.modules[name] = module
	name = (
	'singlestoredb.functions.ext.plugin._dynamic.'
	f'{func_name}_{os.urandom(8).hex()}'
	)
	compiled = compile(full_code, f'<{name}>', 'exec')
	module = types.ModuleType(name)
	module.__file__ = f'<{name}>'
	module.__package__ = 'singlestoredb.functions.ext.plugin._dynamic'
	sys.modules[name] = module

[Copilot]

The lazy import helpers only catch ImportError. In constrained environments (e.g., WASM/WASI, missing shared library loaders, binary wheels built for the wrong platform), importing optional deps often raises OSError (or other exceptions) instead. To preserve the intended "optional" behavior, broaden the exception handling (at least ImportError + OSError) and consider storing/logging the exception for debugging when a module is present but fails to import.

Suggested change

	from functools import lru_cache
	from typing import Any
	from typing import Optional
	@lru_cache(maxsize=None)
	def get_numpy() -> Optional[Any]:
	"""Return numpy module or None if not installed."""
	try:
	return importlib.import_module('numpy')
	except ImportError:
	return None
	@lru_cache(maxsize=None)
	def get_pandas() -> Optional[Any]:
	"""Return pandas module or None if not installed."""
	try:
	return importlib.import_module('pandas')
	except ImportError:
	return None
	@lru_cache(maxsize=None)
	def get_polars() -> Optional[Any]:
	"""Return polars module or None if not installed."""
	try:
	return importlib.import_module('polars')
	except ImportError:
	return None
	@lru_cache(maxsize=None)
	def get_pyarrow() -> Optional[Any]:
	"""Return pyarrow module or None if not installed."""
	try:
	return importlib.import_module('pyarrow')
	except ImportError:
	return None
	import logging
	from functools import lru_cache
	from typing import Any
	from typing import Optional
	logger = logging.getLogger(__name__)
	def _import_optional_module(name: str) -> Optional[Any]:
	"""Return imported optional module or None if it cannot be imported."""
	try:
	return importlib.import_module(name)
	except (ImportError, OSError) as exc:
	logger.debug('Optional dependency %r could not be imported: %s', name, exc)
	return None
	@lru_cache(maxsize=None)
	def get_numpy() -> Optional[Any]:
	"""Return numpy module or None if not installed."""
	return _import_optional_module('numpy')
	@lru_cache(maxsize=None)
	def get_pandas() -> Optional[Any]:
	"""Return pandas module or None if not installed."""
	return _import_optional_module('pandas')
	@lru_cache(maxsize=None)
	def get_polars() -> Optional[Any]:
	"""Return polars module or None if not installed."""
	return _import_optional_module('polars')
	@lru_cache(maxsize=None)
	def get_pyarrow() -> Optional[Any]:
	"""Return pyarrow module or None if not installed."""
	return _import_optional_module('pyarrow')

[Copilot]

_pack_time uses td.total_seconds() (float) to derive microseconds, which can lose precision for large timedeltas and round-trip incorrectly. Prefer integer arithmetic based on td.days / td.seconds / td.microseconds to compute total microseconds exactly (and keep sign handling consistent).

[cursor]

WASM build script exits on startup

Medium Severity

The script uses set -eou pipefail, which parses -o incorrectly and treats u as an option name. Bash exits before running the build steps, so resources/build_wasm.sh cannot execute its WASM packaging workflow.

 

^{Reviewed by Cursor Bugbot for commit 5f84e84. Configure here.}

[cursor]

Missing bounds check on decimal payload length

High Severity

The new DECIMAL decoding paths consume i64 bytes from data without validating remaining buffer size. Truncated or malformed rowdat_1 input can advance past the end of the byte buffer, causing out-of-bounds reads and potential crashes in load_rowdat_1 and load_rowdat_1_numpy.

Additional Locations (1)

• accel.c#L2673-L2681

 

^{Reviewed by Cursor Bugbot for commit 5f84e84. Configure here.}

[cursor]

Handshake assumes single recvmsg reads full name

Medium Severity

The handshake reads the function name with one conn.recvmsg(...) and treats any short read as protocol failure. On SOCK_STREAM, partial reads are valid, so legitimate requests can be rejected intermittently when the name payload arrives in multiple segments.

 

^{Reviewed by Cursor Bugbot for commit 5f84e84. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 6 total unresolved issues (including 5 from previous reviews).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit aeab5ff. Configure here.}