feat: snowflake export worker [CM-975] by mbani01 · Pull Request #3844 · CrowdDotDev/crowd.dev

mbani01 · 2026-02-16T16:59:19Z

This pull request introduces a new "snowflake connectors" service for exporting data to Snowflake and managing export jobs. It adds the necessary database schema, service configuration, Docker setup, and initial implementation for export activities and scheduling. The main themes are the creation of the Snowflake export infrastructure, integration with the job scheduler, and updates to the monorepo's dependency management.

Snowflake Export Infrastructure

Added a new database table integration."snowflakeExportJobs" to track Snowflake export jobs, including fields for platform, S3 path, timestamps, and error handling, along with relevant indexes.
Added a unique constraint on the s3_path column to prevent duplicate export entries.
Modified the export jobs table to add an exportStartedAt timestamp and a metrics JSONB column, while removing the totalRows and totalBytes columns to support more flexible export metrics. [1] [2]
Implemented the core export activity logic in exportActivity.ts, which performs the Snowflake export and writes job metadata. [1] [2]

Service and Job Scheduling

Added a new service app @crowd/snowflake-connectors with its own dependencies, scripts, and Docker configuration. [1] [2] [3] [4]
Integrated a scheduled cron job in the cron service to trigger the Snowflake export workflow daily, using Temporal for orchestration.
Centralized Temporal configuration export in the new service.

Monorepo Dependency and Lockfile Updates

Updated pnpm-lock.yaml to include the new snowflake_connectors app and its dependencies, and adjusted AWS SDK dependency resolutions for compatibility. [1] [2] [3] [4] [5] [6] [7] [8] [9]

Other Platform/Activity Updates

Added new Cvent-related activity types to the activityTypes table for event registration and attendance.

@joanagmaia you can help reviewing the export SQL and transformation

Note

Medium Risk
Introduces a new Temporal worker + polling consumer that creates integrations and processes exported data into activities, plus a new DB table and scheduled workflow; failures or misconfiguration could lead to missed/duplicated processing and new writes to core tables.

Overview
Adds a new snowflake_connectors service that schedules a daily Temporal workflow to export Snowflake data to S3 in Parquet batches, recording each batch in a new integration."snowflakeExportJobs" table (with locking/indices, metrics, and error tracking) for downstream processing.

The service also introduces a long-running consumer that claims pending export jobs, downloads/reads Parquet from S3, transforms rows into activities (initially for Cvent), resolves/creates matching integrations for segments (with Redis caching + distributed lock), and emits activity results for ingestion.

Separately, adds Cvent activity type definitions and scoring, introduces OrganizationSource/OrganizationAttributeSource = cvent and raises its attribute source priority, and adds Docker/compose + lockfile entries to build/run the new worker.

^{Written by Cursor Bugbot for commit 94adbd1. This will update automatically on new commits. Configure here.}

…ents

github-actions · 2026-02-20T17:08:40Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

services/apps/snowflake_connectors/src/core/integrationResolver.ts

services/apps/snowflake_connectors/src/integrations/cvent/transformer.ts

services/apps/snowflake_connectors/src/integrations/index.ts

services/apps/snowflake_connectors/src/main.ts

services/apps/snowflake_connectors/src/integrations/cvent/transformer.ts

backend/src/database/migrations/V1771344764__addSnowflakeExportTable.sql

services/apps/cron_service/src/jobs/snowflakeS3Export.job.ts

services/libs/integrations/src/integrations/cvent/types.ts

services/apps/snowflake_connectors/src/core/snowflakeExporter.ts

services/apps/snowflake_connectors/src/core/s3Consumer.ts

services/apps/snowflake_connectors/src/integrations/cvent/transformer.ts

services/apps/snowflake_connectors/src/index.ts

services/apps/snowflake_connectors/src/core/metadataStore.ts

services/apps/snowflake_connectors/src/core/integrationResolver.ts

joanagmaia · 2026-02-23T14:11:39Z

backend/src/database/migrations/V1771497876__addCventActivityTypes.sql

+('registered-event', 'cvent', false, false, 'Registered for a Cvent event', 'Registered for an event'),
+('attended-event', 'cvent', false, false, 'Attended a Cvent event', 'Attended an event');


Let's make these descriptions more aligned with Nirav's triggered description:

registered-event: User registers to an event.

attended-event: User attends an event.

joanagmaia · 2026-02-23T14:23:50Z

services/apps/snowflake_connectors/src/integrations/cvent/buildSourceQuery.ts

+  const orgAccounts = `
+  org_accounts AS (
+    SELECT account_id, website, domain_aliases
+    FROM analytics.bronze_fivetran_salesforce.accounts
+    WHERE website IS NOT NULL
+    UNION ALL
+    SELECT account_id, website, domain_aliases
+    FROM analytics.bronze_fivetran_salesforce_b2b.accounts
+    WHERE website IS NOT NULL
+  )`


I noticed that there is a domain column in the event_registration source. Can we check if the domain we get there is the same as we are getting from the slakesforce tables? If so, we could simply use the domain, and avoid this additional join

The domain is extracted from the email, and only accurate if a work email is used, but it's not the case for most records I checked

joanagmaia · 2026-02-23T14:28:10Z

services/apps/snowflake_connectors/src/integrations/cvent/transformer.ts

+  readonly platform = PlatformType.CVENT
+
+  transformRow(row: Record<string, unknown>): TransformedActivity | null {
+    const userName = (row.USERNAME as string | null)?.trim() || null


I believe it should be USER_NAME instead?

Good catch!

services/apps/snowflake_connectors/src/integrations/cvent/transformer.ts

joanagmaia · 2026-02-23T14:29:34Z

services/apps/snowflake_connectors/src/integrations/cvent/transformer.ts

+      member: {
+        displayName,
+        identities,
+        organizations: this.buildOrganizations(row),
+      },


Add missing attributes to both member and organizations -> In the document

joanagmaia · 2026-02-23T14:31:33Z

services/apps/snowflake_connectors/src/integrations/cvent/transformer.ts

+      identities.push({
+        platform: PlatformType.CVENT,
+        value: accountName,
+        type: OrganizationIdentityType.USERNAME,
+        verified: false,
+      })


atm we only support a fixed set of platforms for organization "usernames". It's linkedin, crunchbase, facebook and twitter.
Let's not add this identity from cvent

services/apps/snowflake_connectors/src/schedules/snowflakeS3Export.ts

services/apps/snowflake_connectors/src/integrations/cvent/buildSourceQuery.ts

themarolt · 2026-02-23T20:03:18Z

@mbani01 please check and resolve also cursor bot comments.

joanagmaia

LGTM

…gments & remove useless join

mbani01 · 2026-02-24T11:12:23Z

@mbani01 please check and resolve also cursor bot comments.

@themarolt I’ve addressed the relevant ones

cursor · 2026-02-24T11:16:24Z

services/apps/snowflake_connectors/src/integrations/cvent/transformer.ts

+    const displayName =
+      fullName ||
+      (firstName && lastName ? `${firstName} ${lastName}` : firstName || lastName) ||
+      userName


Member displayName can be null despite available LFID

Medium Severity

The displayName fallback chain uses fullName, firstName/lastName, and userName, but never lfUsername. The SQL query guarantees LFID IS NOT NULL (via the WHERE clause on the COALESCE), and lfUsername is extracted from the row, yet it's omitted from the displayName computation. When LFID originates from the joined mu.user_name or u.lf_username tables (rather than er.user_name), all event-registration name fields can be null, resulting in a null displayName for the member despite a valid username being available.

Additional Locations (1)

services/apps/snowflake_connectors/src/integrations/cvent/buildSourceQuery.ts#L13-L18

cursor · 2026-02-24T11:25:57Z

services/apps/snowflake_connectors/src/activities/exportActivity.ts

+    await exporter
+      .destroy()
+      .catch((err) => log.warn({ err }, 'Failed to close Snowflake connection'))
+  }


Export activity leaks Postgres connections

Medium Severity

executeExport creates a DB connection via getDbConnection(WRITE_DB_CONFIG()) but never closes it. Repeated exports can accumulate open connections/pools in the worker process and eventually exhaust Postgres connections.

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

services/apps/snowflake_connectors/src/activities/exportActivity.ts

mbani01 added 3 commits February 16, 2026 16:58

chore: init svc & boilerplate

b034261

chore: wip trigger export using snowflake & save metadata to postgres

c0b4feb

feat: trigger snowflake export using temporal & save result to postgres

5122bc0

mbani01 self-assigned this Feb 18, 2026

mbani01 added 14 commits February 19, 2026 14:22

feat: implement s3 consumer and activity processing

ac7630d

chore: snowflake connectors deps

178b54f

feat: add support for cvent in org sources

4e7c735

feat: implement incremental polling interval to not exhaust db

3f8562d

feat: update query to support incremental runs and newly created segm…

bbcb8b7

…ents

feat: save exportedAt to be used for incremental runs

baa9968

fix: reject duplicated export workflows to avoid data corruption

46780fa

fix: handle partial export failure

65ce6ab

fix: db connections leak

599c7a3

fix: potential data race when auto-creating integration

a124893

fix: backoff sleep on error

4060215

chore: update workflowIdReusePolicy

16245f4

feat: add metrics for observability

289f5b2

chore: lint & format

4355b36

mbani01 marked this pull request as ready for review February 20, 2026 17:08

mbani01 requested review from joanagmaia and themarolt February 20, 2026 17:08

mbani01 changed the title ~~feat: snowflake export worker~~ feat: snowflake export worker [CM-975] Feb 20, 2026

cursor bot reviewed Feb 20, 2026

View reviewed changes

fix: code review suggestions

64d1639

cursor bot reviewed Feb 20, 2026

View reviewed changes

services/apps/snowflake_connectors/src/main.ts Show resolved Hide resolved

services/apps/snowflake_connectors/src/integrations/cvent/transformer.ts Show resolved Hide resolved

themarolt requested changes Feb 23, 2026

View reviewed changes

backend/src/database/migrations/V1771344764__addSnowflakeExportTable.sql Show resolved Hide resolved

services/apps/cron_service/src/jobs/snowflakeS3Export.job.ts Outdated Show resolved Hide resolved

services/libs/integrations/src/integrations/cvent/types.ts Show resolved Hide resolved

mbani01 added 3 commits February 23, 2026 13:22

chore: replace cronjob with temporal schedule

4dd3457

chore: move cvent types to integrations lib

5dd68b1

chore: merge related migrations into one

19c2a10

mbani01 added 2 commits February 23, 2026 13:38

chore: fix lint

30e5836

fix: deps

34c37ee

cursor bot reviewed Feb 23, 2026

View reviewed changes

fix: ts errors

4daef9d

cursor bot reviewed Feb 23, 2026

View reviewed changes

services/apps/snowflake_connectors/src/core/metadataStore.ts Show resolved Hide resolved

chore: format

9f4d350

cursor bot reviewed Feb 23, 2026

View reviewed changes

services/apps/snowflake_connectors/src/core/integrationResolver.ts Show resolved Hide resolved

joanagmaia requested changes Feb 23, 2026

View reviewed changes

mbani01 added 3 commits February 23, 2026 14:50

fix: update activity description

732f8bd

fix: activity mapping and add attributes

623f738

fix: format

50e0e3e

cursor bot reviewed Feb 23, 2026

View reviewed changes

services/apps/snowflake_connectors/src/schedules/snowflakeS3Export.ts Show resolved Hide resolved

services/apps/snowflake_connectors/src/integrations/cvent/buildSourceQuery.ts Outdated Show resolved Hide resolved

mbani01 requested review from joanagmaia and themarolt February 23, 2026 16:58

joanagmaia approved these changes Feb 24, 2026

View reviewed changes

mbani01 added 4 commits February 24, 2026 10:45

fix: use _CROWD_DEV_SEGMENTS_UNION instead of kafka_ingest for cdp se…

ede34c0

…gments & remove useless join

fix: redis lock ttl

10b69aa

chore: explicit jsonb type cast

a4963b3

chore: update copy into command

d43fa56

cursor bot reviewed Feb 24, 2026

View reviewed changes

Merge branch 'main' into feat/snowflake_export_worker

d06f86a

cursor bot reviewed Feb 24, 2026

View reviewed changes

mbani01 added 4 commits February 24, 2026 13:51

fix: timestamp type

8ebdd68

chore: add debug logs

398af42

feat: distinguish skipped activities reasons

5073d7a

fix: revert header disabling from copy into to get columns name

94adbd1

cursor bot reviewed Feb 24, 2026

View reviewed changes

services/apps/snowflake_connectors/src/activities/exportActivity.ts Show resolved Hide resolved

themarolt approved these changes Feb 24, 2026

View reviewed changes

		('registered-event', 'cvent', false, false, 'Registered for a Cvent event', 'Registered for an event'),
		('attended-event', 'cvent', false, false, 'Attended a Cvent event', 'Attended an event');

Comments

Conversation

mbani01 commented Feb 16, 2026 • edited by cursor bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 20, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

joanagmaia Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

joanagmaia Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

mbani01 Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

joanagmaia Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

mbani01 Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

joanagmaia Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

joanagmaia Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

themarolt commented Feb 23, 2026

Uh oh!

joanagmaia left a comment

Choose a reason for hiding this comment

Uh oh!

mbani01 commented Feb 24, 2026

Uh oh!

cursor bot Feb 24, 2026

Choose a reason for hiding this comment

Member displayName can be null despite available LFID

Uh oh!

cursor bot Feb 24, 2026

Choose a reason for hiding this comment

Export activity leaks Postgres connections

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mbani01 commented Feb 16, 2026 •

edited by cursor bot

Loading

Member `displayName` can be null despite available LFID