ETL: incremental company registry sync using change detection

[francescobianco]

Scenario

Teams building data warehouses over the Italian company registry (Registro Imprese) need to run incremental syncs — only fetching records that changed since the last run — rather than full reloads.

Current approach (inefficient)

// Full reload every night — very expensive in API credits
$page = 1;
do {
    $response = json_decode($client->get('https://company.openapi.com/IT-search', [
        'page' => $page++,
    ]), true);
    syncToWarehouse($response['items']);
} while (!empty($response['items']));

Better approach

Use a last_updated_at filter if the API supports it, or store a local hash of each record and compare on each run.

$lastRun = $store->get('last_sync_timestamp');
$response = json_decode($client->get('https://company.openapi.com/IT-search', [
    'updated_after' => $lastRun,
]), true);

Open questions

• Does the company search API support updated_after or equivalent filtering?
• Is there a webhook/event stream alternative to polling for changes?
• Could the SDK expose a since() query builder to make this pattern cleaner?

[sophymarine]

If your ETL pipeline depends on company registry deltas, one thing that becomes painful quickly is that every jurisdiction exposes updates differently (gazettes, change feeds, bulk snapshots, etc.). For cross-border diligence workflows we ended up normalising those differences behind a single MCP endpoint so agents/tools can request registry data without bespoke scrapers per country. OpenRegistry exposes 27 government registries with source-linked records and change tracking, which can simplify incremental sync logic for KYB / diligence pipelines. Might be relevant if this ETL is intended to support multi-jurisdiction company lookups: https://openregistry.sophymarine.com/mcp

[francescobianco]

@sophymarine Thanks for sharing this — the normalization layer across jurisdictions definitely looks interesting for KYB and diligence workflows.

One important aspect we always keep in mind is that scraping and consulting national company registry data is often subject to local regulations, licensing terms, and compliance constraints that vary by country. Because of that, we generally recommend relying on certified and officially authorized API/data providers to avoid legal or operational issues related to data usage and redistribution.

That said, we’re always open to evaluating interesting partnerships and integrations where there is strong compliance alignment and clear value for cross-jurisdiction workflows.