fix(connectors): handle unrecognized types with explicit coverage and diagnostics by atharvalade · Pull Request #3196 · apache/iggy

atharvalade · 2026-04-29T16:35:36Z

Which issue does this PR close?

Rationale

Unrecognized Postgres types silently coerced to String, returning opaque InvalidRecord on failure with no column or type context.

What changed?

The default _ branch in extract_column_value attempted to read any unrecognized Postgres type as Option<String>, failing silently with Error::InvalidRecord and no indication of which column or type caused it.

Added explicit handlers for DATE, TIME, INTERVAL, OID, NAME, BPCHAR, and 13 array types (_BOOL, _INT2, _INT4, _INT8, _FLOAT4, _FLOAT8, _TEXT, _VARCHAR, _CHAR, _BPCHAR, _UUID, _JSON, _JSONB). The default branch now logs a warning with column name and type before attempting String fallback, and returns InvalidRecordValue with a diagnostic message on failure.

Local Execution

Passed
Pre-commit hooks ran

AI Usage

Opus 4.6
Scaffolding, Writing comments, Writing PR Description
Verified via cargo check, clippy, fmt, and full test suite (24/24 pass)
Yes, all code can be explained

codecov · 2026-04-29T17:29:34Z

Codecov Report

❌ Patch coverage is 25.71429% with 260 lines in your changes missing coverage. Please review.
✅ Project coverage is 18.36%. Comparing base (2337883) to head (f2cec44).
⚠️ Report is 3 commits behind head on master.

Files with missing lines	Patch %	Lines
core/connectors/sources/postgres_source/src/lib.rs	25.71%	258 Missing and 2 partials ⚠️

❌ Your patch check has failed because the patch coverage (25.71%) is below the target coverage (50.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@              Coverage Diff              @@
##             master    #3196       +/-   ##
=============================================
- Coverage     74.18%   18.36%   -55.83%     
  Complexity      943      943               
=============================================
  Files          1200     1198        -2     
  Lines        109645    95245    -14400     
  Branches      86534    72152    -14382     
=============================================
- Hits          81345    17495    -63850     
- Misses        25553    77331    +51778     
+ Partials       2747      419     -2328

Components	Coverage Δ
Rust Core	`1.01% <25.71%> (-74.42%)`	⬇️
Java SDK	`58.44% <ø> (ø)`
C# SDK	`69.15% <ø> (-0.29%)`	⬇️
Python SDK	`81.43% <ø> (ø)`
Node SDK	`91.53% <ø> (+0.09%)`	⬆️
Go SDK	`39.91% <ø> (ø)`

Files with missing lines	Coverage Δ
core/connectors/sources/postgres_source/src/lib.rs	`36.82% <25.71%> (-31.09%)`	⬇️

... and 686 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions · 2026-05-07T00:53:52Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs.

If you need a review, please ensure CI is green and the PR is rebased on the latest master. Don't hesitate to ping the maintainers - either @core on Discord or by mentioning them directly here on the PR.

Thank you for your contribution!

hubcio · 2026-05-14T12:11:08Z

/ready

slbotbm

/author

slbotbm

Separately, the current code also does not handle the following types:

TIMETZ
TIMETZ[]
DATE[]
TIME[]
TIMESTAMP[]
TIMESTAMPTZ[]
INTERVAL[]

slbotbm · 2026-05-14T17:03:55Z

/author

slbotbm · 2026-05-16T08:08:23Z

/author

atharvalade · 2026-05-17T07:00:27Z

/ready

slbotbm · 2026-05-18T13:13:48Z

+            _ => {
+                let column_name = column.name();
+                warn!(
+                    "Column '{column_name}' has unrecognized Postgres type '{type_name}', \
+                     attempting String conversion"
+                );
+                let value: Option<String> = row.try_get(column_index).map_err(|e| {
+                    error!("Failed to read column '{column_name}' (type '{type_name}'): {e}");
+                    Error::InvalidRecordValue(format!(
+                        "column '{column_name}' has unsupported Postgres type '{type_name}'"
+                    ))
+                })?;


String is only compatible with text-like types, not types like INET, CIDR, MACADDR, MONEY, BIT/VARBIT. This fallback will fail hard for those cases.

the default branch now uses try_get_raw() + as_str() which reads the raw text representation directly from the wire without sqlx type-checking, so it works for any type Postgres can render as text.

slbotbm · 2026-05-18T13:17:43Z

+            "OID" => {
+                let value: Option<i32> = row
+                    .try_get(column_index)
+                    .map_err(|_| Error::InvalidRecord)?;
+                Ok(value
+                    .map(|v| serde_json::Value::from(v as u32 as u64))
+                    .unwrap_or(serde_json::Value::Null))
+            }


For OID and OID[], could you write a temporary test and check whether this works? From what I understand, sqlx may reject this since try_get checks compatibility before decoding, and fails on mismatched SQL/Rust types. sqlx::postgres::types::Oid being the native type for OID, i32 would still be rejected.

now decoding via sqlx::postgres::types::Oid which is the native sqlx type for OID (wraps u32, implements Decode and PgHasArrayType); OID[] also uses Vec<Option<Oid>> separately from INT4[]

slbotbm · 2026-05-18T13:23:04Z

/author

…iagnostics Closes apache#3175

Closes apache#3175

atharvalade · 2026-05-22T06:04:18Z

/ready

hubcio · 2026-05-22T11:44:43Z

why did u do this many changes in Cargo.lock? you're only adding chrono dep to postgres_source.

atharvalade changed the title ~~fix(postgres): handle unrecognized types with explicit coverage and diagnostics~~ fix(connectors): handle unrecognized types with explicit coverage and diagnostics Apr 29, 2026

lukaszzborek mentioned this pull request Apr 29, 2026

fix(connectors): unify extract_column_value into single free function #3199

Open

github-actions Bot added S-stale Inactive issue or pull request and removed S-stale Inactive issue or pull request labels May 7, 2026

github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label May 14, 2026

slbotbm reviewed May 14, 2026

View reviewed changes

Comment thread core/connectors/sources/postgres_source/src/lib.rs

Comment thread core/connectors/sources/postgres_source/src/lib.rs Outdated

slbotbm suggested changes May 14, 2026

View reviewed changes

slbotbm reviewed May 14, 2026

View reviewed changes

Comment thread core/connectors/sources/postgres_source/src/lib.rs Outdated

Comment thread core/connectors/sources/postgres_source/src/lib.rs Outdated

Comment thread core/connectors/sources/postgres_source/src/lib.rs

slbotbm suggested changes May 14, 2026

View reviewed changes

github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels May 16, 2026

atharvalade force-pushed the fix/postgres-unrecognized-types-coerce-string branch from 73fcce0 to bb6236a Compare May 17, 2026 06:44

github-actions Bot added S-waiting-on-review PR is waiting on a reviewer and removed S-waiting-on-author PR is waiting on author response labels May 17, 2026

slbotbm suggested changes May 18, 2026

View reviewed changes

github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels May 18, 2026

atharvalade added 4 commits May 21, 2026 22:59

fix(postgres): handle unrecognized types with explicit coverage and d…

5f96a93

…iagnostics Closes apache#3175

trigger CI

ee0e187

fix(postgres): address review feedback on type handling

259d75c

Closes apache#3175

fix(postgres): use Oid type and raw text fallback for unknown types

f2cec44

Closes apache#3175

atharvalade force-pushed the fix/postgres-unrecognized-types-coerce-string branch from 97d4024 to f2cec44 Compare May 22, 2026 06:01

github-actions Bot added S-waiting-on-review PR is waiting on a reviewer and removed S-waiting-on-author PR is waiting on author response labels May 22, 2026

hubcio requested changes May 22, 2026

View reviewed changes

github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels May 22, 2026

Conversation

atharvalade commented Apr 29, 2026

Which issue does this PR close?

Rationale

What changed?

Local Execution

AI Usage

Uh oh!

codecov Bot commented Apr 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions Bot commented May 7, 2026

Uh oh!

hubcio commented May 14, 2026

Uh oh!

slbotbm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

slbotbm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

slbotbm commented May 14, 2026

Uh oh!

slbotbm commented May 16, 2026

Uh oh!

atharvalade commented May 17, 2026

Uh oh!

slbotbm May 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

atharvalade May 22, 2026

Choose a reason for hiding this comment

Uh oh!

slbotbm May 18, 2026

Choose a reason for hiding this comment

Uh oh!

atharvalade May 22, 2026

Choose a reason for hiding this comment

Uh oh!

slbotbm commented May 18, 2026

Uh oh!

atharvalade commented May 22, 2026

Uh oh!

hubcio May 22, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov Bot commented Apr 29, 2026 •

edited

Loading

slbotbm May 18, 2026 •

edited

Loading