GH-47435: [Python][Parquet] Add direct key encryption/decryption API by smaheshwar-pltr · Pull Request #49667 · apache/arrow

smaheshwar-pltr · 2026-04-06T17:24:34Z

Rationale for this change

See #47435.

What changes are included in this PR?

Adds direct encryption / decryption Python API

Are these changes tested?

Yes, see PR.

Are there any user-facing changes?

Yes, new Python bindings.

GitHub Issue: [Parquet][Python] API to decrypt parquet file using one DEK and no metadata #47435

github-actions · 2026-04-06T17:24:58Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

smaheshwar-pltr · 2026-04-07T03:16:52Z

cc @ggershinsky, this is a requirement for PyIceberg encryption support, see apache/iceberg-python#3221

ggershinsky · 2026-04-07T15:03:17Z

sgtm, #47435 (comment)

github-actions · 2026-04-08T19:04:34Z

⚠️ GitHub issue #47435 has been automatically assigned in GitHub to PR creator.

github-actions · 2026-04-08T19:05:34Z

⚠️ GitHub issue #47435 has been automatically assigned in GitHub to PR creator.

smaheshwar-pltr · 2026-04-08T19:06:01Z

+    This bypasses the KMS-based :class:`CryptoFactory` API and directly
+    constructs decryption properties from a plaintext key. This is useful
+    when the caller manages key wrapping externally (e.g. via an
+    application-level envelope encryption scheme).
+
+    For most use cases, prefer the higher-level :class:`CryptoFactory`
+    with :class:`DecryptionConfiguration`, which handles envelope
+    encryption and key rotation automatically.


Per #47435 (comment), adding this documentation

I don't think it's true that the CryptoFactory handles key rotation automatically, that part of the comment should probably be removed. It does allow key rotation if you use external key material though.

smaheshwar-pltr · 2026-04-16T20:19:12Z

Not sure who to ping here for review, @AlenkaF @raulcd @rok maybe could I please have some eyes on this?

adamreeve

Thanks for implementing this @smaheshwar-pltr! I've left a few suggestions

adamreeve · 2026-04-17T04:01:17Z

+    This bypasses the KMS-based :class:`CryptoFactory` API and directly
+    constructs decryption properties from a plaintext key. This is useful
+    when the caller manages key wrapping externally (e.g. via an
+    application-level envelope encryption scheme).
+
+    For most use cases, prefer the higher-level :class:`CryptoFactory`
+    with :class:`DecryptionConfiguration`, which handles envelope
+    encryption and key rotation automatically.


I don't think it's true that the CryptoFactory handles key rotation automatically, that part of the comment should probably be removed. It does allow key rotation if you use external key material though.

smaheshwar-pltr · 2026-04-17T05:59:36Z

@adamreeve, thank you for the great review here 🙌 . I've reworked the docs now - they are consistent with my (inexperienced) understanding.

Would appreciate your eyes on this again, please let me know what you think!

adamreeve · 2026-04-19T23:54:15Z

@github-actions crossbow submit preview-docs

github-actions · 2026-04-19T23:56:33Z

Revision: 3e529d5

Submitted crossbow builds: ursacomputing/crossbow @ actions-4b28d0a71b

Task	Status
preview-docs

adamreeve

Thanks for the changes @smaheshwar-pltr, I just noticed one more minor issue that I missed first time around sorry.

adamreeve · 2026-04-21T00:29:06Z

@smaheshwar-pltr, can you please comment on #47435 so I can assign it to you? Eg. just a "take" comment.

Then I'll merge this, the CI failures are unrelated and also failing on main.

adamreeve · 2026-04-21T01:11:00Z

I've created #49821 as a follow-up issue to add support for per-column keys.

pitrou · 2026-04-21T16:30:03Z


+    cdef cppclass CFileDecryptionPropertiesBuilder\
+            " parquet::FileDecryptionProperties::Builder":
+        CFileDecryptionPropertiesBuilder() except +


If all these APIs can raise C++ exceptions, which kind of exceptions will be raised on the Python side?

pitrou · 2026-04-21T16:30:24Z

+        b"0123456789abcdef",
+        b"0123456789abcdef01234567",
+        b"0123456789abcdef0123456789abcdef",


Nit: reuse the KEY_ constants above?

pitrou · 2026-04-21T16:31:18Z

+        dec_props_no_aad = pe.create_decryption_properties(
+            footer_key=self.KEY_128,
+        )
+        with pytest.raises(IOError):


Can we also test something about the exception message?

Suggested change

with pytest.raises(IOError):

with pytest.raises(IOError, match='XXX...'):

pitrou · 2026-04-21T16:32:11Z

+            footer_key=self.KEY_128,
+            aad_prefix=b"wrong_prefix",
+        )
+        with pytest.raises(IOError):


pitrou · 2026-04-21T16:33:54Z

+        CFileDecryptionPropertiesBuilder* builder
+        shared_ptr[CFileDecryptionProperties] props
+
+    footer_key_bytes = tobytes(footer_key)


We shouldn't call tobytes as it will utf8-encode a str object.

pitrou · 2026-04-21T16:34:09Z

+        builder.footer_key(c_footer_key)
+
+        if aad_prefix is not None:
+            c_aad_prefix = tobytes(aad_prefix)


pitrou · 2026-04-21T16:34:42Z

+        )
+
+    c_footer_key = CSecureString(<c_string>footer_key_bytes)
+    builder = new CFileDecryptionPropertiesBuilder()


Why are we using new? We can create a plain value, I think.

pitrou · 2026-04-21T16:36:23Z

+        shared_ptr[CFileEncryptionProperties] props
+        ParquetCipher cipher
+
+    footer_key_bytes = tobytes(footer_key)


Same here and other instances below.

Support Parquet direct encryption

3a4fb52

github-actions Bot added Component: Python Component: Documentation awaiting review Awaiting review labels Apr 6, 2026

smaheshwar-pltr changed the title ~~Support Parquet direct encryption~~ [Python][Parquet] Add direct key encryption/decryption API Apr 7, 2026

Improvement: Comments

b0cfb95

smaheshwar-pltr mentioned this pull request Apr 7, 2026

POC: Encryption read support for REST catalog apache/iceberg-python#3221

Draft

smaheshwar-pltr changed the title ~~[Python][Parquet] Add direct key encryption/decryption API~~ GH-47435: [Python][Parquet] Add direct key encryption/decryption API Apr 8, 2026

smaheshwar-pltr commented Apr 8, 2026

View reviewed changes

smaheshwar-pltr marked this pull request as ready for review April 8, 2026 19:06

smaheshwar-pltr requested review from AlenkaF, raulcd and rok as code owners April 8, 2026 19:06

adamreeve reviewed Apr 17, 2026

View reviewed changes

github-actions Bot added awaiting review Awaiting review awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Apr 17, 2026

Sreesh Maheshwar added 2 commits April 16, 2026 22:40

Address review comments

40ef506

Merge branch 'main' into parquet-direct-key-encryption

76c8211

github-actions Bot added awaiting review Awaiting review and removed awaiting review Awaiting review awaiting committer review Awaiting committer review labels Apr 17, 2026

Nit: Unfix trailing whitespace

3e529d5

smaheshwar-pltr requested a review from adamreeve April 17, 2026 07:44

adamreeve reviewed Apr 20, 2026

View reviewed changes

Comment thread python/pyarrow/_parquet_encryption.pyx Outdated

Fixes

0959c7d

adamreeve approved these changes Apr 20, 2026

View reviewed changes

github-actions Bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Apr 20, 2026

adamreeve mentioned this pull request Apr 21, 2026

[Python][Parquet] Per-column keys for low-level encryption/decryption properties API #49821

Open

pitrou reviewed Apr 21, 2026

View reviewed changes

smaheshwar-pltr mentioned this pull request Apr 22, 2026

[Parquet][Python] API to decrypt parquet file using one DEK and no metadata #47435

Open

	with pytest.raises(IOError):
	with pytest.raises(IOError, match='XXX...'):

Conversation

smaheshwar-pltr commented Apr 6, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions Bot commented Apr 6, 2026

Uh oh!

smaheshwar-pltr commented Apr 7, 2026

Uh oh!

ggershinsky commented Apr 7, 2026

Uh oh!

github-actions Bot commented Apr 8, 2026

Uh oh!

github-actions Bot commented Apr 8, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

smaheshwar-pltr commented Apr 16, 2026

Uh oh!

adamreeve left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

smaheshwar-pltr commented Apr 17, 2026

Uh oh!

adamreeve commented Apr 19, 2026

Uh oh!

github-actions Bot commented Apr 19, 2026

Uh oh!

adamreeve left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

adamreeve commented Apr 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adamreeve commented Apr 21, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

smaheshwar-pltr commented Apr 6, 2026 •

edited by github-actions Bot

Loading

adamreeve commented Apr 21, 2026 •

edited

Loading