fix(scrapy)!: serialize requests and HTTP cache as JSON instead of pickle#951
Merged
Conversation
vdusek
added
adhoc
github-actions
Bot
added
the
tested
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #951 +/- ##
==========================================
+ Coverage 87.08% 89.94% +2.86%
==========================================
Files 48 49 +1
Lines 2973 3053 +80
==========================================
+ Hits 2589 2746 +157
+ Misses 384 307 -77
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…ckle Scrapy requests stored in the Apify request queue and responses stored in the Scrapy HTTP cache were serialized with pickle. Those storages hold JSON, and pickle reconstructs a Python object graph from the stored bytes. Serialize them as JSON instead, via a single shared serializer (`_serialization.py`) used by both the request converter and the HTTP cache. Only the known binary fields (`body`, `headers`) are base64-encoded; pydantic models are dumped via `model_dump`. A non-JSON-serializable `meta`/`cb_kwargs` is reported and the request skipped rather than silently corrupted. A `_class` entry is only honored when already imported as a `scrapy.Request` subclass, and malformed or legacy (pickle-format) cache entries are treated as a cache miss so reads do not crash after the upgrade. BREAKING CHANGE: requests and HTTP cache entries are now stored as JSON, not pickle. Entries written by older versions are ignored (re-fetched), not read.
vdusek
force-pushed
the
fix/scrapy-json-serialization
branch
from
a24fd73 to
d73fcb4
Compare
Pijukatel
approved these changes
Jun 10, 2026
Follow-up to the pickle -> JSON switch, closing the data-loss paths and quality issues raised in review: - Resolve a request `_class` by importing it on demand (then validating it is a `scrapy.Request` subclass) so custom or lazily-imported subclasses survive an Actor migration instead of being dropped. - Keep the `retrieve_response` field reads inside the crash guard so a cache entry missing a key degrades to a miss rather than raising `KeyError`. - Reconstruct a request before marking it handled in the scheduler (still consuming unrecoverable entries so the queue cannot loop on them). - Encode a `str` request body symmetrically (as UTF-8 bytes) and serialize with `ensure_ascii=False` to avoid bloating non-ASCII text. - Store the serialized request as plain JSON, dropping the redundant outer base64 layer. - Name the offending value when `meta`/`cb_kwargs` is not JSON-serializable and document JSON's tuple/dict-key coercions and the non-UTF-8 header trade-off. Adds serializer tests, an HTTP-cache cleanup-sweep test, a cross-process request-reconstruction test, and documents `APIFY_HTTPCACHE_EXPIRATION_MAX_ITEMS`.
szaganek
approved these changes
Jun 10, 2026
vdusek
force-pushed
the
fix/scrapy-json-serialization
branch
from
d0a8c20 to
eabdcbd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scrapy requests (in the Apify request queue) and cached responses (in the Scrapy HTTP cache) were serialized with pickle, but both storages hold JSON. This switches them to JSON via a shared serializer (
apify/scrapy/_serialization.py).Request serialization:
body,headers) are base64-encoded; pydantic models such as Crawlee'sUserDataare dumped viamodel_dump.meta/cb_kwargsis logged and the request skipped, instead of being silently corrupted._classis honored only when it is already imported as ascrapy.Requestsubclass.HTTP cache, so existing caches survive the upgrade:
set_value(..., None), which under apify-client v3 stores JSONnullrather than deleting, leaking dead records.BREAKING CHANGE: requests and HTTP cache entries are now stored as JSON, not pickle. Entries written by older versions are ignored (re-fetched), not read.