Write PMTiles archives in binary mode#888
Open
Symmetricity wants to merge 1 commit into
Open
Conversation
PMTiles files contain compressed binary tile and directory payloads. On Windows, opening the output stream in text mode can translate newline bytes as they are written. That changes the number of bytes written to disk without changing the offsets and lengths recorded in the PMTiles header. Open the PMTiles stream in binary mode and write compressed strings with write(). This preserves the existing format on POSIX while keeping Windows output byte counts aligned with the header.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is AI generated.
Write PMTiles archives in binary mode
PMTiles archives contain compressed binary tile, directory, and metadata
payloads. On Windows, an
std::ofstreamopened in text mode can translatenewline bytes while writing. When that happens, the bytes written to disk no
longer match the offsets and lengths tilemaker has already recorded in the
PMTiles header.
Open the PMTiles output stream with
std::ios::binaryand use explicitwrite()calls for compressed string payloads. This keeps the on-disk bytecount aligned with the PMTiles header and matches how the existing code already
writes other binary data.
The change is intentionally small: it only touches PMTiles file writes in
src/pmtiles.cpp.Reproduce
On Windows, build tilemaker from current
master, then generate a PMTilesarchive:
Verify the archive:
Before this change, Windows-generated PMTiles archives can fail verification
with a header length mismatch:
After this change, the same Windows PMTiles outputs verify successfully.
Testing
git diff --checkcmake -S . -B build-pmtiles-binary-mode -DCMAKE_BUILD_TYPE=RelWithDebInfocmake --build build-pmtiles-binary-mode --target tilemaker -j2./build-pmtiles-binary-mode/tilemaker test/monaco.pbf --config=resources/config-openmaptiles.json --process=resources/process-openmaptiles.lua --output=/tmp/tilemaker-pmtiles-binary-mode.pmtiles --verbosepmtiles verify /tmp/tilemaker-pmtiles-binary-mode.pmtilesA stacked fork CI run also verified the Windows PMTiles outputs successfully
with
pmtiles verify. The overall run still failed because the new generatedtile verifier detected existing semantic differences between repeated tile
generation runs; that is separate from this PMTiles archive-structure fix.
Related Issues And PRs
I did not find an open upstream issue, PR, or discussion that directly reports
this Windows text-mode PMTiles archive corruption.
Related PMTiles context:
separate because it prevents Windows text-mode byte translation from changing
payload lengths after offsets have been calculated.
does not address clustering or directory layout.
gzip. This change does not alter compression behavior.