repository: make check and delete work at N=1 with sha256 pack ids#9783
Open
mr-raj12 wants to merge 1 commit into
Open
repository: make check and delete work at N=1 with sha256 pack ids#9783mr-raj12 wants to merge 1 commit into
mr-raj12 wants to merge 1 commit into
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #9783 +/- ##
==========================================
- Coverage 84.88% 84.84% -0.05%
==========================================
Files 92 92
Lines 15172 15166 -6
Branches 2275 2281 +6
==========================================
- Hits 12879 12867 -12
- Misses 1589 1594 +5
- Partials 704 705 +1 ☔ View full report in Codecov by Harness. |
ThomasWaldmann
requested changes
Jun 16, 2026
28c817b to
f2ca1ff
Compare
A pack is named sha256(pack_bytes), not chunk_id. delete becomes a no-op (real removal is store_delete at the pack level); list iterates the chunk index; build_chunkindex_from_repo reads pack headers for real chunk_ids and gains init_flags so compact computes usage; tests drop a chunk's pack.
f2ca1ff to
4121509
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
borg check,deleteandcompactall assumed that a pack file's name is the chunk_id it holds. That is true only at N=1 while pack ids equal chunk ids. WithBORG_TESTONLY_SHA256_PACK_ID=1(and later at N>1) a pack is namedsha256(pack_bytes), so the file name is the pack_id, not the chunk_id.The object header already carries the chunk_id, meta size and data size (
repoobj.py), so the chunk index can be rebuilt from the headers (real chunk_ids) instead of the file names, and removal can find the pack through the index.Changes:
repoobj.py: addRepoObj.iter_object_headers(), which walks a pack's object headers and yields(chunk_id, obj_offset, obj_size). Works for one object per pack (N=1) and for several (N>1).repository.py:delete()becomes a logged no-op. A pack may hold several objects, so we can not remove one object by dropping its pack without losing the others; real removal goes throughstore_deleteat the pack level.list()iterates the chunk index and yields chunk_ids, instead of listingpacks/(which would yield pack_ids).cache.py:build_chunkindex_from_repo()rebuilds the index fromstore_list+iter_object_headers(real chunk_id/offset/size) rather than callingRepository.list(), which would recurse into the very index it is building. Adds aninit_flagsargument socompactcan start the chunks as unused and compute usage itself.compact_cmd.py:--statsbuilds the index from a real header scan instead of apack_id == chunk_idfake; the delete loop drops each unused chunk's pack viastore_delete.archive.py/debug_cmd.py:check --repairanddebug delete-objresolve the pack via the chunk index and thenstore_deleteit;debug delete-objmaps a missing pack (stale index entry) back to "not found".delete_chunkhelper drops a chunk's pack at the store level (used by the check / extract / mount damage tests), sinceRepository.deleteno longer removes anything;test_empty_repositorydrops packs through the store; test objects get a unique chunk_id so identical payloads do not collapse into one pack under sha256.Verified with the local test subset, with and without
BORG_TESTONLY_SHA256_PACK_ID=1: the repository- and archive-level missing-chunk checks pass under the switch and nothing regresses without it. Archive--verify-datacorruption detection under sha256 pack ids (a fewcheck --verify-datatests) is left for a separate change.refs #8572
Checklist
master