Skip to content

[bug](cloud restore) separate delete bitmap in prepare restore job to fix value length exceed limit error#61545

Open
xy720 wants to merge 2 commits intoapache:masterfrom
xy720:refactor-cloud-restore
Open

[bug](cloud restore) separate delete bitmap in prepare restore job to fix value length exceed limit error#61545
xy720 wants to merge 2 commits intoapache:masterfrom
xy720:refactor-cloud-restore

Conversation

@xy720
Copy link
Member

@xy720 xy720 commented Mar 20, 2026

What problem does this PR solve?

Issue Number: close #xxx

put delete bitmap key in prepare restore job rather than in commit restore job.

To avoid value length exceed limit happen when put RestoreJobCloudPB value.

The delete bitmap key will be recycle when recycling tablets.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Mar 20, 2026

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@xy720
Copy link
Member Author

xy720 commented Mar 20, 2026

run buildall

@xy720
Copy link
Member Author

xy720 commented Mar 20, 2026

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 26670 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cb5fd512cc984eef49743574adf78b438cd5abf0, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17677	4487	4279	4279
q2	q3	10652	772	526	526
q4	4675	350	252	252
q5	7551	1214	1027	1027
q6	178	176	144	144
q7	789	838	696	696
q8	9301	1478	1295	1295
q9	4852	4782	4673	4673
q10	6321	1921	1643	1643
q11	460	254	240	240
q12	752	583	466	466
q13	18053	2921	2169	2169
q14	233	227	210	210
q15	q16	742	751	678	678
q17	747	863	408	408
q18	5972	5486	5246	5246
q19	1215	967	602	602
q20	517	476	372	372
q21	4499	1843	1451	1451
q22	567	382	293	293
Total cold run time: 95753 ms
Total hot run time: 26670 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4727	4635	4645	4635
q2	q3	3976	4359	3833	3833
q4	890	1233	781	781
q5	4071	4399	4353	4353
q6	180	179	138	138
q7	1766	1689	1539	1539
q8	2508	2699	2629	2629
q9	7620	7438	7380	7380
q10	3789	4025	3617	3617
q11	525	464	440	440
q12	496	602	461	461
q13	2771	3334	2755	2755
q14	298	311	292	292
q15	q16	731	797	739	739
q17	1216	1312	1317	1312
q18	7385	6902	6858	6858
q19	867	871	894	871
q20	2252	2167	2002	2002
q21	3947	3525	3289	3289
q22	470	444	370	370
Total cold run time: 50485 ms
Total hot run time: 48294 ms

@hello-stephen
Copy link
Contributor

Cloud UT Coverage Report

Increment line coverage 61.87% (86/139) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 78.63% (1796/2284)
Line Coverage 64.39% (32280/50132)
Region Coverage 65.28% (16164/24760)
Branch Coverage 55.67% (8605/15456)

@doris-robot
Copy link

TPC-DS: Total hot run time: 169685 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cb5fd512cc984eef49743574adf78b438cd5abf0, data reload: false

query5	4340	661	512	512
query6	339	231	214	214
query7	4213	496	268	268
query8	343	262	236	236
query9	8703	2712	2688	2688
query10	494	399	343	343
query11	7073	5196	4980	4980
query12	186	131	127	127
query13	1275	455	358	358
query14	5750	3844	3513	3513
query14_1	2828	2906	2879	2879
query15	214	198	177	177
query16	1005	464	460	460
query17	910	745	648	648
query18	2486	468	363	363
query19	222	212	189	189
query20	142	128	132	128
query21	217	135	114	114
query22	13428	14155	15045	14155
query23	16499	16028	15571	15571
query23_1	15888	15566	15885	15566
query24	7128	1638	1233	1233
query24_1	1221	1243	1239	1239
query25	608	462	396	396
query26	1237	255	149	149
query27	2793	489	304	304
query28	4439	1807	1835	1807
query29	853	565	471	471
query30	303	235	200	200
query31	1001	976	889	889
query32	87	69	72	69
query33	515	333	278	278
query34	864	864	521	521
query35	653	670	585	585
query36	1105	1142	953	953
query37	132	94	83	83
query38	3019	2991	2930	2930
query39	865	843	835	835
query39_1	815	806	800	800
query40	235	158	137	137
query41	68	61	60	60
query42	272	267	269	267
query43	243	246	228	228
query44	
query45	199	190	184	184
query46	878	996	617	617
query47	2124	2139	2049	2049
query48	317	322	231	231
query49	625	492	392	392
query50	680	280	220	220
query51	4150	4136	4010	4010
query52	271	277	266	266
query53	293	346	284	284
query54	300	273	268	268
query55	91	88	85	85
query56	311	328	312	312
query57	1948	1799	1769	1769
query58	288	274	279	274
query59	2774	2948	2745	2745
query60	352	347	327	327
query61	160	149	155	149
query62	590	588	543	543
query63	311	273	275	273
query64	5021	1281	985	985
query65	
query66	1472	460	353	353
query67	24676	24662	24579	24579
query68	
query69	407	327	296	296
query70	961	973	919	919
query71	342	312	309	309
query72	2788	2706	2549	2549
query73	548	559	312	312
query74	9681	9661	9418	9418
query75	2983	2785	2498	2498
query76	2274	1033	672	672
query77	362	432	309	309
query78	11468	11334	10771	10771
query79	3049	791	580	580
query80	1723	661	550	550
query81	590	276	239	239
query82	1019	152	116	116
query83	340	269	246	246
query84	305	117	105	105
query85	930	474	448	448
query86	501	308	288	288
query87	3238	3159	2994	2994
query88	3539	2649	2625	2625
query89	417	368	347	347
query90	2070	182	175	175
query91	169	163	133	133
query92	87	79	73	73
query93	1489	830	489	489
query94	648	327	296	296
query95	590	338	320	320
query96	658	520	232	232
query97	2529	2477	2375	2375
query98	248	223	224	223
query99	1004	965	917	917
Total cold run time: 253922 ms
Total hot run time: 169685 ms

@xy720 xy720 changed the title [bug](cloud restore) separate delete bitmap in prepare restore statge to fix value length exceed limit error [bug](cloud restore) separate delete bitmap in prepare restore job to fix value length exceed limit error Mar 20, 2026
@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.71% (19785/37535)
Line Coverage 36.25% (184864/509908)
Region Coverage 32.49% (143030/440292)
Branch Coverage 33.67% (62555/185811)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.60% (26304/36738)
Line Coverage 54.47% (276838/508194)
Region Coverage 51.82% (230268/444338)
Branch Coverage 53.13% (98980/186293)

@xy720
Copy link
Member Author

xy720 commented Mar 20, 2026

run vault_p0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants