hey.. wanted to float this before a PR since the workflow here is agree on a plan first.
proposing Best-of-N jailbreak (Hughes et al. 2024, arXiv:2412.03556) as a converter + a single-turn attack. i checked the tree and theres no BoN anywhere... not in the converters, not in the attack strategies.
the idea is dead simple and thats the point.. take the objective, apply a stochastic text augmentation (intra-word scramble + random capitalization + ascii noise), send it, score it, and keep re-sampling fresh augmentations until one slips past the safety training or you hit the sample budget N. ASR climbs with N. black-box, text-only.
shape:
BestOfNConverter — the augmentation. one sigma knob, power-scaled the way the reference impl does (scramble/case at sigma**0.5, noise at sigma**3) so the default reproduces the paper's ~0.6/0.6/0.06 rates instead of one flat rate. fresh randomness per call, so reapplying gives you the N distinct samples.BestOfNAttack — thin PromptSendingAttack subclass, same pattern as FlipAttack/ManyShot. prepends the converter and maps n_samples onto the retry budget, so the existing retry loop IS the sample-until-break loop. no bespoke machinery.
paper is multimodal but this is text-only... thats the core attack and it composes cleanly with the converter stack. audio/image BoN could be a follow-up.
its written and green locally — converter + attack + tests (including an end-to-end one proving distinct augmentations actually reach the target across attempts) + bib entry + a converters-notebook example. passes ruff and the doc gate. holding the PR til you confirm you want it and the shape's right (own converter+attack vs folding into something existing). thoughts?
hey.. wanted to float this before a PR since the workflow here is agree on a plan first.
proposing Best-of-N jailbreak (Hughes et al. 2024, arXiv:2412.03556) as a converter + a single-turn attack. i checked the tree and theres no BoN anywhere... not in the converters, not in the attack strategies.
the idea is dead simple and thats the point.. take the objective, apply a stochastic text augmentation (intra-word scramble + random capitalization + ascii noise), send it, score it, and keep re-sampling fresh augmentations until one slips past the safety training or you hit the sample budget N. ASR climbs with N. black-box, text-only.
shape:
BestOfNConverter— the augmentation. onesigmaknob, power-scaled the way the reference impl does (scramble/case atsigma**0.5, noise atsigma**3) so the default reproduces the paper's ~0.6/0.6/0.06 rates instead of one flat rate. fresh randomness per call, so reapplying gives you the N distinct samples.BestOfNAttack— thinPromptSendingAttacksubclass, same pattern as FlipAttack/ManyShot. prepends the converter and mapsn_samplesonto the retry budget, so the existing retry loop IS the sample-until-break loop. no bespoke machinery.paper is multimodal but this is text-only... thats the core attack and it composes cleanly with the converter stack. audio/image BoN could be a follow-up.
its written and green locally — converter + attack + tests (including an end-to-end one proving distinct augmentations actually reach the target across attempts) + bib entry + a converters-notebook example. passes ruff and the doc gate. holding the PR til you confirm you want it and the shape's right (own converter+attack vs folding into something existing). thoughts?