FEAT Add PromptIntel remote dataset loader#1400
FEAT Add PromptIntel remote dataset loader#1400anandansundar wants to merge 5 commits intoAzure:mainfrom
Conversation
anandansundar
left a comment
There was a problem hiding this comment.
Initial dataset loader from prompt intel
romanlutz
left a comment
There was a problem hiding this comment.
Awesome contribution @anandansundar ! This looks very good. A few small things:
Did you run the integration test for datasets test_fetch_dataset_integration for this dataset?
Also make sure to rerun the notebook 1_loading_datasets.py since it will otherwise not show this new option.
Did you run pre-commit run --all-files?
Finally, you need to accept the CLA (see the CLA bot's message) 🙂
There was a problem hiding this comment.
Pull request overview
This PR adds a new _PromptIntelDataset class to fetch adversarial prompts from the PromptIntel API and load them into PyRIT's SeedDataset format. The implementation includes comprehensive filtering options (severity, category, search), pagination support, and a max_prompts limit. The PR includes 23 unit tests covering initialization, data transformation, pagination, API errors, and filter parameters.
Changes:
- Adds
_PromptIntelDatasetremote dataset loader with API integration for fetching adversarial prompts - Includes filtering by severity, category, and search terms with proper validation
- Adds comprehensive test suite covering all major functionality
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 10 comments.
| File | Description |
|---|---|
| pyrit/datasets/seed_datasets/remote/promptintel_dataset.py | New dataset loader class that fetches prompts from PromptIntel API with pagination and filtering support |
| pyrit/datasets/seed_datasets/remote/init.py | Registers the new _PromptIntelDataset class in the module exports |
| tests/unit/datasets/test_promptintel_dataset.py | Comprehensive test suite with 23 tests covering initialization, data transformation, pagination, errors, and filters |
…s, defer API key validation
…gory, rename method, remove dead code
Adds _PromptIntelDataset class that fetches prompts from the PromptIntel API and transforms them into PyRIT SeedDataset format (SeedPrompt + SeedObjective pairs grouped by prompt ID). Supports pagination, severity/category/search filters, and max_prompts limit. Includes 23 unit tests.
…s, defer API key validation
…gory, rename method, remove dead code
761a66b to
b75f34f
Compare
romanlutz
left a comment
There was a problem hiding this comment.
Looks great! Just want to try it out myself before merging.
Adds _PromptIntelDataset class that fetches prompts from the PromptIntel API and transforms them into PyRIT SeedDataset format (SeedPrompt + SeedObjective pairs grouped by prompt ID). Supports pagination, severity/category/search filters, and max_prompts limit. Includes 23 unit tests.
Description
Tests and Documentation