Skip to content

[DRAFT] FEAT Add Turkish Conversation Prompt-Injection dataset loader#2173

Draft
3nesdeniz wants to merge 1 commit into
microsoft:mainfrom
3nesdeniz:feat/turkish-conversation-prompt-injection-dataset
Draft

[DRAFT] FEAT Add Turkish Conversation Prompt-Injection dataset loader#2173
3nesdeniz wants to merge 1 commit into
microsoft:mainfrom
3nesdeniz:feat/turkish-conversation-prompt-injection-dataset

Conversation

@3nesdeniz

Copy link
Copy Markdown

Description

Closes #2171.

This PR adds a first-class PyRIT loader for the public Turkish Conversation Prompt-Injection Dataset (CC BY 4.0). The dataset contains 750 curated Turkish examples across 10 prompt-injection families, including 150 benign boundary cases paired with related attacks. It is intended to support both attack detection and over-refusal testing in Turkish LLM security workflows.

The new loader:

  • defaults to attack examples from all published splits for red-team use;
  • provides typed filters for attack, benign, or all labels;
  • provides typed train, validation, test, and combined-split selection;
  • supports filtering by one or more attack families;
  • preserves row-level provenance, including source context, pair ID, source type, split, label, and attack family;
  • maps attack families to harm_categories while leaving benign examples uncategorized;
  • supports the existing Hugging Face token and cache behavior; and
  • raises an explicit error when a valid filter produces an empty dataset.

The loader is registered with SeedDatasetProvider, documented in the dataset-loading guide, and linked to a bibliography entry. Attack examples contain adversarial instructions and should only be used in controlled, authorized security testing.

Tests and Documentation

  • Added 17 focused unit tests covering defaults, label filters, multi-family filtering, all split mappings, provenance preservation, empty-result behavior, enum validation, token handling, and discovery metadata.
  • uv run pytest tests/unit/datasets/test_turkish_conversation_prompt_injection_dataset.py tests/unit/datasets/test_seed_dataset_provider.py -q: 206 passed.
  • Full unit suite: 10,306 passed, 119 skipped.
  • Targeted pre-commit run for all changed files: passed, including Ruff, notebook Ruff, ty, notebook sanitation, documentation structure, and secret detection.
  • jupytext --test-strict for the updated dataset-loading notebook: passed.
  • The updated notebook was executed end to end against the live public dataset.
  • Targeted strict Jupyter Book site build for the updated loading guide and bibliography: passed.
  • Live dataset smoke validation confirmed 750 rows, the published 600/150 label distribution, 530/100/120 split distribution, all 10 attack families, and provider discovery.

AI assistance was used for implementation drafting and test planning. I reviewed the resulting code and documentation, validated them against the live dataset, and ran all checks listed above.

@3nesdeniz

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FEAT Add Turkish Conversation Prompt-Injection dataset loader

1 participant