Skip to content

Commit db493ef

Browse files
BazookaMusicCopilot
andcommitted
Python: port prompt injection queries (system + user) from JS PR #21953
Replace the experimental py/prompt-injection query with two queries mirroring the JavaScript split: - py/system-prompt-injection (system prompt / tool description / developer prompt) - py/user-prompt-injection (user-role prompt) Supports OpenAI (+Agents), Anthropic, Google GenAI, LangChain and OpenRouter via MaD models plus role-filtered framework sinks that MaD cannot express. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 330e904 commit db493ef

50 files changed

Lines changed: 1491 additions & 419 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

python/ql/integration-tests/query-suite/not_included_in_qls.expected

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -87,7 +87,8 @@ ql/python/ql/src/experimental/Security/CWE-079/EmailXss.ql
8787
ql/python/ql/src/experimental/Security/CWE-091/XsltInjection.ql
8888
ql/python/ql/src/experimental/Security/CWE-094/Js2Py.ql
8989
ql/python/ql/src/experimental/Security/CWE-1236/CsvInjection.ql
90-
ql/python/ql/src/experimental/Security/CWE-1427/PromptInjection.ql
90+
ql/python/ql/src/experimental/Security/CWE-1427/SystemPromptInjection.ql
91+
ql/python/ql/src/experimental/Security/CWE-1427/UserPromptInjection.ql
9192
ql/python/ql/src/experimental/Security/CWE-176/UnicodeBypassValidation.ql
9293
ql/python/ql/src/experimental/Security/CWE-208/TimingAttackAgainstHash/PossibleTimingAttackAgainstHash.ql
9394
ql/python/ql/src/experimental/Security/CWE-208/TimingAttackAgainstHash/TimingAttackAgainstHash.ql
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
---
2+
category: minorAnalysis
3+
---
4+
* Added prompt-injection sink models (`system-prompt-injection` and `user-prompt-injection` kinds) for the `openai`, `agents`, `anthropic`, `google-genai`, `openrouter` and `langchain` frameworks.

python/ql/lib/semmle/python/frameworks/agent.model.yml

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,4 +3,10 @@ extensions:
33
pack: codeql/python-all
44
extensible: sinkModel
55
data:
6-
- ['agents', 'Member[Agent].Argument[instructions:]', 'prompt-injection']
6+
# Agent instructions, handoff descriptions and tool descriptions are system-level prompts
7+
- ['agents', 'Member[Agent].Argument[instructions:]', 'system-prompt-injection']
8+
- ['agents', 'Member[Agent].Argument[handoff_description:]', 'system-prompt-injection']
9+
- ['agents', 'Member[FunctionTool].Argument[description:]', 'system-prompt-injection']
10+
# The input passed to a run is user-level content
11+
- ['agents', 'Member[Runner].Member[run,run_sync,run_streamed].Argument[1]', 'user-prompt-injection']
12+
- ['agents', 'Member[Runner].Member[run,run_sync,run_streamed].Argument[input:]', 'user-prompt-injection']

python/ql/lib/semmle/python/frameworks/anthropic.model.yml

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -3,12 +3,11 @@ extensions:
33
pack: codeql/python-all
44
extensible: sinkModel
55
data:
6-
- ['Anthropic', 'Member[messages].Member[create].Argument[system:]', 'prompt-injection']
7-
- ['Anthropic', 'Member[messages].Member[stream].Argument[system:]', 'prompt-injection']
8-
- ['Anthropic', 'Member[beta].Member[messages].Member[create].Argument[system:]', 'prompt-injection']
9-
- ['Anthropic', 'Member[messages].Member[create].Argument[messages:].ListElement.DictionaryElement[content]', 'prompt-injection']
10-
- ['Anthropic', 'Member[messages].Member[stream].Argument[messages:].ListElement.DictionaryElement[content]', 'prompt-injection']
11-
- ['Anthropic', 'Member[beta].Member[messages].Member[create].Argument[messages:].ListElement.DictionaryElement[content]', 'prompt-injection']
6+
# The `system` field is a system-level prompt
7+
- ['Anthropic', 'Member[messages].Member[create,stream].Argument[system:]', 'system-prompt-injection']
8+
- ['Anthropic', 'Member[messages].Member[create,stream].Argument[system:].ListElement.DictionaryElement[text]', 'system-prompt-injection']
9+
- ['Anthropic', 'Member[beta].Member[messages].Member[create,stream].Argument[system:]', 'system-prompt-injection']
10+
- ['Anthropic', 'Member[beta].Member[messages].Member[create,stream].Argument[system:].ListElement.DictionaryElement[text]', 'system-prompt-injection']
1211

1312
- addsTo:
1413
pack: codeql/python-all
Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
extensions:
2+
- addsTo:
3+
pack: codeql/python-all
4+
extensible: sinkModel
5+
data:
6+
# `system_instruction` on the generation config is a system-level prompt
7+
- ['google.genai', 'Member[types].Member[GenerateContentConfig].Argument[system_instruction:]', 'system-prompt-injection']
8+
# User-level content
9+
- ['GoogleGenAI', 'Member[models].Member[generate_content,generate_content_stream].Argument[contents:]', 'user-prompt-injection']
10+
- ['GoogleGenAI', 'Member[models].Member[generate_images,generate_videos].Argument[prompt:]', 'user-prompt-injection']
11+
- ['GoogleGenAI', 'Member[chats].Member[create].ReturnValue.Member[send_message,send_message_stream].Argument[0]', 'user-prompt-injection']
12+
- ['GoogleGenAI', 'Member[chats].Member[create].ReturnValue.Member[send_message,send_message_stream].Argument[message:]', 'user-prompt-injection']
13+
14+
- addsTo:
15+
pack: codeql/python-all
16+
extensible: typeModel
17+
data:
18+
- ['GoogleGenAI', 'google.genai', 'Member[Client].ReturnValue']
Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
extensions:
2+
- addsTo:
3+
pack: codeql/python-all
4+
extensible: sinkModel
5+
data:
6+
# Message constructors. The first positional argument or the `content` keyword
7+
# carries the message text.
8+
- ['langchain_core.messages', 'Member[SystemMessage].Argument[0]', 'system-prompt-injection']
9+
- ['langchain_core.messages', 'Member[SystemMessage].Argument[content:]', 'system-prompt-injection']
10+
- ['langchain.schema', 'Member[SystemMessage].Argument[0]', 'system-prompt-injection']
11+
- ['langchain.schema', 'Member[SystemMessage].Argument[content:]', 'system-prompt-injection']
12+
- ['langchain_core.messages', 'Member[HumanMessage].Argument[0]', 'user-prompt-injection']
13+
- ['langchain_core.messages', 'Member[HumanMessage].Argument[content:]', 'user-prompt-injection']
14+
- ['langchain.schema', 'Member[HumanMessage].Argument[0]', 'user-prompt-injection']
15+
- ['langchain.schema', 'Member[HumanMessage].Argument[content:]', 'user-prompt-injection']
16+
# Invoking a chat model with user input.
17+
- ['LangChainChatModel', 'Member[invoke,stream,predict,call].Argument[0]', 'user-prompt-injection']
18+
- ['LangChainChatModel', 'Member[batch].Argument[0].ListElement', 'user-prompt-injection']
19+
20+
- addsTo:
21+
pack: codeql/python-all
22+
extensible: typeModel
23+
data:
24+
- ['LangChainChatModel', 'langchain_openai', 'Member[ChatOpenAI,AzureChatOpenAI].ReturnValue']
25+
- ['LangChainChatModel', 'langchain_anthropic', 'Member[ChatAnthropic].ReturnValue']
26+
- ['LangChainChatModel', 'langchain_google_genai', 'Member[ChatGoogleGenerativeAI].ReturnValue']
27+
- ['LangChainChatModel', 'langchain_mistralai', 'Member[ChatMistralAI].ReturnValue']
28+
- ['LangChainChatModel', 'langchain_groq', 'Member[ChatGroq].ReturnValue']
29+
- ['LangChainChatModel', 'langchain_cohere', 'Member[ChatCohere].ReturnValue']
30+
- ['LangChainChatModel', 'langchain_ollama', 'Member[ChatOllama].ReturnValue']
31+
- ['LangChainChatModel', 'langchain_aws', 'Member[ChatBedrock,ChatBedrockConverse].ReturnValue']

python/ql/lib/semmle/python/frameworks/openai.model.yml

Lines changed: 11 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3,10 +3,17 @@ extensions:
33
pack: codeql/python-all
44
extensible: sinkModel
55
data:
6-
- ['OpenAI', 'Member[beta].Member[assistants].Member[create].Argument[instructions:]', 'prompt-injection']
7-
- ['OpenAI', 'Member[chat].Member[completions].Member[create].Argument[messages:].ListElement.DictionaryElement[content]', 'prompt-injection']
8-
- ['OpenAI', 'Member[responses].Member[create].Argument[instructions:]', 'prompt-injection']
9-
- ['OpenAI', 'Member[responses].Member[create].Argument[input:]', 'prompt-injection']
6+
# System-level prompts and instructions
7+
- ['OpenAI', 'Member[responses].Member[create].Argument[instructions:]', 'system-prompt-injection']
8+
- ['OpenAI', 'Member[beta].Member[assistants].Member[create].Argument[instructions:]', 'system-prompt-injection']
9+
- ['OpenAI', 'Member[beta].Member[assistants].Member[update].Argument[instructions:]', 'system-prompt-injection']
10+
- ['OpenAI', 'Member[beta].Member[threads].Member[runs].Member[create].Argument[instructions:]', 'system-prompt-injection']
11+
- ['OpenAI', 'Member[beta].Member[threads].Member[runs].Member[create].Argument[additional_instructions:]', 'system-prompt-injection']
12+
# User-level prompts
13+
- ['OpenAI', 'Member[responses].Member[create].Argument[input:]', 'user-prompt-injection']
14+
- ['OpenAI', 'Member[completions].Member[create].Argument[prompt:]', 'user-prompt-injection']
15+
- ['OpenAI', 'Member[images].Member[generate,edit].Argument[prompt:]', 'user-prompt-injection']
16+
- ['OpenAI', 'Member[audio].Member[transcriptions,translations].Member[create].Argument[prompt:]', 'user-prompt-injection']
1017

1118
- addsTo:
1219
pack: codeql/python-all
Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
extensions:
2+
- addsTo:
3+
pack: codeql/python-all
4+
extensible: sinkModel
5+
data:
6+
# Embeddings input is user-level content
7+
- ['OpenRouter', 'Member[embeddings].Member[create].Argument[input:]', 'user-prompt-injection']
8+
9+
- addsTo:
10+
pack: codeql/python-all
11+
extensible: typeModel
12+
data:
13+
- ['OpenRouter', 'openrouter', 'Member[OpenRouter].ReturnValue']
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
---
2+
category: newQuery
3+
---
4+
* Replaced the experimental `py/prompt-injection` query with two new experimental queries, `py/system-prompt-injection` and `py/user-prompt-injection`, to distinguish untrusted data flowing into system-level prompts and tool descriptions from data flowing into user-role prompts. The queries model the `openai`, `agents`, `anthropic`, `google-genai`, `openrouter` and `langchain` frameworks.

python/ql/src/experimental/Security/CWE-1427/PromptInjection.qhelp

Lines changed: 0 additions & 24 deletions
This file was deleted.

0 commit comments

Comments
 (0)