MDEV-39933 JSON_NORMALIZE returns wrong result for invalid JSON with … by akshatnehra · Pull Request #5242 · MariaDB/server

akshatnehra · 2026-06-15T17:30:18Z

Description

This PR fixes MDEV-39933 where JSON_NORMALIZE() incorrectly returns a normalized result instead of NULL for invalid JSON containing embedded NUL bytes after charset conversion.

json_normalize() uses strlen() to determine the converted string length, but strlen() stops at the first NUL byte. This silently truncates the string to the valid JSON prefix, so JSON_NORMALIZE sees only {"a":1} instead of {"a":1}\0junk — returning a normalized result for what JSON_VALID correctly rejects.

The fix involves:

Using my_convert()'s return value (actual bytes written) instead of strlen(), which correctly accounts for embedded NUL bytes.

How can this PR be tested?

Run the MTR test:

build/mysql-test/mtr main.mdev_39933

Or manually:

-- JSON_VALID correctly says this is invalid
SELECT JSON_VALID(CONVERT(CONCAT('{"a":1}', CHAR(0), 'junk') USING latin1));
-- Returns: 0

-- JSON_NORMALIZE should agree (return NULL for invalid JSON)
SELECT JSON_NORMALIZE(CONVERT(CONCAT('{"a":1}', CHAR(0), 'junk') USING latin1));

Results from my testing

Before changes

MariaDB> SELECT JSON_NORMALIZE(CONVERT(CONCAT('{"a":1}', CHAR(0), 'junk') USING latin1)) AS result;
+------------+
| result     |
+------------+
| {"a":1.0E0}|
+------------+

MariaDB> SELECT JSON_VALID(CONVERT(CONCAT('{"a":1}', CHAR(0), 'junk') USING latin1)) AS valid;
+-------+
| valid |
+-------+
|     0 |
+-------+

JSON_VALID says it's invalid but JSON_NORMALIZE returns a result — inconsistent.

After changes

MariaDB> SELECT JSON_NORMALIZE(CONVERT(CONCAT('{"a":1}', CHAR(0), 'junk') USING latin1)) AS result;
+--------+
| result |
+--------+
| NULL   |
+--------+

MariaDB> SELECT JSON_VALID(CONVERT(CONCAT('{"a":1}', CHAR(0), 'junk') USING latin1)) AS valid;
+-------+
| valid |
+-------+
|     0 |
+-------+

Both functions now agree: invalid JSON returns NULL/0.

Affected versions

Reproduced on 10.11, 11.4, 11.8, and 12.3 (as confirmed in Jira comments). Lowest checked version: 10.11 with commit b89cb5d.

Reproducer:

SELECT JSON_NORMALIZE(CONVERT(CONCAT('{"a":1}', CHAR(0), 'junk') USING latin1));

Basing the PR against the correct MariaDB version

This is a bug fix and the PR is based against the latest MariaDB development branch.

PR quality check

I have checked the CODING_STANDARDS.md file and my PR conforms to this where appropriate.
For any trivial modifications to the PR, I am fine with the reviewer making the changes themselves.

…embedded NUL json_normalize() uses strlen() to determine the length of the charset- converted string, but strlen() stops at the first NUL byte. When the input contains embedded NUL bytes after charset conversion (e.g., from latin1 to utf8mb4), the converted string is silently truncated to the valid JSON prefix, causing JSON_NORMALIZE to return a normalized result instead of NULL. Fix: use the return value of my_convert() as the actual converted length instead of strlen(). my_convert() returns the number of bytes written, which correctly accounts for embedded NUL bytes. All new code of the whole pull request, including one or several files that are either new files or modified ones, are contributed under the BSD-new license. I am contributing on behalf of my employer Amazon Web Services, Inc.

gemini-code-assist

Code Review

This pull request fixes an issue in json_normalize where invalid JSON data containing embedded NUL bytes or trailing junk was incorrectly handled. By using the return value of my_convert instead of strlen to determine the size of the converted UTF-8 string, the function now correctly identifies and rejects invalid JSON. Test cases have been added to verify this behavior. I have no feedback to provide as there are no review comments.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

gkodinov

Thank you for your contribution! This is a preliminary review.

This is a bug fix. Please re-base on 10.11.
I also have some further suggestions below.

gkodinov · 2026-06-16T11:14:03Z

@@ -1030,15 +1030,14 @@ json_normalize(DYNAMIC_STRING *result,
    if (!s_utf8)
      return 1;
    memset(s_utf8, 0x00, in_size);


while you're at it, can you please remove this memset and replace it with a s_utf8[in_size] = 0; after the my_convert call please?

gkodinov · 2026-06-16T11:22:56Z

+# MDEV-39933: Incorrect result of JSON_NORMALIZE on invalid json data
+#
+# JSON with trailing literal junk (no NUL) - should return NULL
+SELECT JSON_NORMALIZE('{"a":1}0junk');


is this really reproducing anything? I suppose not.
In my unpatched server:

MariaDB [(none)]> SELECT JSON_NORMALIZE('{"a":1}0junk'); +--------------------------------+ | JSON_NORMALIZE('{"a":1}0junk') | +--------------------------------+ | NULL | +--------------------------------+ 1 row in set (0.002 sec)

gkodinov · 2026-06-16T11:23:40Z

+JSON_NORMALIZE('{"a":1}0junk')
+NULL
+# JSON_VALID correctly rejects JSON with embedded NUL byte + trailing junk
+SELECT JSON_VALID(CONVERT(CONCAT('{"a":1}', CHAR(0), 'junk') USING latin1));


this one too:

MariaDB [(none)]> SELECT JSON_VALID(CONVERT(CONCAT('{"a":1}', CHAR(0), 'junk') USING latin1)); +----------------------------------------------------------------------+ | JSON_VALID(CONVERT(CONCAT('{"a":1}', CHAR(0), 'junk') USING latin1)) | +----------------------------------------------------------------------+ | 0 | +----------------------------------------------------------------------+ 1 row in set (0.003 sec)

gkodinov · 2026-06-16T11:24:51Z

+JSON_NORMALIZE(CONVERT(CONCAT('{"a":1}', CHAR(0)) USING latin1))
+NULL
+# Valid JSON through conversion should still work
+SELECT JSON_NORMALIZE(CONVERT('{"a":1}' USING latin1));


no need to re-test that IMHO

gkodinov · 2026-06-16T11:25:37Z

+JSON_NORMALIZE(CONVERT('{"a":1}' USING latin1))
+{"a":1.0E0}
+# Multi-byte source charset (utf16)
+SELECT JSON_NORMALIZE(CONVERT(CONCAT('{"a":1}', CHAR(0), 'x') USING utf16));


this is a bit too much and is not touching on any of the changed lines.

gkodinov · 2026-06-16T11:26:04Z

+JSON_NORMALIZE(CONVERT(CONCAT('{"a":1}', CHAR(0), 'x') USING utf16))
+NULL
+# NUL embedded inside a JSON string value
+SELECT JSON_NORMALIZE(CONVERT(CONCAT('{"a":"b', CHAR(0), 'c"}') USING latin1));


also this. too much IMHO

gkodinov · 2026-06-16T11:26:36Z

+JSON_NORMALIZE(CONVERT(CONCAT('[1,2,3]', CHAR(0), 'x') USING latin1))
+NULL
+# Nested object - valid through conversion
+SELECT JSON_NORMALIZE(CONVERT('{"a":{"b":1}}' USING latin1));


what is this testing?

gkodinov · 2026-06-16T11:26:51Z

+JSON_NORMALIZE(CONVERT('{"a":{"b":1}}' USING latin1))
+{"a":{"b":1.0E0}}
+# Empty string
+SELECT JSON_NORMALIZE(CONVERT('' USING latin1));


what is this testing?

gkodinov · 2026-06-16T11:27:37Z

+SELECT JSON_NORMALIZE(CONVERT('{"a":{"b":1}}' USING latin1));
+
+--echo # Empty string
+SELECT JSON_NORMALIZE(CONVERT('' USING latin1));


we typically end files with :

--echo End of 10.11 tests.

gemini-code-assist Bot reviewed Jun 15, 2026

View reviewed changes

gkodinov added the External Contribution All PRs from entities outside of MariaDB Foundation, Corporation, Codership agreements. label Jun 16, 2026

gkodinov self-assigned this Jun 16, 2026

gkodinov requested changes Jun 16, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

MDEV-39933 JSON_NORMALIZE returns wrong result for invalid JSON with …#5242

MDEV-39933 JSON_NORMALIZE returns wrong result for invalid JSON with …#5242
akshatnehra wants to merge 1 commit into
MariaDB:mainfrom
akshatnehra:MDEV-39933

akshatnehra commented Jun 15, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gkodinov left a comment

Uh oh!

gkodinov Jun 16, 2026

Uh oh!

gkodinov Jun 16, 2026

Uh oh!

gkodinov Jun 16, 2026

Uh oh!

gkodinov Jun 16, 2026

Uh oh!

gkodinov Jun 16, 2026

Uh oh!

gkodinov Jun 16, 2026

Uh oh!

gkodinov Jun 16, 2026

Uh oh!

gkodinov Jun 16, 2026

Uh oh!

gkodinov Jun 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

akshatnehra commented Jun 15, 2026

Description

How can this PR be tested?

Results from my testing

Affected versions

Basing the PR against the correct MariaDB version

PR quality check

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gkodinov left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

2 participants