SageMakerClient ignores the passed boto3 session for the "sagemaker" client since 2.13.0 — all resource API calls sign with default-chain credentials

[brifordwylie]

PySDK Version

• PySDK V2 (2.x)
• PySDK V3 (3.x)

Describe the bug
Since sagemaker-core 2.13.0, SageMakerClient.__init__ (sagemaker-core/src/sagemaker/core/utils/utils.py) builds its main sagemaker client from a freshly created botocore session instead of the session argument the caller passed in:

# TODO: Remove post-launch. This loads a custom botocore service model
# (from the 'sample/' directory) that includes pre-GA Job APIs ...
bc_session = bc_session_mod.get_session()
...
custom_session = Session(botocore_session=bc_session, region_name=env_region)

self.sagemaker_client = custom_session.client("sagemaker", ...)   # <-- passed `session` ignored

The block was introduced in #5919 ("MTRL Launch PR", merged 2026-06-03, released as sagemaker-core 2.13.0) to load a custom service model for pre-GA Job APIs. Side effect: custom_session resolves credentials from the default provider chain (env vars / AWS_PROFILE / IMDS), silently discarding the caller's session credentials. The other clients in the same constructor (sagemaker-runtime, sagemaker-featurestore-runtime, sagemaker-metrics) still correctly use the passed session.

Because every resource class routes through Base.get_sagemaker_client(session=...) → SageMakerClient(...).get_client("sagemaker"), every V3 resource API call (FeatureGroup.create, Model.create, EndpointConfig.create, describe/delete/list, …) is signed with ambient credentials instead of the session the caller provided.

This is a credential-handling regression: code that assumes a scoped role and passes that session explicitly runs with the wrong identity. It surfaces as confusing permission errors (see logs), but in the other direction it can also escalate — calls intended to run under a restricted role run under a more privileged ambient identity instead.

Still present in the latest release (sagemaker-core 2.15.0 / sagemaker 3.15.0); the constructor is byte-identical from 2.13.1 through 2.15.0. Last correct version: 2.12.0 (self.sagemaker_client = session.client("sagemaker", ...)).

To reproduce
Minimal proof (no AWS resources created) — the returned client does not hold the passed session's credentials:

import boto3
from sagemaker.core.utils.utils import SageMakerClient

# Any session whose credentials differ from the default provider chain,
# e.g. an assumed role:
sts = boto3.client("sts")
creds = sts.assume_role(
    RoleArn="arn:aws:iam::<ACCOUNT_ID>:role/<SomeRole>",
    RoleSessionName="repro",
)["Credentials"]
session = boto3.Session(
    aws_access_key_id=creds["AccessKeyId"],
    aws_secret_access_key=creds["SecretAccessKey"],
    aws_session_token=creds["SessionToken"],
    region_name="us-west-2",
)

client = SageMakerClient(session=session, region_name="us-west-2").get_client("sagemaker")
client_key = client._request_signer._credentials.get_frozen_credentials().access_key

print("passed session key:", session.get_credentials().access_key)
print("client signs with :", client_key)
assert client_key == session.get_credentials().access_key, "client ignored the passed session"

On 2.12.0 the assert passes; on 2.13.0–2.15.0 it fails (the client holds default-chain credentials).

Real-world failure mode: call FeatureGroup.create(..., offline_store_config=..., role_arn=<role with Glue/S3 perms>, session=<assumed-role session>) from a machine whose default profile is a restricted SSO role. CreateFeatureGroup is signed with the SSO credentials; SageMaker forwards the caller's access for the offline-store Glue setup, and the feature group lands in CreateFailed:

FeatureSet <name> creation failed with status: CreateFailed
Failure reason: User: arn:aws:sts::<ACCOUNT_ID>:assumed-role/AWSReservedSSO_DataScientist_.../user
is not authorized to perform: glue:GetDatabase on resource: arn:aws:glue:us-west-2:<ACCOUNT_ID>:catalog
because no identity-based policy allows the glue:GetDatabase action (Service: Glue, Status Code: 400)

The same code with sagemaker-core 2.12.0 (identical credentials, identical account) succeeds.

Expected behavior
When a session is passed to SageMakerClient (directly or via any resource method's session= parameter), all clients it constructs — including the sagemaker client — sign requests with that session's credentials. If the custom service-model loader is still needed, it should be attached to the passed session's botocore session (or the credentials transplanted onto bc_session), not to a brand-new default-chain session.

Screenshots or logs
See CreateFailed reason above. Also note two more quirks in the same block, worth fixing together:

• SageMakerClient is a singleton (SingletonMeta), so even the passed-session handling of the other clients only honors whichever session arrives first in the process.
• logger.info(f"Runs on sagemaker {env_stage}, region:{env_region}") logs on every construction and reads like debug output.

System information

• SageMaker Python SDK version: sagemaker 3.13.1 / sagemaker-core 2.13.1 (broken); verified still present in sagemaker-core 2.15.0; last good 2.12.0
• Framework name (eg. PyTorch) or algorithm (eg. KMeans): n/a (Feature Store / core resource classes)
• Framework version: n/a
• Python version: 3.13
• CPU or GPU: CPU
• Custom Docker image (Y/N): N (bare venv on macOS; also reproduced conceptually in containers — masked there because ambient credentials == intended role)

Additional context

• Introduced by MTRL Launch PR #5919, released in sagemaker-core 2.13.0 (2026-06-03).
• The bug is easy to miss in EC2/ECS/Batch/SageMaker jobs because the default chain resolves to the attached role — i.e., the wrong session happens to be the right identity. It bites exactly when callers pass an explicit session because ambient credentials are not the intended identity (assumed roles, cross-account work, multi-profile laptops).
• Related-but-different session-respect reports: Fix/refresh respects session to fix job waiting #5790 (refresh), fix: lazy-init boto3 clients in AIRHub to avoid NoRegionError at import time #5935 (AIRHub import-time client creation).
• We've pinned sagemaker-core >= 2.12.0, < 2.13 as a workaround.