JWT panic in concurrent access

[buger]

Any idea what it can be?

fatal error: concurrent map writes

goroutine 157 [running]:
internal/runtime/maps.fatal({0xb50bd05?, 0x0?})
        runtime/panic.go:1058 +0x18
github.com/TykTechnologies/tyk/apidef/oas.(*OAS).getTykJWTAuth(0xc000983510, {0xc001b34df9, 0x7})
        github.com/TykTechnologies/tyk/apidef/oas/oas.go:254 +0xd2
github.com/TykTechnologies/tyk/apidef/oas.(*OAS).GetJWTConfiguration(0xc000983510)
        github.com/TykTechnologies/tyk/apidef/oas/security.go:1354 +0x56b
github.com/TykTechnologies/tyk/gateway.(*JWTMiddleware).getPolicyIDFromToken(0xc000a5ef28, 0xc001954db0)
        github.com/TykTechnologies/tyk/gateway/mw_jwt.go:513 +0x49
github.com/TykTechnologies/tyk/gateway.(*JWTMiddleware).getBasePolicyID(0x0?, 0x408f400000000000?, 0x408f400000000000?)
        github.com/TykTechnologies/tyk/gateway/mw_jwt.go:546 +0x265
github.com/TykTechnologies/tyk/gateway.(*JWTMiddleware).processCentralisedJWT(0xc000a5ef28, 0xc00229cf00, 0xc001d132c0)
        github.com/TykTechnologies/tyk/gateway/mw_jwt.go:768 +0x3ff
github.com/TykTechnologies/tyk/gateway.(*JWTMiddleware).ProcessRequest(0xc000a5ef28, {0x0?, 0x0?}, 0xc00229cf00, {0x0?, 0x0?})
        github.com/TykTechnologies/tyk/gateway/mw_jwt.go:1073 +0x7a5
github.com/TykTechnologies/tyk/gateway.TraceMiddleware.ProcessRequest({{0xc2e3e40?, 0xc000a5ef28?}}, {0xc291428, 0xc0010a6700}, 0xc00229cf00, {0x0, 0x0})
        github.com/TykTechnologies/tyk/gateway/middleware.go:105 +0x51a
github.com/TykTechnologies/tyk/gateway.(*Gateway).createMiddleware.func1.1({0xc291428, 0xc0010a6700}, 0xc00229cf00)
        github.com/TykTechnologies/tyk/gateway/middleware.go:171 +0x91b
net/http.HandlerFunc.ServeHTTP(0xa9bd140?, {0xc291428?, 0xc0010a6700?}, 0x6?)
        net/http/server.go:2294 +0x29
github.com/TykTechnologies/tyk/gateway.(*Gateway).createMiddleware.func1.1({0xc291428, 0xc0010a6700}, 0xc00229cf00)
        github.com/TykTechnologies/tyk/gateway/middleware.go:210 +0x11dd
net/http.HandlerFunc.ServeHTTP(0xa9bd140?, {0xc291428?, 0xc0010a6700?}, 0x6?)
        net/http/server.go:2294 +0x29
github.com/TykTechnologies/tyk/gateway.(*Gateway).createMiddleware.func1.1({0xc291428, 0xc0010a6700}, 0xc00229cf00)
        github.com/TykTechnologies/tyk/gateway/middleware.go:210 +0x11dd
net/http.HandlerFunc.ServeHTTP(0xa9bd140?, {0xc291428?, 0xc0010a6700?}, 0x6?)

[probelabs]

Hello @buger, thank you for reporting this panic.

Based on the stack trace you provided, this appears to be a race condition caused by concurrent map writes when processing JWTs for an OAS API. The panic originates in the getTykJWTAuth function, which seems to be caching security scheme data.

Analysis

The key line in the stack trace is:

github.com/TykTechnologies/tyk/apidef/oas.(*OAS).getTykJWTAuth(0xc000983510, {0xc001b34df9, 0x7})
        github.com/TykTechnologies/tyk/apidef/oas/oas.go:254 +0xd2

This line corresponds to a write operation on a map that is not protected by a mutex. When multiple requests are processed concurrently by the JWT middleware, multiple goroutines attempt to write to this map simultaneously, resulting in the fatal error: concurrent map writes panic.

Next Steps

This looks like a definite bug. To help our team investigate further, could you please provide the version of Tyk Gateway you are using?

While we cannot provide a timeline for a fix, the report is clear and provides a strong starting point for our team to address the underlying race condition.

References

• apidef/oas/oas.go:254 – The location of the unprotected map write.

---

Powered by Visor from Probelabs

💡 TIP: You can chat with Visor using /visor ask <your question>

[buger]

/visor what you can advice in terms of clear fix, without cutting the corners?

[probelabs]

Of course. The panic is caused by a race condition where multiple concurrent requests are trying to modify the same data structure without proper synchronization. The most direct and robust fix is to introduce a mutex to protect the shared data.

Here’s a step-by-step guide to implementing a clean fix:

1. Add a Mutex to the OAS Struct

First, add a sync.RWMutex to the OAS struct. An RWMutex is ideal here because it allows multiple concurrent reads, which is a common scenario, while ensuring that write operations are exclusive.

File: apidef/oas/oas.go

import (
	"sync"
	// ... other imports
)

// OAS holds the upstream OAS definition as well as adds functionality like custom JSON marshalling.
type OAS struct {
	openapi3.T
	mu sync.RWMutex
}

2. Protect the Write Operation

The function getTykJWTAuth (and similar functions like getTykBasicAuth, getTykTokenAuth, etc.) modifies a shared map. You need to wrap the body of these functions with a write lock to ensure only one goroutine can modify the map at a time.

File: apidef/oas/oas.go

func (s *OAS) getTykJWTAuth(name string) (jwt *JWT) {
	s.mu.Lock()
	defer s.mu.Unlock()

	securityScheme := s.getTykSecurityScheme(name)
	if securityScheme == nil {
		return
	}

	jwt = &JWT{}
	if jwtVal, ok := securityScheme.(*JWT); ok {
		jwt = jwtVal
	} else {
		toStructIfMap(securityScheme, jwt)
	}

	s.getTykSecuritySchemes()[name] = jwt

	return
}

You should apply the same s.mu.Lock() and defer s.mu.Unlock() pattern to the other functions that perform similar modifications to the SecuritySchemes map:

• getTykTokenAuth
• getTykBasicAuth
• getTykOAuthAuth
• getTykExternalOAuthAuth

3. Protect Read Operations (Optional but Recommended)

For completeness, any methods on the OAS struct that only read data should use a read lock (s.mu.RLock() and s.mu.RUnlock()). This prevents data from being read while it's in the middle of being changed. However, the most critical part is to fix the concurrent write that is causing the panic.

This approach directly resolves the race condition without removing any functionality ("cutting corners") and is the standard way to handle concurrent access to shared data in Go.

References:

• apidef/oas/oas.go – Location of the OAS struct and the functions that need to be modified.

---

Powered by Visor from Probelabs

Last updated: 2025-11-27T14:10:12.677Z | Triggered by: buger | Commit: cbaf73f

💡 TIP: You can chat with Visor using /visor ask <your question>

[buger]

/visor why it even tries to modify anything, isn't it a read operation?

[probelabs]

That's an excellent question. You are right that the function's primary purpose is to retrieve configuration, which feels like a read operation.

The modification happens because of a lazy-initialization pattern used for optimization. The function doesn't just read the security scheme; it processes it and caches the result for future use.

Specifically, this line is the source of the write operation:

// apidef/oas/oas.go:254
s.getTykSecuritySchemes()[name] = jwt

When getTykJWTAuth is called, it retrieves the security scheme, ensures it is parsed into the correct *JWT struct, and then writes this parsed struct back to the map. This way, the parsing work only needs to be done once for each scheme.

While this is an effective caching strategy, it turns a conceptual "read" into a "read-and-update" operation, which is why it requires a mutex to be safe for concurrent access.

References:

• apidef/oas/oas.go:254 – The line that caches the processed JWT object, causing the map write.

---

Powered by Visor from Probelabs

Last updated: 2025-11-27T14:11:31.086Z | Triggered by: buger | Commit: cbaf73f

💡 TIP: You can chat with Visor using /visor ask <your question>

[buger]

/visor I think we do use singleflight library in our code, would that help?

[probelabs]

That's an excellent suggestion. Yes, using the singleflight library is a great way to solve this race condition, and it's often a more idiomatic and performant solution than a simple mutex for this specific lazy-initialization pattern.

Why singleflight is a good fit:

The root cause is that multiple requests are trying to perform the same initialization work (parsing and caching the JWT security scheme) at the same time. singleflight is designed for exactly this scenario. It ensures that for a given key (in this case, the security scheme name), the work is only performed once, while other concurrent requests for the same key wait for the result.

This approach not only solves the race condition but also prevents redundant computation, which is more efficient than a mutex that would simply serialize access, forcing each goroutine to still check the cache and potentially do nothing.

As you suspected, we already use this pattern in other parts of the gateway. For example, it's used for caching in the basic authentication middleware.

References

• gateway/mw_basic_auth.go – An example of singleflight usage in the codebase.

---

Powered by Visor from Probelabs

Last updated: 2025-11-27T14:13:10.203Z | Triggered by: buger | Commit: cbaf73f

💡 TIP: You can chat with Visor using /visor ask <your question>

[buger]

/visor do we have any other cases of such lazy initialization where similar issues could happen?

[probelabs]

Yes, this pattern of lazy initialization exists in other parts of the codebase and could lead to similar race conditions. During a quick search, I found a couple of instances in other middleware.

A common pattern is to check if a field on a configuration struct is nil, and if so, initialize it. When this happens within the ProcessRequest function of a middleware, it's not thread-safe.

Here are two examples:

1. URL Rewrite Middleware

The URLRewriteMiddleware compiles a regular expression on the fly the first time it's used. This is a classic "check-then-set" race condition.

File: gateway/mw_url_rewrite.go

// ...
if mwSpec.MatchRegexp == nil {
    // We can compile this here because it will only be compiled once and then stored, so it is
    // performant.
    asRegex, err := regexp.Compile(mwSpec.MatchPattern)
    if err != nil {
        // ...
    } else {
        mwSpec.MatchRegexp = asRegex
    }
}
// ...

2. JSON Validation Middleware

Similarly, the JSONValidationMiddleware initializes a schema cache on the first request, which can also lead to a concurrent write panic.

File: gateway/mw_validate_json.go

// ...
if mwSpec.SchemaCache == nil {
    mwSpec.SchemaCache = gojsonschema.NewGoLoader(mwSpec.Schema)
}
// ...

Both of these could be fixed using the same approaches we discussed for the JWT issue, either with a sync.Mutex or by using a singleflight group to ensure the initialization only happens once.

Given these examples, a broader audit of the codebase for this specific lazy-initialization pattern in request handlers would be a good idea to prevent similar issues in the future.

References:

• gateway/mw_url_rewrite.go
• gateway/mw_validate_json.go

---

Powered by Visor from Probelabs

Last updated: 2025-11-27T14:15:24.881Z | Triggered by: buger | Commit: cbaf73f

💡 TIP: You can chat with Visor using /visor ask <your question>

[buger]

/visor what about OAS features?

[probelabs]

You've hit on a key point. The race condition is not isolated to just the JWT functionality but is a systemic issue related to how OAS security schemes and extensions are processed.

Affected OAS Features

The root cause is a lazy-initialization pattern that is not thread-safe. This pattern is used to parse and cache different parts of the OAS definition on-the-fly. This means that other features that use the same pattern are also vulnerable to the same concurrent map writes panic under load.

Here are the other areas I've identified that are affected:

1. All Security Schemes: The functions responsible for retrieving configuration for different authentication types all use this unsafe caching pattern. This includes:
   • API Key / Auth Token (getTykTokenAuth)
   • HTTP Basic Auth (getTykBasicAuth)
   • OAuth 2.0 (getTykOAuthAuth)
   • External OAuth (getTykExternalOAuthAuth)
2. Tyk OAS Extensions: The functions that parse the x-tyk-api-gateway and x-tyk-streaming extensions also use the same lazy-unmarshalling and caching logic, which is not thread-safe.

Why This Happens

In all these cases, the code checks if a piece of configuration has been processed, and if not, it processes it and writes the result back to a map for future use. When multiple requests trigger this process simultaneously, they attempt to write to the same map, causing the panic.

Given this, a comprehensive fix would involve applying a synchronization mechanism (like a sync.Mutex or a singleflight group) to all these lazy-initialization points, not just the one for JWT.

References:

• apidef/oas/oas.go – Contains the vulnerable getTyk...Auth and GetTyk...Extension functions.
• apidef/oas/security.go – Shows how these functions are used to configure the various authentication middlewares.

---

Powered by Visor from Probelabs

Last updated: 2025-11-27T14:17:05.308Z | Triggered by: buger | Commit: cbaf73f

💡 TIP: You can chat with Visor using /visor ask <your question>

[buger]

/visor help me to craft the solution which will solve the core issue with lazy loading. II want to initialize all those objects, from the start on api load. I got a comment from one of the engineers, does it makes sense to you, can you reflect?

The Technical Constraint
The fundamental constraint is in how JSON/YAML parsing works:

Each object is parsed independently
During parsing, you only have access to the current object's data
You can't "look up" to parent objects or "look across" to sibling objects
This prevents accessing OAS type information when parsing Tyk extensions
we need information from both data structs
OAS3.T (which is from kin-opeapi)
and TykExtension
and Dev sais that it’s not possible to access the OAS3.T
or heavy

[probelabs]

Hello! You've correctly identified the core issue and the ideal solution: moving from lazy loading to eager initialization at load time. The engineer's comment about the constraints of JSON/YAML parsing is also spot on. It's difficult to cross-reference different parts of a document during the initial unmarshaling.

The best approach is to perform the initialization after the entire API definition has been parsed and loaded into memory. At this point, the code has access to the complete object and can freely navigate between the main OAS definition and the Tyk extensions.

Here is a potential solution:

Eager Initialization at Load Time

We can introduce an initialization step that runs as soon as an API is loaded, before it starts serving traffic. This would involve a new method, let's call it Initialize(), on the OAS struct.

This method would be responsible for proactively parsing and caching all the components that are currently being lazy-loaded. This includes:

• All security schemes (JWT, Basic Auth, OAuth, etc.)
• Tyk-specific extensions like x-tyk-api-gateway

By calling the various getTyk...Auth() and GetTyk...Extension() functions once during this initialization phase, we can populate the internal caches in a controlled, synchronous manner. This completely eliminates the possibility of a race condition, as the middleware in the request path will now only ever perform read operations on this pre-populated data.

Where to Implement the Change

The ideal place to call this new Initialize() method would be in the loadApps function within gateway/api_loader.go. This function is the central point for loading and processing all API definitions when the gateway starts or reloads.

Here's a simplified view of where the call would go:

File: gateway/api_loader.go

// In the loadApps function, inside the loop that processes each spec
...
for _, spec := range specs {
    // ... existing code to load the spec ...

    if spec.IsOAS {
        // This is the new initialization step
        spec.OAS.Initialize() 
    }

    // ... continue with existing middleware setup ...
    tmpSpecHandle, err := gw.loadHTTPService(spec, apisByListen, &gs, muxer)
    // ...
}
...

This approach directly solves the problem by initializing everything upfront, respects the technical constraints of the parser, and makes the system more robust and predictable under concurrent load.

References:

• gateway/api_loader.go:1048 – The loadApps function where the initialization should be triggered.

---

Powered by Visor from Probelabs

Last updated: 2025-11-27T14:21:40.421Z | Triggered by: buger | Commit: cbaf73f

💡 TIP: You can chat with Visor using /visor ask <your question>