Conversation
There was a problem hiding this comment.
Overall, this looks good! Here are some findings, which are all individually pretty small. I'll also tag @begelundmuller for the proper backend review.
Issues
1. Missing tool name and message type validation in GetAIToolCall (runtime/server/chat.go:539-545)
The handler finds any message by ID but doesn't verify it's a MessageTypeCall for query_metrics_view. If a caller passes the ID of a different tool or a result message, the handler would try to decode it as a metrics query, producing confusing errors.
// Suggest adding after the FilterByID lookup:
if callMsg.Tool != ai.QueryMetricsViewName {
return nil, status.Errorf(codes.InvalidArgument, "call %q is not a query_metrics_view call", req.CallId)
}
if callMsg.Type != ai.MessageTypeCall {
return nil, status.Errorf(codes.InvalidArgument, "message %q is not a tool call", req.CallId)
}2. Missing gRPC status codes on internal errors (runtime/server/chat.go:547-555)
Errors from UnmarshalMessageContent, WeakDecode, and ToStruct propagate as raw Go errors. gRPC surfaces these as codes.Unknown. Wrap with status.Errorf(codes.Internal, ...) for consistency.
3. Version downgrade in openapi.yaml
v0.81.2 → v0.80.2 in both OpenAPI files. Appears to be a merge artifact — should be fixed before merging.
4. callId extraction from URL can fail for trailing-slash URLs (enhance-citation-links.ts:18-19, 119)
The regex /-\/ai\/.*?\/call\/.*?\/?$/ allows trailing slashes, but url.pathname.split("/").pop() returns "" for trailing-slash URLs. These two can disagree silently.
Options: either use a capture regex (e.g., /\/-\/ai\/[^/]+\/call\/([^/]+)\/?$/) so the match and extraction are a single operation, or replace both with explicit URL segment parsing. Either way, the current approach of regex-test + separate split().pop() doesn't handle the trailing slash the regex permits.
Suggestions
5. enhance-citation-links.ts is hard to follow as-is
This file currently mixes three concerns in one blob: the Svelte action (DOM click handling, goto, embed awareness), the reactive store plumbing (subscribing to resources + conversation queries), and the URL mapping logic (parsing citation URLs, looking up messages, constructing explore URLs).
Extracting the URL mapping logic — specifically the mapper closure and the citation URL parsing helpers — into a separate file (e.g., citation-url-mapper.ts) would make the code easier to follow and, importantly, would make the mapping logic unit-testable. Right now it's buried inside a derived store inside a closure, so you can't test it without standing up the full reactive context. Good test cases would include: new citation URL → correct callId extraction (with trailing slash), legacy URL with malformed JSON → bracket-stripping fallback, unknown URL → passthrough, missing message → passthrough.
6. Add GetAIToolCall endpoint tests
The test in metrics_view_query_test.go was updated to verify URL format, but there's no test for the GetAIToolCall gRPC handler itself. This is a new API endpoint with access control — it should have at least a happy-path test and error-path tests (conversation not found, call not found, wrong tool type).
7. Add an eval for citation URL format
Using the existing eval framework (newEval() in runtime/ai/ai_test.go), add an eval case like TestAnalystCitationFormat that:
- Calls
router_agentwith a question that triggersquery_metrics_view - Asserts the tool result's
open_urlmatches the expected short link format - Completes the conversation and inspects the assistant's markdown for well-formed citation links (no missing parentheses, no truncation)
This would codify expectations for AI citation behavior and catch regressions if the prompt or URL format changes. The .completions.yaml output serves as a reference artifact.
proto/rill/runtime/v1/api.proto
Outdated
| // GetAIToolCall Resolves an AI tool call to its query arguments. Returns the query_metrics_view arguments for the given tool call, enabling the frontend to build a dashboard URL. | ||
| rpc GetAIToolCall(GetAIToolCallRequest) returns (GetAIToolCallResponse) { | ||
| option (google.api.http) = { | ||
| get: "/v1/instances/{instance_id}/ai/conversations/{conversation_id}/call/{call_id}" | ||
| }; | ||
| } |
There was a problem hiding this comment.
Since they're called "messages" in other APIs, shouldn't this be called GetAIMessage (and message_id and so on)
proto/rill/runtime/v1/api.proto
Outdated
| // Response message for RuntimeService.GetAIToolCall | ||
| message GetAIToolCallResponse { | ||
| // MetricsResolverQuery as JSON | ||
| google.protobuf.Struct query = 1; | ||
| } |
There was a problem hiding this comment.
Can this return a Message type to keep it generic? And then check tool == "query_metrics_view" and so forth on the frontend?
There was a problem hiding this comment.
Ya I think that makes sense. We could use this for other use cases in the future as well without the need to add a new API
runtime/ai/analyst_agent_test.go
Outdated
| require.Equal(t, "United States", res.Response) | ||
| } | ||
|
|
||
| var citationUrlExtractRegex = regexp.MustCompile(`\(\[[^]]+]\(([^)]+)\)\)`) |
There was a problem hiding this comment.
As this is used only once and its a test case, I suggest inlining it as there is no performance benefit to making it a global
There was a problem hiding this comment.
Removed it by using require.Contains
runtime/ai/analyst_agent_test.go
Outdated
| citationUrlMatches := citationUrlExtractRegex.FindStringSubmatch(agentRes.Response) | ||
| require.Len(t, citationUrlMatches, 2) | ||
| expectedCitationUrl := fmt.Sprintf(`https://ui.rilldata.com/-/dashboards/bids_metrics/-/ai/%s/call/%s`, s.ID(), calls[2].ID) | ||
| require.Equal(t, expectedCitationUrl, citationUrlMatches[1]) |
There was a problem hiding this comment.
Could this just be require.Contains(t, agentRes.Response, expectedCitationURL)?
runtime/ai/metrics_view_query.go
Outdated
| } | ||
|
|
||
| openURL.Path, err = url.JoinPath(openURL.Path, "-", "open-query") | ||
| openURL.Path, err = url.JoinPath(openURL.Path, "-", "ai", sessionID, "call", callID) |
There was a problem hiding this comment.
Maybe call the route "message" instead of "call"? Not sure.
There was a problem hiding this comment.
Ya if we change the API to return a generic message then ya, it makes sense. Probably also makes sense to call it /-/ai/<session_id>/message/<message_id>/-/open. This will allow us to add something at /-/ai/<session_id>/message/<message_id> in the future and not have to have obtuse backwards compatibility handling. What say @ericpgreen2 ?
runtime/server/chat.go
Outdated
| serveSSEUntilClose(w, events) | ||
| } | ||
|
|
||
| func (s *Server) GetAIToolCall(ctx context.Context, req *runtimev1.GetAIToolCallRequest) (*runtimev1.GetAIToolCallResponse, error) { |
There was a problem hiding this comment.
Can we add a test for this in chat_test.go?
4124866 to
ecf8fc5
Compare
ecf8fc5 to
e465713
Compare
begelundmuller
left a comment
There was a problem hiding this comment.
Backend looks good!
ericpgreen2
left a comment
There was a problem hiding this comment.
Frontend looks good too 👍
* fix: malformed citation urls * Try to sanitise the query param * Convert citation urls to link with conversation and query IDs * Revert to using call_id * Add UI redirect * Fix naviation in citation links * PR comments * More PR comments
Sometimes citation urls have additional brackets in the end. The guess is, the response generation gets messed up when the url itself has brackets and it gets appended to the url.
We are changing how citation urls work. Instead of query being referenced by LLM in response, we instead create a url that links to the conversation and the call id. This is then resolved to a query in UI by calling
GetAIToolCall.Closes APP-493
Checklist: