[Enh]: Semantic search with Redis (Phase 2)

[JerryNixon]

What?

Add semantic search over prepopulated Redis vector indexes for configured entities.

Why?

Allow fast natural language retrieval of known entities while keeping query behavior deterministic and bounded.

How?

Add REST parameters, a GraphQL semantic query surface, and entity configuration that maps an entity to a Redis vector index. The index returns entity keys and similarity scores. Data API builder then resolves those keys through the normal entity retrieval pipeline, using Redis Level-2 cache if present and falling back to SQL when needed.

sequenceDiagram
    participant Client
    participant DAB.Endpoint
    participant DAB.Embedding
    participant Redis
    participant SQL

    Client->>DAB.Endpoint: GET /api/colors?$semantic=xxx

    DAB.Endpoint->>DAB.Endpoint: Parse and validate $semantic

    DAB.Endpoint->>DAB.Embedding: Generate embedding("bright blue")
    DAB.Embedding-->>DAB.Endpoint: vector

    DAB.Endpoint->>Redis: Vector search (index-name, TopK=first)
    Redis-->>DAB.Endpoint: keys + similarity

    DAB.Endpoint->>Redis: Lookup entity rows in L2 cache
    Redis-->>DAB.Endpoint: cached rows (partial)

    DAB.Endpoint->>SQL: SELECT ... WHERE key IN (cache misses)
    SQL-->>DAB.Endpoint: entity rows

    DAB.Endpoint->>Redis: Populate cache for misses

    DAB.Endpoint->>DAB.Endpoint: Merge rows with similarity\nFilter by threshold\nSort by similarity desc

    DAB.Endpoint-->>Client: JSON results with similarity

Loading

REST

New keyword

Parameter	Description
$semantic	Required. A semicolon separated list of key:value pairs defining semantic lookup settings.

Example

GET /api/colors?$semantic=text:bright%20blue;first:10;threshold:0.85

Supported keys (within $semantic)

Key	Description
text	Required. Free form text used to perform the semantic lookup.
first	Optional. Number of top results returned.
threshold	Optional. Minimum similarity score required for a match.

Encoding rules

• $semantic is parsed after normal URL decoding of the query string.
• Values run until the next ;.
• If a value must contain ; or : the client must URL encode those characters.
• ; must be encoded as %3B.
• : must be encoded as %3A.

Note

text values may include spaces without quoting. Spaces should be URL encoded as %20.

Rules

1. $semantic is only valid on entity read operations.
2. $semantic is not valid for item lookup by primary key.
3. $semantic is not valid for stored procedure execution.
4. $semantic may appear at most once. Duplicate $semantic parameters are rejected.
5. $semantic is rejected if semantic search is not configured for the entity.
6. Unknown semantic keys are rejected.
7. Keys are case insensitive. Canonical keys are text, first, and threshold.
8. text is required and must not be empty after trimming.
9. first is optional. If omitted the entity semantic-search.first value is used. If not configured the built in default is used.
10. threshold is optional. If omitted the entity semantic-search.threshold value is used. If not configured the built in default is used.
11. first must be an integer greater than or equal to 1.
12. first must be less than or equal to Int16.MaxValue.
13. threshold must be a float between 0 and 1 inclusive.
14. $filter cannot be combined with $semantic.
15. $orderby cannot be combined with $semantic.
16. $after cannot be combined with $semantic.
17. $first cannot be combined with $semantic.
18. $select may be combined with $semantic.
19. $select applies only to returned entity fields.
20. similarity is always included in the response even if not listed in $select.
21. $select does not affect semantic ranking or filtering.
22. Results are ordered by descending similarity.
23. The number of returned records is controlled only by the effective first value.
24. Records with similarity below the effective threshold are excluded.
25. If no records meet the threshold an empty collection is returned.
26. The response always includes a similarity value for each returned record.
27. If semantic search execution fails the request fails. No fallback to standard retrieval occurs.

Important

Error if $semantic is used and semantic search is not configured on the entity.

Results

similarity is injected into each returned record.

{
  "value": [
    {
      "id": 4,
      "name": "Yellow",
      "similarity": 0.92
    },
    {
      "id": 7,
      "name": "Orange",
      "similarity": 0.88
    }
  ]
}

GraphQL

Semantic results use a separate semantic type so similarity is only available for semantic queries.

query {
  semanticColors(
    semantic: {
      text: "bright colors"
      threshold: 0.85
      first: 1
    }
  ) {
    id
    name
    similarity
  }
}

Argument	Description
semantic.text	(Required) Free form text used to perform the semantic lookup.
semantic.first	(Optional) Number of top results returned.
semantic.threshold	(Optional) Minimum similarity score required for a match.

Important

The semantic query surface appears only when semantic search is configured for the entity.

Schema

type Color {
  id: ID
  name: String
}

type SemanticColor {
  id: ID
  name: String
  similarity: Float
}

input SemanticInput {
  text: String!
  first: Int
  threshold: Float
}

type Query {
  colors: [Color]
  semanticColors(semantic: SemanticInput): [SemanticColor]
}

Results

{
  "data": {
    "semanticColors": [
      {
        "id": 4,
        "name": "Yellow",
        "similarity": 0.92
      }
    ]
  }
}

Configuration

{
  "entities": {
    "products": {
      "semantic-search": {
        "provider": "redis",
        "index-name": "idx:products:semantic",
        "threshold": 0.85,
        "first": 10
      }
    }
  }
}

Property	Description
provider	Optional. Default redis. Semantic search provider.
index-name	Required. Redis vector index used for the semantic search.
threshold	Optional. Default 0.85. Default similarity threshold.
first	Optional. Default 10. Default number of records returned.

Command Line

Configure semantic search for an entity.

dab add entity products \
  --semantic-search.provider redis \
  --semantic-search.index-name "idx:products:semantic" \
  --semantic-search.threshold 0.85 \
  --semantic-search.first 10

Update semantic settings for an existing entity.

dab update entity products \
  --semantic-search.provider redis \
  --semantic-search.index-name "idx:products:semantic" \
  --semantic-search.threshold 0.85 \
  --semantic-search.first 10

Required Level 2 Cache Configuration

Level 2 caching must be enabled for semantic search.

{
  "runtime": {
    "cache": {
      "enabled": true,
      "ttl-seconds": 5,
      "level-2": {
        "enabled": true,
        "provider": "redis",
        "connection-string": "@env('DAB_REDIS_CONNECTION_STRING')",
        "partition": "myapp"
      }
    }
  }
}

Note
This subsystem is unchanged. It is required for semantic caching.

Required Embedding Configuration

The embedding subsystem must be enabled to generate query vectors.

{
  "runtime": {
    "embeddings": {
      "enabled": true,
      "provider": "azure-openai | openai",
      "base-url": "...",
      "api-key": "...",
      "model": "...",
      "api-version": "2024-02-01",
      "dimensions": 1536,
      "timeout-ms": 30000,
      "endpoint": {
        "enabled": false,
        "roles": ["authenticated"]
      },
      "health": {
        "enabled": true,
        "threshold-ms": 5000,
        "test-text": "health check",
        "expected-dimensions": 1536
      }
    }
  }
}

Note
This subsystem is unchanged. It is required to generate embeddings for semantic lookup.

JSON Configuration Rules

Entity rules

1. semantic-search is optional. Entities behave normally unless semantic-search is configured.
2. semantic-search.provider is required when semantic-search exists.
3. semantic-search.provider enum supports only redis.
4. semantic-search.index-name is required and must reference an existing Redis vector index.
5. semantic-search.threshold is optional and defaults to 0.85 if not specified.
6. semantic-search.threshold must be between 0 and 1.
7. semantic-search.first is optional and defaults to 10 if not specified.
8. semantic-search.first must be >= 1.
9. semantic-search.first must be <= Int16.MaxValue.

Runtime dependencies

10. cache.enabled must be true.
11. cache.level-2.enabled must be true.
12. cache.level-2.provider must be redis and match semantic-search.provider.
13. embeddings.enabled must be true.
14. embeddings.dimensions must match the vector dimension used by the Redis index.

Error handling

Event	HTTP	Exception
$semantic used but entity has no semantic-search configuration	400	SemanticSearchNotConfiguredException — "Semantic search requested but this entity does not have semantic-search configured."
Invalid $semantic value such as missing text, non numeric first, or threshold outside 0..1	400	InvalidSemanticParameterException — "One or more semantic parameters are invalid."
$semantic used with incompatible parameters like $filter, $orderby, $after, or $first	400	SemanticParameterConflictException — "Semantic search cannot be combined with $filter, $orderby, $after, or $first."
Embedding provider endpoint unreachable	503	SemanticSearchException — "Embedding provider endpoint could not be reached."
Embedding provider authentication rejected	502	SemanticSearchException — "Embedding provider rejected authentication."
Embedding generation request exceeds configured timeout	504	SemanticSearchException — "Embedding generation exceeded the configured timeout."
Embedding provider returned invalid response	502	SemanticSearchException — "Embedding provider returned an unexpected response format."
Embedding provider returned empty vector	502	SemanticSearchException — "Embedding provider returned an empty embedding vector."
Embedding vector dimension mismatch	500	SemanticSearchException — "Embedding vector dimension does not match configured dimensions."
Redis connection cannot be established	503	SemanticSearchException — "Redis provider could not be reached."
Redis authentication rejected	502	SemanticSearchException — "Redis provider rejected authentication."
Redis search request exceeds timeout	504	SemanticSearchException — "Redis semantic search query timed out."
Configured Redis index not found	500	SemanticSearchException — "Configured semantic-search index-name was not found."
Redis instance does not support vector search	500	SemanticSearchException — "Redis instance does not support vector search."
Redis returned malformed semantic results	500	SemanticSearchException — "Redis returned malformed semantic search results."
semantic-search configured but Level 2 cache disabled	startup	SemanticSearchException — "Semantic search requires Level 2 Redis cache to be enabled."
semantic-search configured but embeddings subsystem disabled	startup	SemanticSearchException — "Semantic search requires the embeddings subsystem to be enabled."
All results fall below similarity threshold	200	— Semantic search completed but no results met the similarity threshold.
Redis returns fewer results than requested	200	— Semantic search returned fewer results than requested.

OpenTelemetry

Record semantic search activity with traces and metrics so failures and latency can be diagnosed without introducing many custom exceptions.

Spans

Span	Purpose	Key Attributes
dab.semantic	Overall semantic search execution	dab.entity, dab.semantic.first, dab.semantic.threshold, dab.semantic.text.length
dab.embedding	Embedding generation request	ai.provider, ai.model, ai.dimensions
dab.redis.search	Redis vector search	db.system=redis, db.operation=vector_search, db.redis.index, dab.semantic.first

Error Recording

Event	Span	Attributes
Embedding provider unreachable	dab.embedding	error.type=connection
Embedding authentication failure	dab.embedding	error.type=auth
Embedding timeout	dab.embedding	error.type=timeout
Redis connection failure	dab.redis.search	error.type=connection
Redis authentication failure	dab.redis.search	error.type=auth
Redis query timeout	dab.redis.search	error.type=timeout
Invalid Redis search results	dab.redis.search	error.type=invalid_result

Metrics

Metric	Type	Description
dab.semantic.requests	Counter	Number of semantic search requests
dab.semantic.duration	Histogram	End to end semantic search latency
dab.embedding.duration	Histogram	Embedding generation latency
dab.redis.search.duration	Histogram	Redis vector search latency
dab.semantic.results	Histogram	Number of records returned

Useful Dimensions

Attribute	Example
dab.entity	colors
dab.semantic.threshold	0.85
dab.semantic.first	10
ai.model	text-embedding-3-large
db.redis.index	idx:products:semantic
status	success, error, empty