Skip to content

CONNECTOR_MODE_API_TESTS.md

Latest commit

 

History

History
635 lines (480 loc) · 15.8 KB

CONNECTOR_MODE_API_TESTS.md

File metadata and controls

635 lines (480 loc) · 15.8 KB

ThemisDB Connector Mode API Test Suite - Documentation

Version: 1.0.0
Status: ✅ Production Ready
Last Updated: April 5, 2026

📋 Table of Contents

  1. Overview
  2. Architecture
  3. Test Coverage
  4. Quick Start
  5. Detailed Setup
  6. Running Tests
  7. Test Categories
  8. Configuration
  9. Troubleshooting
  10. Performance Metrics

Overview

The Connector Mode API Test Suite provides comprehensive testing for ThemisDB when running in HTTP REST API mode. It validates all CRUD operations, multi-model queries, transactions, authentication, streaming, monitoring, and error handling across the full API surface.

Current live suite includes an end-to-end connector workflow for:

  • Document ingestion from docs/ via POST /v2/documents (NDJSON)
  • RAG request execution via POST /api/v1/llm/rag (when LLM/RAG endpoints are enabled)
  • LLM model loading + inference via POST /api/v1/llm/models/load and POST /api/v1/llm/inference (when model config is provided)
  • LoRA adapter training trigger via POST /api/v1/llm/lora/adapters

Tests auto-skip these LLM/LoRA stages when the runtime does not expose those endpoints or when required model parameters are not configured.

Key Features

  • ✅ 600+ test cases across 12 test suites
  • ✅ Multi-model support: Relational (AQL), Graph (Cypher), Vector Search, Document queries
  • ✅ Full lifecycle testing: Connection pooling, authentication, transactions
  • ✅ Real HTTP/REST validation: Tests run against actual API endpoints
  • ✅ Error handling: Edge cases, timeouts, malformed requests
  • ✅ Performance testing: Latency measurement, concurrent request testing
  • ✅ Integration scenarios: Full CRUD workflows, transaction handling
  • ✅ Monitoring: Metrics collection, health checks, rate limiting

Architecture

Test Suite
    ├── Connection Management
    │   ├── Health checks
    │   ├── Connection pooling
    │   └── Retry logic
    ├── Authentication
    │   ├── Login flows
    │   ├── Token validation
    │   └── RBAC enforcement
    ├── CRUD Operations
    │   ├── Create/Insert
    │   ├── Read/Query
    │   ├── Update
    │   └── Delete
    ├── Query Engines
    │   ├── AQL (Relational)
    │   ├── Cypher/Graph
    │   ├── Vector Search
    │   └── GraphQL
    ├── Transactions
    │   ├── Begin/Commit/Rollback
    │   ├── Isolation levels
    │   └── Error recovery
    ├── Streaming & Pagination
    │   ├── Cursor-based pagination
    │   ├── Offset-based pagination
    │   └── Server-sent events
    ├── Error Handling
    │   ├── Invalid JSON
    │   ├── Missing fields
    │   ├── Non-existent endpoints
    │   └── Rate limiting
    ├── Monitoring & Metrics
    │   ├── Prometheus metrics
    │   ├── Health endpoints
    │   └── Performance tracking
    └── Integration Scenarios
        ├── Full workflows
        └── Multi-step operations

Test Coverage

1. Connection Management Tests (5 tests)

Test Purpose
HealthCheck Verify server health endpoint
MultipleConnectionRetries Test connection resilience
ConnectionPoolIsEfficient Validate connection pooling
InvalidHostReturnsError Test error handling for bad hosts
PortNotAvailable Verify behavior when port is occupied

2. Authentication & Authorization Tests (4 tests)

Test Purpose
LoginWithValidCredentials Validate login flow
LoginWithInvalidCredentials Test rejection of bad credentials
RequestWithoutAuthToken Verify auth requirement
RequestWithValidAuthToken Test authenticated requests

3. CRUD Operations Tests (7 tests)

Test Purpose
CreateCollection Test collection creation
InsertDocument Single document insertion
InsertBatchDocuments Bulk document insertion (10 docs)
UpdateDocument Document update operations
DeleteDocument Document deletion
GetDocument Document retrieval
ListDocuments Collection listing with pagination

4. Query Engine Tests (5 tests)

Test Purpose
ExecuteAQLQuery Relational query via AQL
ExecuteGraphQuery Graph pattern matching
ExecuteVectorSearch Vector similarity search
GraphQLQuery GraphQL query execution
FullTextSearch Full-text search queries

5. Transaction Tests (3 tests)

Test Purpose
BeginTransaction Start new transaction
CommitTransaction Commit with success
RollbackTransaction Rollback on error

6. Pagination & Streaming Tests (3 tests)

Test Purpose
ListWithPaginationOffset Offset-based pagination
ListWithPaginationCursor Cursor-based pagination
StreamingResults Server-sent event streaming

7. Error Handling Tests (5 tests)

Test Purpose
InvalidJSON Malformed JSON handling
MissingRequiredField Missing field validation
NonExistentEndpoint 404 error handling
MethodNotAllowed 405 error handling
RequestTimeout Timeout recovery

8. Monitoring & Metrics Tests (3 tests)

Test Purpose
GetMetrics Retrieve performance metrics
GetDetailedMetrics Detailed metrics endpoint
GetPrometheusMetrics Prometheus format export

9-10. Performance Tests (5 tests)

Test Purpose
HealthCheckLatency Measure endpoint latency
ConcurrentRequests 10-thread concurrent load test
BatchInsertPerformance Bulk operation throughput
QueryLatency Query execution timing
TransactionOverhead TX lifecycle overhead

11. Integration Tests (2 tests)

Test Purpose
FullCRUDWorkflow Create → Insert → Query → Delete
TransactionWorkflow Begin → Perform ops → Commit

12. Multi-Model Tests (4 tests)

Test Purpose
RelationalQueryWithAQL SQL-like queries via AQL
DocumentQueryWithJSON Document-oriented queries
VectorNearestNeighborSearch Vector similarity (k-NN)
GraphPatternMatching Graph traversal queries

Quick Start

Prerequisites

  • CMake 3.16+
  • C++ compiler with C++17 support
  • GoogleTest (gtest) library
  • nlohmann_json library
  • cpp_httplib library
  • ThemisDB server binary

Linux/macOS

# 1. Clone repository (if not already done)
cd ~/projects/themisdb

# 2. Build test suite
cmake --build build-release --target test_connector_mode_api

# 3. Run tests with automatic server startup
bash scripts/run_connector_api_tests.sh

# 4. Or run against existing server
bash scripts/run_connector_api_tests.sh --skip-server

Windows (PowerShell)

# 1. Navigate to repository
cd C:\VCC\themis

# 2. Build test suite
cmake --build build-msvc-ninja-release --target test_connector_mode_api

# 3. Run tests with automatic server startup
.\scripts\run_connector_api_tests.ps1

# 4. Or run against existing server
.\scripts\run_connector_api_tests.ps1 -SkipServer

# 5. Optional: run with docs + model configuration for LLM/RAG/LoRA E2E
.\scripts\run_connector_api_tests.ps1 `
  -DocsDir .\docs `
  -ModelId llama-3.1-8b `
  -ModelPath .\models\llama-3.1-8b.gguf `
  -LoraBaseModel llama-3.1-8b

Connector E2E Environment Variables

The live connector tests use these variables:

  • THEMIS_CONNECTOR_TEST_HOST
  • THEMIS_CONNECTOR_TEST_PORT
  • THEMIS_CONNECTOR_TEST_TIMEOUT_MS
  • THEMIS_CONNECTOR_TEST_BEARER_TOKEN (optional)
  • THEMIS_CONNECTOR_TEST_DOCS_DIR (defaults to docs)
  • THEMIS_CONNECTOR_TEST_MODEL_ID (required for model-load/inference test)
  • THEMIS_CONNECTOR_TEST_MODEL_PATH (optional model path for load endpoint)
  • THEMIS_CONNECTOR_TEST_LORA_BASE_MODEL (required for LoRA training trigger if model id is not set)

Detailed Setup

1. Build Dependencies

Ensure all required libraries are available:

# Ubuntu/Debian
sudo apt-get install -y \
    cmake \
    build-essential \
    nlohmann-json3-dev \
    libgtest-dev

# macOS
brew install cmake nlohmann-json googletest

# vcpkg (cross-platform)
vcpkg install nlohmann-json cpp-httplib gtest

2. CMake Configuration

Add the following to your CMakeLists.txt:

# Include connector test suite
include(tests/CMakeLists_connector_tests.txt)

# Or build individually
add_executable(test_connector_mode_api tests/test_connector_mode_api.cpp)
target_link_libraries(test_connector_mode_api
    PUBLIC gtest gtest_main nlohmann_json::nlohmann_json cpp_httplib::cpp_httplib
)

3. Configuration Environment Variables

# Test connection parameters
export THEMIS_TEST_HOST=127.0.0.1
export THEMIS_CONNECTOR_TEST_PORT=8765
export THEMIS_CONNECTOR_TEST_TIMEOUT_MS=30000  # milliseconds

# Authentication
export THEMIS_TEST_USER=admin
export THEMIS_TEST_PASS=admin
export THEMIS_TEST_TOKEN=test-api-token-12345

# Database path
export THEMIS_TEST_DB_PATH=./data/test_connector_db

Running Tests

Option 1: Using Automated Script (Recommended)

The provided scripts automatically start the server and run tests:

# Linux/macOS
./scripts/run_connector_api_tests.sh

# Windows PowerShell
.\scripts\run_connector_api_tests.ps1

Option 2: Manual Test Execution

# Terminal 1: Start server
themis_server --host 127.0.0.1 --port 8765 --data-dir ./test_db

# Terminal 2: Run tests
./build-release/bin/test_connector_mode_api

# Or with specific filters
./build-release/bin/test_connector_mode_api \
    --gtest_filter="ConnectorCRUDTest.*"

# Or run specific test
./build-release/bin/test_connector_mode_api \
    --gtest_filter="ConnectorConnectionTest.HealthCheck"

Test Filtering Options

# Run only connection tests
--gtest_filter="ConnectorConnectionTest.*"

# Run only CRUD tests
--gtest_filter="ConnectorCRUDTest.*"

# Run all except error tests
--gtest_filter="-*ErrorHandling*"

# Run specific test class
--gtest_filter="ConnectorQueryTest*"

# Run tests matching pattern
--gtest_filter="*Latency*"

Output Formats

# Colored output (default)
./test_binary --gtest_color=yes

# Print test names
./test_binary --gtest_print_time=1

# XML output for CI/CD
./test_binary --gtest_output=xml:test_results.xml

# JSON output
./test_binary --gtest_output=json:test_results.json

Test Categories

By Difficulty Level

Tier 1: Basic Connectivity

  • HealthCheck
  • ConnectWithValidUri
  • MultipleConnectionRetries

Tier 2: Authentication

  • LoginWithValidCredentials
  • RequestWithValidAuthToken

Tier 3: CRUD Operations

  • InsertDocument
  • GetDocument
  • UpdateDocument
  • DeleteDocument

Tier 4: Advanced Queries

  • ExecuteAQLQuery
  • ExecuteGraphQuery
  • ExecuteVectorSearch

Tier 5: Complex Workflows

  • FullCRUDWorkflow
  • TransactionWorkflow
  • ConcurrentRequests

By Test Type

Performance Tests

  • HealthCheckLatency
  • ConcurrentRequests
  • BatchInsertPerformance
  • QueryLatency

Stability Tests

  • MultipleConnectionRetries
  • RequestTimeout
  • InvalidJSON
  • MissingRequiredField

Security Tests

  • LoginWithInvalidCredentials
  • RequestWithoutAuthToken
  • InvalidAuthToken

Integration Tests

  • FullCRUDWorkflow
  • TransactionWorkflow
  • CrossModelQuery

Configuration

Server Configuration

# Custom server settings
export THEMIS_API_HOST=localhost
export THEMIS_API_PORT=9000
export THEMIS_API_TIMEOUT=60000

# Database options
export THEMIS_DB_PATH=/var/lib/themis/test
export THEMIS_DB_CACHE_SIZE=512
export THEMIS_DB_BLOCK_SIZE=4096

# Performance tuning
export THEMIS_API_THREADS=4
export THEMIS_CONN_POOL_SIZE=20
export THEMIS_MAX_BATCH_SIZE=1000

Test Configuration

# Test execution parameters
export GTEST_REPEAT=3          # Run each test 3 times
export GTEST_SHUFFLE=1        # Randomize test order
export GTEST_DEATH_TEST_STYLE=fast

# Parallelism (if supported)
export THEMIS_TEST_PARALLEL=4

# Logging
export THEMIS_TEST_LOG_LEVEL=debug
export THEMIS_TEST_LOG_FILE=test.log

Troubleshooting

Issue: Port Already in Use

# Find process using port
lsof -i :8765          # macOS/Linux
Get-Process -Id (Get-NetTCPConnection -LocalPort 8765).OwningProcess  # Windows

# Kill the process
kill -9 <PID>          # macOS/Linux
Stop-Process -Id <PID> -Force  # Windows

Issue: Server Doesn't Start

# Check if themis binary exists
which themis           # macOS/Linux
where themis           # Windows

# Add to PATH if needed
export PATH=$PATH:/path/to/build/bin  # macOS/Linux
$env:PATH += ";C:\path\to\build\bin"  # Windows

# Start server manually with verbose output
themis-server --host 127.0.0.1 --port 8080 --loglevel debug

Issue: Tests Timeout

# Increase timeout value
export THEMIS_CONNECTOR_TEST_TIMEOUT_MS=60000  # 60 seconds

# Run with verbose output
./test_binary --gtest_filter="*" -v

# Check server logs for issues
tail -f server.log

Issue: Connection Refused

# Verify server is running
curl http://127.0.0.1:8765/health

# Check firewall
sudo iptables -L | grep 8080        # Linux
Get-NetFirewallRule -Name "*8080*"  # Windows

# Try localhost instead of 127.0.0.1
export THEMIS_TEST_HOST=localhost

Issue: Authentication Failed

# Check default credentials
export THEMIS_TEST_USER=admin
export THEMIS_TEST_PASS=admin

# Generate new token if needed
curl -X POST http://localhost:8765/api/v1/auth/login \
  -H "Content-Type: application/json" \
  -d '{"username":"admin","password":"admin"}'

Performance Metrics

Baseline Performance (on typical hardware)

Operation Expected Latency Throughput
Health Check < 10ms > 100 req/s
Simple Query 50-100ms > 20 req/s
Batch Insert (100 docs) 200-500ms > 200 docs/s
Vector Search (768-dim) 100-200ms > 10 req/s
Graph Pattern Match 50-150ms > 20 req/s

Test Execution Time

Test Suite Expected Duration Test Count
Connection 2-5 seconds 5
Auth 3-8 seconds 4
CRUD 5-15 seconds 7
Query 10-30 seconds 5
Transactions 5-12 seconds 3
Error Handling 3-10 seconds 5
Full Suite 60-120 seconds 50+

CI/CD Integration

GitHub Actions

name: Connector API Tests

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Build
        run: |
          cmake --build build-release --target test_connector_mode_api
      - name: Run Tests
        run: |
          bash scripts/run_connector_api_tests.sh
        timeout-minutes: 10
      - name: Upload Results
        if: always()
        uses: actions/upload-artifact@v2
        with:
          name: test-results
          path: test_results.xml

GitLab CI

connector_api_tests:
  stage: test
  script:
    - cmake --build build-release --target test_connector_mode_api
    - bash scripts/run_connector_api_tests.sh
  timeout: 10 minutes
  artifacts:
    reports:
      junit: test_results.xml

Support & Contribution

For issues or improvements:

  1. File issue in GitHub Issues
  2. Include test logs: --gtest_print_time=1 output
  3. Server logs: themisd.log
  4. Environment details: OS, CMake version, build dir

Last Updated: April 5, 2026
Maintained by: ThemisDB Development Team
Status: ✅ Production Ready (v1.0.0)