Testing Guide

Complete guide to testing unstructuredDataHandler

📋 Overview

unstructuredDataHandler includes comprehensive test suites covering:

Unit Tests: Component-level testing
Integration Tests: Multi-component workflow testing
Smoke Tests: Basic functionality validation
E2E Tests: End-to-end workflow testing
Manual Tests: Interactive validation and quality checks

Test Results Summary

Total Tests: 233
Passing: 203 (87.2%)
Failed: 14 (test infrastructure issues, not bugs)
Skipped: 15 (require external services)
Duration: ~50 seconds

🚀 Quick Start

Run All Tests

# Set Python path
export PYTHONPATH=.

# Run full test suite
python -m pytest test/ -v

# With coverage
python -m pytest test/ -v --cov=src/ --cov-report=xml

Run Specific Test Suites

# Unit tests only
python -m pytest test/unit/ -v

# Integration tests
python -m pytest test/integration/ -v

# Smoke tests
python -m pytest test/smoke/ -v

# E2E tests
python -m pytest test/e2e/ -v

Run Tests by Category

# All agent tests
python -m pytest test/ -k "agent" -v

# All parser tests
python -m pytest test/ -k "parser" -v

# All LLM tests
python -m pytest test/ -k "llm" -v

# All requirements extraction tests
python -m pytest test/ -k "requirements" -v

📊 Test Suite Organization

Test Directory Structure

test/
├── unit/                       # Component unit tests
│   ├── test_agents.py         # Agent tests
│   ├── test_document_parser.py # Parser tests
│   ├── test_llm_clients.py    # LLM client tests
│   ├── test_requirements.py   # Requirements extractor tests
│   └── ...
├── integration/                # Integration tests
│   ├── test_document_agent_integration.py
│   ├── test_requirements_extraction.py
│   └── ...
├── smoke/                      # Basic functionality tests
│   ├── test_imports.py        # Import validation
│   ├── test_config.py         # Configuration loading
│   └── ...
├── e2e/                        # End-to-end tests
│   ├── test_full_workflow.py  # Complete workflows
│   └── ...
├── manual/                     # Manual test scripts
│   ├── helper_function_verification.py
│   ├── quick_integration_test.py
│   ├── quality_validation.py
│   └── unit_test_helpers.py
└── benchmark/                  # Performance tests
    └── benchmark_performance.py

🧪 Unit Tests

Running Unit Tests

# All unit tests
PYTHONPATH=. python -m pytest test/unit/ -v

# Specific module
PYTHONPATH=. python -m pytest test/unit/test_document_parser.py -v

# Specific test
PYTHONPATH=. python -m pytest test/unit/test_document_parser.py::test_parse_pdf -v

Unit Test Coverage

Component	Tests	Status	Coverage
Agents	25	✅ Passing	85%
Parsers	15	✅ Passing	90%
LLM Clients	20	✅ Passing	80%
Requirements Extractor	30	✅ Passing	95%
Memory	10	✅ Passing	75%
Skills	15	✅ Passing	80%
Utils	20	✅ Passing	90%

Expected Results

$ PYTHONPATH=. python -m pytest test/unit/ -v

============== test session starts ==============
collected 135 items

test/unit/test_agents.py::test_base_agent PASSED
test/unit/test_document_parser.py::test_parse_pdf PASSED
...

============== 135 passed in 15.23s ==============

🔗 Integration Tests

Running Integration Tests

# All integration tests
PYTHONPATH=. python -m pytest test/integration/ -v

# Requirements extraction integration
PYTHONPATH=. python -m pytest test/integration/test_requirements_extraction.py -v

# Document agent integration
PYTHONPATH=. python -m pytest test/integration/test_document_agent_integration.py -v

Integration Test Scenarios

Requirements Extraction Workflow

def test_requirements_extraction_full_workflow():
    """Test complete requirements extraction from PDF."""
    
    # 1. Parse document
    parser = EnhancedDocumentParser()
    markdown, attachments = parser.get_docling_markdown("test.pdf")
    
    # 2. Chunk markdown
    chunks = parser.split_markdown_for_llm(markdown)
    
    # 3. Extract requirements
    agent = DocumentAgent()
    result = agent.extract_requirements(
        file_path="test.pdf",
        enable_quality_enhancements=True
    )
    
    # 4. Verify results
    assert result["success"]
    assert len(result["requirements"]) > 0
    assert "quality_metrics" in result

Expected Results

$ PYTHONPATH=. python -m pytest test/integration/ -v

============== test session starts ==============
collected 21 items

test/integration/test_requirements_extraction.py::test_full_workflow PASSED
test/integration/test_document_agent_integration.py::test_batch_processing PASSED
...

============== 20 passed, 1 skipped in 45.12s ==============

💨 Smoke Tests

Running Smoke Tests

# All smoke tests
PYTHONPATH=. python -m pytest test/smoke/ -v

Smoke Test Categories

Import Validation

def test_all_imports():
    """Verify all modules can be imported."""
    from src.agents.document_agent import DocumentAgent
    from src.parsers.enhanced_document_parser import EnhancedDocumentParser
    from src.llm.ollama_client import OllamaClient
    # ... etc

Configuration Loading

def test_config_loads():
    """Verify configuration files load successfully."""
    from src.utils.config_loader import load_llm_config
    
    config = load_llm_config()
    assert config["provider"]
    assert config["model"]

Expected Results

$ PYTHONPATH=. python -m pytest test/smoke/ -v

============== test session starts ==============
collected 10 items

test/smoke/test_imports.py::test_all_imports PASSED
test/smoke/test_config.py::test_config_loads PASSED
...

============== 10 passed in 2.45s ==============

🎯 E2E Tests

Running E2E Tests

# All E2E tests
PYTHONPATH=. python -m pytest test/e2e/ -v

# Specific workflow
PYTHONPATH=. python -m pytest test/e2e/test_full_workflow.py -v

E2E Test Scenarios

Complete Requirements Extraction

@pytest.mark.e2e
def test_complete_extraction_workflow():
    """Test end-to-end requirements extraction."""
    
    # Setup
    agent = DocumentAgent()
    test_pdf = "test/fixtures/requirements_sample.pdf"
    
    # Execute
    result = agent.extract_requirements(
        file_path=test_pdf,
        enable_quality_enhancements=True,
        auto_approve_threshold=0.75
    )
    
    # Validate
    assert result["success"]
    assert len(result["requirements"]) > 0
    
    # Check quality metrics
    metrics = result["quality_metrics"]
    assert metrics["average_confidence"] > 0.70
    assert metrics["auto_approve_count"] > 0
    
    # Check requirement structure
    for req in result["requirements"]:
        assert "requirement_id" in req
        assert "requirement_body" in req
        assert "confidence" in req

🖐️ Manual Tests

Manual Test Scripts

Located in test/manual/:

1. Helper Function Verification

File: test/manual/helper_function_verification.py

Purpose: Test individual utility functions

Usage:

PYTHONPATH=. python test/manual/helper_function_verification.py

Tests:

Markdown splitting
Heading parsing
Section merging
JSON extraction
Requirement normalization

2. Quick Integration Test

File: test/manual/quick_integration_test.py

Purpose: E2E test with real documents

Usage:

PYTHONPATH=. python test/manual/quick_integration_test.py

Workflow:

Parse test document
Extract requirements
Display results
Save to JSON

3. Quality Validation

File: test/manual/quality_validation.py

Purpose: Human validation of quality metrics

Usage:

PYTHONPATH=. python test/manual/quality_validation.py

Features:

Interactive requirement review
Confidence score validation
Quality flag verification
Manual approval workflow

4. Unit Test Helpers

File: test/manual/unit_test_helpers.py

Purpose: Test specific helper functions

Usage:

PYTHONPATH=. python test/manual/unit_test_helpers.py

📈 Performance Benchmarks

Running Benchmarks

File: test/benchmark/benchmark_performance.py

Usage:

PYTHONPATH=. python test/benchmark/benchmark_performance.py

Requirements:

Test documents in test/fixtures/
Ollama server running
qwen2.5:7b model installed

Metrics:

Parse time per document
Chunking time
LLM inference time
Total extraction time
Accuracy metrics
Confidence distributions

Sample Output:

Document Parsing Benchmarks
===========================
Document: requirements_sample.pdf (25 pages)
  Parse time: 5.23s
  Chunk time: 0.45s
  LLM time: 23.67s
  Total time: 29.35s
  
  Requirements extracted: 108
  Average confidence: 0.965
  Auto-approved: 108 (100%)

🔧 Test Configuration

pytest.ini

[pytest]
testpaths = test
python_files = test_*.py
python_classes = Test*
python_functions = test_*

markers =
    unit: Unit tests
    integration: Integration tests
    smoke: Smoke tests
    e2e: End-to-end tests
    slow: Slow tests (>10s)
    requires_ollama: Requires Ollama server
    requires_api_key: Requires cloud API key

addopts =
    -v
    --tb=short
    --strict-markers
    --disable-warnings

Running with Markers

# Only unit tests
pytest -m unit

# Only fast tests
pytest -m "not slow"

# Tests that don't require Ollama
pytest -m "not requires_ollama"

# Integration tests only
pytest -m integration

🎯 Testing Best Practices

Before Running Tests

# 1. Set Python path
export PYTHONPATH=.

# 2. Install dependencies
pip install -r requirements-dev.txt

# 3. Check Ollama (if needed)
lsof -i :11434

# 4. Verify config
ls -l config/model_config.yaml

Writing New Tests

import pytest
from src.agents.document_agent import DocumentAgent

class TestDocumentAgent:
    """Test suite for DocumentAgent."""
    
    @pytest.fixture
    def agent(self):
        """Create agent instance."""
        return DocumentAgent()
    
    def test_extract_requirements_success(self, agent):
        """Test successful requirements extraction."""
        result = agent.extract_requirements(
            file_path="test/fixtures/sample.pdf",
            enable_quality_enhancements=False
        )
        
        assert result["success"]
        assert len(result["requirements"]) > 0
    
    @pytest.mark.slow
    @pytest.mark.requires_ollama
    def test_quality_enhancements(self, agent):
        """Test quality enhancements (slow)."""
        result = agent.extract_requirements(
            file_path="test/fixtures/sample.pdf",
            enable_quality_enhancements=True
        )
        
        assert "quality_metrics" in result
        assert result["quality_metrics"]["average_confidence"] > 0

🐛 Troubleshooting Tests

"ModuleNotFoundError"

Solution: Set PYTHONPATH

export PYTHONPATH=.
python -m pytest test/ -v

"Connection refused" (Ollama tests)

Solution: Start Ollama or skip those tests

# Start Ollama
ollama serve &

# Or skip Ollama tests
pytest -m "not requires_ollama"

"API key not found" (Cloud provider tests)

Solution: Set API key or skip those tests

# Set API key
export OPENAI_API_KEY="your-key"

# Or skip API key tests
pytest -m "not requires_api_key"

Tests Hanging

Solution: Use timeout and verbose output

# Add timeout
pytest --timeout=60 test/

# Increase verbosity
pytest -vv test/

Inconsistent Test Results

Solution: Run tests in isolation

# Run each test file separately
for file in test/unit/*.py; do
    pytest "$file" -v
done

📊 Test Coverage

Generate Coverage Report

# Generate coverage
PYTHONPATH=. python -m pytest test/ --cov=src/ --cov-report=html

# Open report
open htmlcov/index.html

Coverage Goals

Component	Current	Target
Agents	85%	90%
Parsers	90%	95%
LLM Clients	80%	85%
Requirements	95%	95% ✅
Overall	87%	90%

✅ CI/CD Integration

GitHub Actions Workflows

Python Tests

File: .github/workflows/python-test-static.yml

Runs:

Unit tests
Integration tests
Smoke tests
Coverage report
Mypy type checking

Triggers: Push, Pull Request

Pylint

File: .github/workflows/pylint.yml

Runs: Code quality checks (Python 3.10-3.13)

Triggers: Push, Pull Request

Local CI Simulation

# Run what CI runs
PYTHONPATH=. python -m pytest test/ -v --cov=src/
python -m mypy src/ --ignore-missing-imports
python -m pylint src/ --exit-zero

🔗 Related Documentation

Quick Start: doc/user-guide/quick-start.md
Configuration: doc/user-guide/configuration.md
Development Setup: doc/developer-guide/development-setup.md
Contributing: CONTRIBUTING.md

Last Updated: 2025-01-XX
Version: 2.0

Uh oh!

FilesExpand file tree

testing.md

Latest commit

History

testing.md

File metadata and controls

Testing Guide

📋 Overview

Test Results Summary

🚀 Quick Start

Run All Tests

Run Specific Test Suites

Run Tests by Category

📊 Test Suite Organization

Test Directory Structure

🧪 Unit Tests

Running Unit Tests

Unit Test Coverage

Expected Results

🔗 Integration Tests

Running Integration Tests

Integration Test Scenarios

Requirements Extraction Workflow

Expected Results

💨 Smoke Tests

Running Smoke Tests

Smoke Test Categories

Import Validation

Configuration Loading

Expected Results

🎯 E2E Tests

Running E2E Tests

E2E Test Scenarios

Complete Requirements Extraction

🖐️ Manual Tests

Manual Test Scripts

1. Helper Function Verification

2. Quick Integration Test

3. Quality Validation

4. Unit Test Helpers

📈 Performance Benchmarks

Running Benchmarks

🔧 Test Configuration

pytest.ini

Running with Markers

🎯 Testing Best Practices

Before Running Tests

Writing New Tests

🐛 Troubleshooting Tests

"ModuleNotFoundError"

"Connection refused" (Ollama tests)

"API key not found" (Cloud provider tests)

Tests Hanging

Inconsistent Test Results

📊 Test Coverage

Generate Coverage Report

Coverage Goals

✅ CI/CD Integration

GitHub Actions Workflows

Python Tests

Pylint

Local CI Simulation

🔗 Related Documentation