Architecture¶

COSMIC uses a 6-stage pipeline to create semantically coherent chunks.

Pipeline Overview¶

Document → Structure Analysis → Semantic Boundaries → Domain Classification
                                                              ↓
              Reference Linking ← LLM Verification ← Boundary Fusion
                      ↓
               COSMICChunks (with rich metadata)

Stage 1: Structure Analysis¶

Location: src/cosmic/pipeline/structure.py

Detects structural elements in the document:

Headings (H1-H6)
Lists (ordered and unordered)
Tables
Code blocks
Paragraphs

Output:

Structure score (0-1)
Structural element positions
Recommended processing pathway

Configuration:

structure:
  enabled: true
  full_threshold: 0.7
  semantic_threshold: 0.4

Stage 2: Semantic Boundary Detection¶

Location: src/cosmic/pipeline/semantic.py

Computes Discourse Coherence Score (DCS) between adjacent sentences.

DCS Formula¶

DCS = α × topical_coherence + β × coreference_density + γ × discourse_signal

Components:

Component	Weight	Description
Topical coherence	α = 0.4	Embedding similarity between sentences
Coreference density	β = 0.35	Entity continuity across sentences
Discourse signal	γ = 0.25	Presence of discourse markers

Lower DCS → Higher boundary confidence

Boundaries are detected where DCS drops below the threshold.

Stage 3: Domain Classification¶

Location: src/cosmic/pipeline/domain.py

Uses MST-based clustering to classify chunks into domains.

Process:

Compute embeddings for candidate chunks
Build minimum spanning tree
Cluster using edge weight thresholds
Match clusters to domain taxonomy

Domain Taxonomy: configs/taxonomies/default.yaml

Supports domains like:

Technical
Medical
Legal
Scientific
General

Stage 4: Boundary Fusion¶

Location: src/cosmic/pipeline/fusion.py

Combines structural and semantic signals.

Formula:

fused_score = w_struct × structural_score + w_semantic × semantic_score

Default weights:

Structural: 0.6
Semantic: 0.4

Boundaries with fused_score > acceptance_threshold are kept.

Stage 5: LLM Verification¶

Location: src/cosmic/pipeline/verification.py

Optional verification of uncertain boundaries using an LLM.

Process:

Identify boundaries with confidence < 0.8
Send context to LLM for verification
LLM confirms or rejects boundary
Update boundary list

Supported providers:

OpenAI-compatible APIs
Ollama (local)

Skip with: --no-llm flag

Stage 6: Reference Linking¶

Location: src/cosmic/pipeline/reference.py

Detects and resolves cross-references between chunks.

Methods:

Explicit references - Regex patterns for "see section X", "as mentioned above"
Coreference resolution - spaCy-based entity linking

Output:

references_chunks: IDs this chunk references
referenced_by_chunks: IDs that reference this chunk
has_unresolved_references: Boolean flag

Skip with: --no-reference flag

Fallback Chain¶

When stages fail, COSMIC degrades gracefully:

Full COSMIC → Semantic-only → Sliding window → Fixed-length
(all stages)   (DCS only)     (similarity)     (token split)

Each level maintains functionality while reducing complexity.

Key Components¶

COSMICChunker¶

Location: src/cosmic/chunker.py

Main orchestrator that:

Lazy-initializes pipeline components
Implements strategy selection
Coordinates stage execution
Handles fallbacks

COSMICChunk¶

Location: src/cosmic/core/chunk.py

Frozen dataclass output containing:

Content and position
Domain classification
Quality metrics
Cross-references
Intent analysis
Provenance metadata

Document¶

Location: src/cosmic/core/document.py

Input representation with:

Sentence segmentation
Page tracking
Metadata storage

DiscourseCoherenceScorer¶

Location: src/cosmic/scoring/dcs.py

Computes DCS scores using:

Sentence-transformers embeddings
spaCy coreference chains
Discourse marker detection

EmbeddingModel¶

Location: src/cosmic/models/embeddings.py

Sentence-transformers wrapper with:

LRU cache for efficiency
Batch processing
GPU/CPU support

Project Structure¶

src/cosmic/
├── core/           # Data structures
│   ├── chunk.py    # COSMICChunk dataclass
│   ├── config.py   # Configuration system
│   ├── document.py # Document representation
│   └── enums.py    # Enumerations
│
├── pipeline/       # 6 pipeline stages
│   ├── structure.py    # Stage 1
│   ├── semantic.py     # Stage 2
│   ├── domain.py       # Stage 3
│   ├── fusion.py       # Stage 4
│   ├── verification.py # Stage 5
│   └── reference.py    # Stage 6
│
├── scoring/        # Scoring algorithms
│   ├── dcs.py      # Discourse Coherence Score
│   └── clustering.py # MST clustering
│
├── models/         # ML model wrappers
│   ├── embeddings.py # Sentence-transformers
│   ├── llm.py        # LLM client
│   ├── ollama.py     # Ollama integration
│   └── coreference.py # spaCy coreference
│
├── fallback/       # Degradation strategies
│   ├── semantic_only.py
│   ├── sliding_window.py
│   └── fixed_length.py
│
├── chunker.py      # Main entry point
├── cli.py          # Command-line interface
└── batch.py        # Batch processing