AI-Powered Spreadsheet & Document Analysis with Multi-Agent Orchestration and Cross-Document Reference

We built a multi-agent AI document intelligence platform with vector database-backed retrieval for large documents, specialized agents for different document types, an orchestrator for cross-document reasoning, and write-back capabilities for spreadsheet editing.

Architecture

• Orchestrator: AI orchestrator agent coordinating multi-step workflows across specialized agents
• Spreadsheet Agent: Handles Excel/CSV/Google Sheets analysis, formula generation, and cell edits
• Document Agent: Handles PDF/Word document reading, extraction, and summarization
• Cross-Reference Agent: Performs joins, comparisons, and reconciliation across document types
• Vector Database: Milvus for semantic indexing of document chunks and spreadsheet rows
• LLM Layer: Multi-model approach with function calling
• Backend: Python/FastAPI for document processing and agent orchestration
• Frontend: React dashboard with file upload, chat interface, and live spreadsheet preview
• Storage: S3 for original files, PostgreSQL for metadata and job tracking

Multi-Agent Architecture

Agent Roles

1. Orchestrator Agent

The central coordinator that receives user queries, decomposes them into sub-tasks, and delegates to specialized agents. It analyzes user intent, creates execution plans, manages data flow between agents, aggregates results, and handles error recovery.

2. Spreadsheet Agent

Specialized for tabular data operations including schema understanding, natural language to query translation, aggregations and filtering, formula generation, cell editing and column fills, chart suggestions, and data validation/anomaly detection.

3. Document Agent

Specialized for unstructured and semi-structured documents including OCR and layout-aware text extraction, section identification, key-value extraction from contracts, summarization, semantic clause search, and table extraction from PDFs/Word docs.

4. Cross-Reference Agent

Specialized for multi-document reasoning including entity matching across documents, data reconciliation and discrepancy identification, timeline analysis, dependency resolution for conflicting data, and SQL-like join operations across document types.

Vector Database Layer

Why Vector DB for Documents

Large documents and spreadsheets can't fit in a single LLM context window. The vector database enables semantic search across millions of rows and document chunks, retrieval of only relevant portions per query, cross-document entity linking via embedding similarity, and persistent indexing that doesn't need re-processing on every query.

Indexing Strategy

Spreadsheet Indexing:

Each row is converted to a natural language representation by concatenating key column values, then embedded and stored with references back to the original file, sheet, and row index for write-back operations.

Document Indexing:

Documents are extracted with layout awareness, chunked into semantic segments with overlap, embedded, and stored with references to the source file, section, and page number.

Cross-Document Entity Index:

A separate index links entities (vendors, products, people, invoice numbers) across documents, enabling cross-reference queries to quickly find all mentions of an entity regardless of source file.

Retrieval Pipeline

When a user asks a cross-document question, the orchestrator identifies which documents and agents are needed, performs vector searches to find relevant data across all sources, delegates to specialized agents for processing, and aggregates results into a coherent response.

Orchestration Engine

Query Decomposition

The orchestrator breaks complex queries into multi-step execution plans. For example, a question like "Find vendors with late deliveries, check contract penalty clauses, and calculate claimable penalties" would be decomposed into sequential steps: querying delivery data via the Spreadsheet Agent, searching contracts via the Document Agent, and joining results via the Cross-Reference Agent.

Agent Communication

• Agents communicate via structured messages with typed payloads
• The orchestrator maintains execution context with intermediate results
• Failed steps trigger retry or fallback strategies
• Partial results are returned if some steps complete but others fail

Spreadsheet Edit & Write-Back

Edit Capabilities

The platform supports cell updates, column fills, row insertion, conditional formatting, new sheet creation, and formula injection — all proposed by AI agents and applied with user approval.

Write-Back Pipeline

1. Agent determines the edit operation (which cells, what values)
2. Edit preview shown to user with diff highlighting (old vs. new values)
3. User approves or modifies the proposed changes
4. Backend applies changes to the file using appropriate libraries per format
5. Modified file saved as a new version with edit audit trail
6. Vector index updated for changed rows

Version Control

• Every edit creates a new file version (original preserved)
• Diff log shows exactly what changed, when, and why
• Rollback to any previous version with one click
• Edit attribution: which agent or user made each change

Processing Pipeline for New Documents

File Upload Flow

1. User uploads files (drag-and-drop or API)
2. File type detected and routed to appropriate processor
3. Spreadsheets: Parsed, schema inferred, rows embedded and indexed
4. PDFs: OCR (if scanned) → layout extraction → chunking → embedding → indexing
5. Word Docs: Text extraction → section parsing → chunking → embedding → indexing
6. Entity Extraction: NER identifies people, organizations, dates, amounts across all docs
7. Cross-Document Linking: Entity index updated with new mentions
8. File metadata stored in PostgreSQL, embeddings in vector DB, originals in S3

Supported Formats

The platform supports Excel, CSV, and Google Sheets (with full write-back), native and scanned PDFs (read-only), and Word docs and Google Docs (limited write-back).

Key Features

1. Multi-Agent Architecture — Specialized agents for spreadsheets, documents, and cross-referencing
2. AI Orchestrator — Decomposes complex queries into multi-step execution plans
3. Cross-Document Reference — Entity linking and data reconciliation across file types
4. Vector-Powered Retrieval — Semantic search handles datasets beyond LLM context limits
5. Spreadsheet Write-Back — AI edits cells, fills columns, and injects formulas with user approval
6. Large Dataset Support — 50,000+ row spreadsheets indexed and queryable via vector search
7. Version Control — Every edit versioned with diff log and rollback capability
8. Natural Language Queries — Ask complex analytical questions in plain English
9. Multi-Format Support — Excel, CSV, Google Sheets, PDF, Word, Google Docs
10. Edit Preview — Diff-highlighted preview before any changes are applied