Introducing Our New Co-Scientist Architecture

When we started building LabCollector, the goal was simple: give labs one place to capture everything — samples, experiments, protocols, inventory, results. Years later, that mission has succeeded almost too well. Our customers' LabCollector instances now hold an extraordinary depth of institutional knowledge: thousands of ELN pages, LIMS records, attachments, SOPs, and scanned documents.

The problem we kept hearing from you wasn't "we don't have enough data." It was: "We have all this data — why is it still so hard to find the answer?"

That question is what led us to build the Co-Scientist Architecture — the retrieval and AI reasoning layer now powering smart search across LabCollector. I want to walk you through how it works, because I think the "how" matters as much as the "what" when you're trusting an AI system with your scientific data.

LabCollector Co-Scientist Architecture — a four-stage pipeline from Ingestion & Indexing through Embeddings & RAG, Query Orchestration, and Hybrid Retrieval & Intent Routing, turning raw documents into trusted insights. — The Co-Scientist architecture — four stages, each with one job, turning raw documents into trusted answers.

The problem with bolting AI onto a database

A chatbot that answers questions is easy to demo. A chatbot that answers questions correctly, respects who's allowed to see what, and doesn't quietly make things up when it's unsure — that's a much harder engineering problem, and it's the one we chose to solve properly rather than ship a shortcut.

So instead of a single black-box model, we built a four-stage pipeline. Each stage has one job, does it well, and hands off clean, structured information to the next.

1. Ingestion & Indexing

This stage pulls in everything LabCollector already knows about — ELN pages (titles, structured bodies), LIMS records across every module, and attachments in whatever format they arrive in: PDF, Office documents, images. Scanned lab notebooks and paper-derived PDFs go through OCR so that handwritten and printed content becomes searchable text, not a locked image. Everything is then normalized — cleaned, deduplicated, and structured — before it's upserted into our smartsearch_documents index with status and content hashes, so we only ever reprocess what's actually changed.

2. Embeddings & RAG

Once content is clean, we convert it into a form a language model can actually reason over. Documents are chunked (with sensible caps so no single record dominates), then passed through an embeddings model to capture their semantic meaning as vectors, stored in our smartsearch_embeddings layer. Semantic search can be toggled on or off per deployment, and indexing is budget-aware — it tracks coverage as it climbs and stops cleanly when a processing budget is reached, rather than running unchecked.

3. Query Orchestration

The orchestration layer takes the question plus conversation history, checks a fast answer cache first (so repeated or similar questions don't recompute everything from scratch), and if needed, hands off to an agent that plans out how to answer — routing to the right tools: safe, read-only SQL, search, metrics, list, or extract operations against your MySQL data, embeddings, and the underlying LLM. The answer is then streamed back to the interface in real time, with a short-lived cache so the system stays fast without staying stale.

4. Hybrid Retrieval & Intent Routing

A raw question is rewritten to expand acronyms and synonyms scientists use every day, then searched two ways at once: lexical (exact full-text matching) and semantic (vector similarity), with results reranked and merged. Intent routing selects the right tool for the type of question being asked, and — critically — ACL filtering runs before anything reaches the user, so results are always scoped to what that person is actually permitted to see. What comes out the other end is a ranked, grounded answer — not a plausible-sounding guess.

Why this design, and not something simpler

We could have shipped a single embedding-search-plus-chatbot loop much faster. We didn't, because in a lab environment, "close enough" isn't good enough. Every part of this architecture was built around a few non-negotiables:

Permission-scoped answers

Nobody sees data through the AI that they couldn't already see directly.

Hybrid search

Lexical and semantic retrieval together, because scientific terminology needs both exact matches and conceptual understanding.

OCR-ready ingestion

Scanned and handwritten records are first-class citizens, not blind spots.

Cached responses

Fast answers without wasting compute on repeated questions.

Grounded scientific insights

Answers are traceable back to your actual records, not generated from thin air.

Anonymized data

No confidential data is shared beyond what the architecture needs to function.

From raw documents to trusted answers

That's the phrase we kept coming back to while designing this system, and it's become our north star for what AI in LabCollector should mean: not a novelty feature, but a genuinely trustworthy way to turn the documents you already have into answers you can actually rely on.

This is the direction we're building LabCollector in — and it's just the beginning. We're excited for you to try it.

— Pierre

Founder & CEO, AgileBio