Data Aggregator

🔎 data-aggregator-mcp

One MCP server to find and fetch research data across archives, omics registries, and literature — behind a single normalized model.

search one query across 17 sources — Zenodo, DataCite (Dryad / Figshare / Dataverse / OSF / OpenNeuro / Mendeley), NCBI omics (GEO / SRA / BioProject), BioStudies (EBI, incl. ArrayExpress), literature (PubMed / OpenAIRE), HuggingFace datasets, DataONE (eco / environmental), OmicsDI (proteomics / metabolomics), DANDI (neurophysiology), CZ CELLxGENE (single-cell), OpenML (ML datasets), RCSB PDB (structures), UniProtKB (proteins), the GWAS Catalog, GBIF (biodiversity), data.gov (US federal open data), and NASA CMR (Earth science) — deduplicated, normalized, and cross-linked. resolve any hit to its file manifest, citation, trust signals, and the data it points at. fetch it to disk with checksum verification.

mcp-name: io.github.musharna/data-aggregator-mcp

✨ Why this

Most data MCPs wrap a single source. This one unifies them behind six tools and one DataResource model, so an agent searches once and gets back comparable records:

• Multi-domain, one model — generalist archives + raw omics + literature, deduplicated by DOI (the fetchable record wins over bare metadata).
• Taxonomy synonym expansion — organism="Orobanche aegyptiaca" also matches Phelipanche aegyptiaca (NCBI Taxonomy), so a species rename doesn't cost you results.
• Paper → data bridge — resolve a paper and get links to the GEO / SRA / BioProject / DataCite records it produced.
• Verified fetch — streams to disk with md5 verification where the source exposes a checksum, optional archive unpacking, and a fail-loud integrity sniff that rejects an HTML paywall page served as a "PDF".
• Citations, access & full text — render a citation in any CSL style, get normalized access/license, and pull open-access full text — all in one resolve.
• Trust signals — usage metrics (citations / views / downloads / likes), version status (is_latest / superseded_by), and last_updated freshness, surfaced wherever the source exposes them.
• Interop exports — resolve(format="croissant") or "ro-crate" hands a dataset to an ML or research-packaging pipeline as standard JSON-LD.
• Operate on data in place — operate reads the schema, previews rows, or runs a read-only SQL SELECT against a remote Parquet/CSV/TSV without downloading it (Parquet footer + DuckDB httpfs range reads). Optional [operate] extra; base install is unchanged.
• Relate across records — relate takes a handful of resolved ids and reports how they connect — shared accession, shared cross-identifier, an explicit link, or version lineage — naming the literal shared value as evidence. Metadata hints only: it never reads files or executes a join.

→ Full rationale and a comparison vs. single-source servers, breadth gateways, and ML-dataset tools: docs/POSITIONING.md.

⚡ Quickstart

Run with no install:

uvx data-aggregator-mcp

Register with Claude Code:

claude mcp add data-aggregator -- uvx data-aggregator-mcp

A typical agent flow:

search("drought stress RNA-seq", organism="Sorghum bicolor")
  → [ geo:GSE..., sra:SRX..., zenodo:..., pubmed:... ]   # deduped, taxa-normalized

resolve("sra:SRX079566")
  → DataResource{ files: [ENA FASTQ urls…], access: "open", taxa: [...] }

fetch("sra:SRX079566", dest="./data")
  → ["./data/SRX079566_1.fastq.gz", …]                   # md5-verified

pip install data-aggregator-mcp
data-aggregator-mcp        # or: python -m data_aggregator_mcp

To use the operate tool (query remote tabular files in place), install the optional extra:

pip install "data-aggregator-mcp[operate]"

Add to a client's MCP config (e.g. Claude Desktop claude_desktop_config.json):

{
  "mcpServers": {
    "data-aggregator": {
      "command": "uvx",
      "args": ["data-aggregator-mcp"],
      "env": { "NCBI_API_KEY": "your-optional-key" }
    }
  }
}

🌐 Transports

stdio (default) — the server runs as a child of the client, so fetch() writes to your own disk. Nothing to configure; every command above uses it.

Streamable HTTP — the same six tools, prompts, and resources over HTTP:

data-aggregator-mcp --transport http     # → http://127.0.0.1:8000/mcp/

flag	default	notes
--transport {stdio,http}	stdio
--host	127.0.0.1	this machine only; any non-loopback value requires --allow-host
--port	8000
--allow-host HOST:PORT	auto on loopback	permitted Host header, repeatable — required off loopback
--allow-origin ORIGIN	derived	permitted browser Origin header, repeatable
--stateless	off	fresh transport per request, no session affinity
--json-response	off	plain JSON responses instead of SSE streams

The endpoint is served at /mcp/ — with the trailing slash. /mcp answers 307 redirecting there, which is fine for any client that follows redirects (a 307 preserves the POST body); point one that doesn't straight at /mcp/. In stateful mode, sessions idle for 30 minutes are reaped.

DNS-rebinding protection is always on. A loopback bind derives its own host/origin allowlist, so the default needs no configuration. A non-loopback bind (--host 0.0.0.0, a LAN address, a container interface) refuses to start without at least one explicit --allow-host — guessing an allowlist there is precisely the hole the protection exists to close, so it fails loud instead of open:

data-aggregator-mcp --transport http --host 0.0.0.0 \
  --allow-host data.example.org:8000

Once running, a request whose Host header is outside the allowlist is refused with 421 Invalid Host header.

⚠️ fetch(dest=…) writes to the server's filesystem, not the client's. Over stdio those are the same disk; over HTTP they may be different machines, and the caller gets back paths it cannot read. Treat dest on an HTTP deployment as server-side staging, or use stdio when you need the bytes locally. search, resolve, operate, relate, and list_sources are unaffected — they return data, not paths.

🗂️ Sources

Source	Discover	Fetch	Checksum
Zenodo	✅	✅	md5
DataCite → Figshare	✅	✅	md5
DataCite → Dataverse	✅	✅	md5
DataCite → OSF	✅	✅	md5
DataCite → Dryad	✅	manifest only¹	sha-256 (listed)
DataCite → Mendeley & others	✅	—	—
NCBI SRA	✅	✅ (ENA FASTQ)	md5
NCBI GEO	✅	✅ (suppl/)	none²
NCBI BioProject	✅	→ SRA links	—
PubMed / OpenAIRE	✅	✅ (OA full text)	none²
HuggingFace datasets	✅	✅ (resolve URL)	none
DataONE (eco/env)	✅	✅ (Member Node)	md5 / sha-256
OmicsDI → PRIDE	✅	✅ (HTTPS FTP)	size only
OmicsDI → MetaboLights	✅	✅ (HTTPS FTP)	none
OmicsDI → other MS repos	✅	—	—
DataCite → OpenNeuro	✅	✅ (snapshot)	none²
DANDI (neurophysiology)	✅	✅ (302→S3)	none²
CZ CELLxGENE (single-cell)	✅	✅ (H5AD/RDS)	none²
OpenML (ML datasets)	✅	✅ (ARFF)	md5
RCSB PDB (structures)	✅	✅ (.cif/.pdb)	none²
GWAS Catalog	✅	→ PMID bridge	—

¹ Dryad downloads are token / bot-challenge gated, so fetch fails loud; resolve still lists the files. ² No upstream checksum — fetch verifies content-type instead (rejects an HTML page served in place of a binary).

🛠️ Tools

search(query?, size?, sources?, organism?, disease?, tissue?, chemical?, assay?, kind?, published_after?, published_before?, rank?, cursor?, collapse_mirrors?, understand?, multi_query?, provenance?)

Fan out across all wired sources in parallel and return compact DataResource records, deduped by DOI. Per-source failures land in errors{} — never silently dropped.

• organism — expand the query with NCBI-Taxonomy synonyms; the expansion is echoed in taxon_expansion, and results carry normalized taxa[] ({taxid, name}) plus a described_in link to plant-genomics-mcp for plant taxa.
• sources — restrict the fan-out, e.g. ["omics"].
• size — max results (1–50).
• kind — keep only dataset / sequencing_run / study / publication / software.
• published_after / published_before — filter by publication year.
• rank — relevance (default) or semantic (re-rank the fetched page by embedding similarity to the query; needs EMBEDDING_API_BASE, degrades to relevance order otherwise).
• understand — opt into LLM query understanding (default false). A free-text query is normalized into a focused keyword query: conversational fluff ("I'm looking for…", "where can I find…") is stripped while the scientific and entity terms are kept so they still match by text. The LLM also detects structured entities (organism/disease/tissue/chemical/assay, kind) — these are echoed in query_understanding.extracted for transparency but not auto-applied, because ANDing LLM-inferred facets across free-text keyword upstreams over-constrains and hurts recall. Only the cleaned keyword_core and explicit year scopes are applied; the ontology resolvers still run on the facets you pass (the LLM proposes, you dispose). Needs an LLM endpoint (LLM_API_BASE); with none configured the search runs unchanged and notes it in errors['understand']. Effectiveness is query- and model-dependent — opt-in / default-off; validate the recall lift on your own corpus and LLM (see the eval harness below). On our small verified set multi_query= is the stronger, always-safe recall lever; understand= is approximately neutral with a weak local model.
• multi_query — opt into diverse multi-query recall expansion (default false). An LLM generates up to a few deliberately-diverse reformulations of your query (different facets/synonyms/framings, not paraphrases), each is fanned out across every source, and the deduped union is re-ranked against your original query — surfacing relevant records a single keyword query would miss. Bounded at MAX_QUERY_VARIANTS (4, incl. the original, which is always kept so recall never drops below baseline), so it costs at most N× the upstream calls. Composes with understand= (which structures variant 0). The variants used are echoed in query_expansion. Needs an LLM endpoint (LLM_API_BASE); with none configured the search runs as a normal single query and notes it in errors['multi_query'].
• cursor — opaque token from a prior result's next_cursor; pages forward across every source. In cursor mode the other params are read from the token, so query is optional.

resolve(id, cite?, format?, trust?, fair?, use?)

Full record + files manifest. Routes by id shape — zenodo:7654321, a bare DOI, datacite:10.5061/dryad.x, an omics id (sra:SRX079566, geo:GSE332789, bioproject:PRJNA1468572), a literature id (pubmed:34320281, openaire:<id>), a HuggingFace id (hf:owner/name), a DataONE id (dataone:doi:10.5063/F1HT2M7Q), or an OmicsDI id (omicsdi:pride:PXD000001). Attaches, where available:

• files[] — ENA FASTQ manifest (SRA), GEO suppl/, or the host repo's native manifest (Figshare / Dataverse / OSF / Dryad).
• links[] — paper → data: pubmed: → sra: / geo: / bioproject: (NCBI elink); openaire: → datacite: (ScholeXplorer Scholix).
• access / license — normalized status (open / embargoed / restricted / closed / unknown) and license where the source exposes it.
• identifiers — normalized {pmid, pmcid, doi}, plus an open-access full-text FileEntry (EuropePMC XML, or an Unpaywall PDF fallback) for papers.
• citation — pass cite=<format>: bibtex, ris, csl-json, or any CSL style name (apa, mla, vancouver, …). DOI records use content negotiation; others render CSL-JSON from metadata. Off by default; failures degrade quietly.
• trust signals — metrics (citations / views / downloads / likes), is_latest / superseded_by (derived from version links), and last_updated freshness, where the source provides them.
• trust=true — attach retraction status (via Crossref) under trust{}. One extra Crossref call; meaningful for DOI-bearing records only.
• fair=true — attach an RDA-grounded FAIRness score (0–100 + F/A/I/R sub-scores + actionable gaps) computed from the record metadata under fair{}. Pure/local — no extra network call.
• use=<intent> — attach a licence-compatibility advisory under license_compat{} for the intended use (commercial / redistribute / modify / ml-training). Returns ALLOW/REVIEW/DENY with the governing clause. Metadata-derived advisory, not legal advice; an absent/unrecognized licence yields REVIEW.
• format — pass format="croissant" (file-level Croissant JSON-LD), "ro-crate" (minimal RO-Crate 1.1), or "provenance" (one-call RO-Crate 1.1 data-availability dossier bundling version-currency, licence+SPDX, FAIR score, and retraction status) to attach a standard manifest under the matching field.

fetch(id, dest?, files?, max_bytes?, force?, extract?)

Download files to disk and return their paths. Streams under a max_bytes guard (force to override) with md5 verification wherever a checksum exists.

• files — restrict to a subset of the resolved manifest.
• extract — unpack downloaded zip / tar archives in place, guarded against path traversal and runaway extracted size. Off by default.
• Unverified fetches (GEO suppl/, literature full text) get a content-type sniff that fails loud if a declared binary is actually an HTML page.
• Fetchable: Zenodo, SRA, GEO, DataONE (Member-Node objects, md5/sha-256 verified), DataCite-hosted Figshare / Dataverse / OSF, HuggingFace datasets, PRIDE / MetaboLights (via OmicsDI, unverified), and literature open-access full text. Dryad, other DataCite repos, and other OmicsDI repos (MassIVE / GNPS / ...) are discovery-only and raise FetchNotSupportedError.

list_sources()

Wired sources with their capabilities — layer, kinds, supported filters, fetchability, operable flag, id examples, auth, and rate limits.

operate(op, id, file?, query?, n?, columns?)

Inspect or query a remote tabular file (Parquet / CSV / TSV) without downloading it. Addresses a file by catalog id + file name (defaults to the first tabular file on the resolved record). Ops:

• schema — column names + types (reads the Parquet footer / sniffs the CSV header; no full load).
• preview — a small sample of rows.
• head — the first n rows (default 20), optionally restricted to columns.
• sql — a read-only SELECT (the file is the view data), e.g. SELECT col, count(*) FROM data GROUP BY 1.
• peek — per-column profile via DuckDB SUMMARIZE (type, null-rate, approximate distinct count, min/max, numeric quartiles) without downloading the file. Like head/sql, reads the whole file and honors the source-size ceiling.

Backed by the Parquet footer reader + DuckDB httpfs range reads. sql runs in a locked-down DuckDB (read-only, local filesystem disabled, single-SELECT validation, row / wall-clock caps). Requires the optional [operate] extra (pip install data-aggregator-mcp[operate]); without it, operate returns a clear install-the-extra message and the other four tools are unaffected.

Any HuggingFace dataset with a datasets-server converted view is operable (schema / preview / head / sql): resolve surfaces the auto-converted Parquet files (source="hf-datasets-server") even for datasets stored as JSON/JSONL/arrow, so pass file=<config>/<split>/...parquet to pick a split when there are several.

relate(ids)

Cross-resource join/harmonization hints. Given 2–10 resource ids, relate resolves each (TTL-cached) and reports how they relate and on what key they could be joined:

• shared_accession — same BioProject/SRA/GEO accession on ≥2 records → joinable key.
• shared_identifier — same doi/pmid/pmcid across records → same work / paper↔data link.
• explicit_link — one record's links[] points at another input record.
• version_lineage — one record supersedes another (dedupe, don't join, those).

Hints only. relate never reads file columns, fetches files, or executes a join/merge/conversion — every hint names the shared value as evidence. Per-id resolve failures are reported in errors, not fatal; an empty result carries an explanatory note.

Prompts

Three workflow prompts surface in clients (e.g. /mcp__data_aggregator__* in Claude Code):

• find_data — find datasets for a topic, optionally scoped to an organism.
• data_behind_paper — find the datasets / accessions behind a paper.
• search_resolve_fetch — walk the end-to-end search → resolve → fetch flow.

⚙️ Configuration

All optional, set via environment variables:

• NCBI_API_KEY — raises the NCBI E-utilities rate limit (3 → 10 req/s) used by the omics, literature, and taxonomy lookups.
• DATA_GOV_API_KEY — a free api.data.gov key for the data.gov source. Absent ⇒ requests fall back to the shared public DEMO_KEY, rate-limited to roughly 30 requests/hour per IP — fine for light discovery, worth setting for anything heavier.
• UNPAYWALL_EMAIL — enables the Unpaywall fallback leg of literature full-text retrieval (the EuropePMC leg works without it).
• NCBI_EMAIL — contact address sent to NCBI's ID converter; falls back to UNPAYWALL_EMAIL when unset.
• DATAVERSE_BASE_URL — resolve Dataverse DOIs against a different installation (default https://dataverse.harvard.edu).
• CACHE_TTL_SECONDS — resolve-cache lifetime in seconds (default 3600; an unparseable value falls back to that default).
• EMBEDDING_API_BASE / EMBEDDING_API_KEY / EMBEDDING_MODEL — an OpenAI-compatible embeddings endpoint enabling rank=semantic. Absent ⇒ semantic re-rank degrades to relevance order. Key is optional (keyless local servers supported); model defaults to text-embedding-3-small.
• LLM_API_BASE / LLM_API_KEY / LLM_MODEL — an OpenAI-compatible /chat/completions endpoint enabling search(understand=true) (NL→structured query rewriting) and search(multi_query=true) (diverse multi-query recall expansion). Absent ⇒ both run the raw query unchanged and note it in errors['understand'] / errors['multi_query']. Key is optional (keyless local servers supported); model defaults to gpt-4o-mini (a passthrough string — set it to whatever your endpoint serves). multi_query fans out at most MAX_QUERY_VARIANTS (4, incl. the original) variants, bounding the N× cost.

To measure the recall lift of understand=true / multi_query=true on a small labeled set, run the gated eval harnesses (need a live LLM endpoint):

DATA_AGGREGATOR_MCP_LIVE=1 LLM_API_BASE=... python scripts/eval_understand.py
DATA_AGGREGATOR_MCP_LIVE=1 LLM_API_BASE=... python scripts/eval_multi_query.py

They print per-query and mean recall@20 (understand / multi-query off vs. on). See the fixtures at scripts/eval_understand_fixture.json and scripts/eval_multi_query_fixture.json.

🧪 Develop

uv venv && uv pip install -e ".[dev]"
uv run pytest -q
uv run ruff check src tests
DATA_AGGREGATOR_MCP_LIVE=1 uv run pytest -k live -q   # real-API probes

The README demo (examples/assets/demo.svg) is recorded network-free from examples/_demo_stdio.py — see the header of that file to re-record.

License

MIT — see LICENSE.