# FAIRdata.ai

> Research data, prepared for the machines that use it. FAIRdata.ai turns public
> academic datasets into evidence-checked, machine-readable records, collections
> and ingestion recipes, so models, agents and research workflows can find,
> understand and use them more responsibly.

## What this is

A public registry of research datasets, assessed for FAIR metadata quality and
curation evidence, with normalised licence signals and resolved file URLs. The
qualifying subset is organised into themed collections with versioned manifests.

Public read access requires no authentication. Rate limit: 240 requests per 60
seconds per client IP, reported in RateLimit-* headers.

## What "qualified" means here

A dataset is Machine-Ready when: source-record FAIR composite >= 75 AND curation-evidence proxy >= 50 AND an unambiguous permissive licence signal AND at least one declared file format AND a direct file URL resolved at assessment time.

Read the full ruleset — every criterion, threshold, evidence type and status —
at https://fairdata.ai/api/v1/qualification/ruleset.

## What it does NOT mean

Machine-readable does not mean permission to train. Qualification does not
establish that data are scientifically correct, representative, free of personal
or sensitive information, covered by appropriate consent, or legally available
for any particular use. Licence detection and discipline classification are
automated and can be wrong. File access is resolved point-in-time.

Every qualification response includes a "limits" array. If you relay a status,
relay the limits with it.

## Endpoints for agents (no key required)

- GET https://fairdata.ai/api/v1/meta — capabilities, collection version, ruleset, formats, error catalogue
- GET https://fairdata.ai/api/v1/health — service and data freshness
- GET https://fairdata.ai/api/v1/collections — collections with real aggregates
- GET https://fairdata.ai/api/v1/collections/{id} — one collection
- GET https://fairdata.ai/api/v1/collections/{id}/qualification — per-criterion evidence
- GET https://fairdata.ai/api/v1/collections/{id}/datasets — members, filtered and paginated
- GET https://fairdata.ai/api/v1/collections/{id}/manifest — versioned manifest with checksum
- GET https://fairdata.ai/api/v1/collections/{id}/croissant — Croissant 1.0 JSON-LD
- GET https://fairdata.ai/api/v1/collections/{id}/receipt — fairdata.lock.json
- GET https://fairdata.ai/api/v1/collections/versions — published versions and retention
- GET https://fairdata.ai/api/v1/datasets/resolve?doi={doi} — resolve a DOI
- GET https://fairdata.ai/api/v1/search?q={query} — substring search over qualifying datasets
- GET https://fairdata.ai/api/record/{doi} — the full enriched record (whole registry)
- GET https://fairdata.ai/api/record/{doi}/{schema.org|croissant|ro-crate|fdp|datacite|ai-readiness}
- GET https://fairdata.ai/api/record/{doi}/files — resolved file listing
- GET https://fairdata.ai/api/search/semantic?q={query} — meaning-based search over the whole registry

Errors are RFC 9457 problem+json with a stable "code". A JSON endpoint never
returns HTML. 503 snapshot_unavailable means the data could not be read — it is
not an empty result, and must not be reported as "no collections".

## MCP

Endpoint: POST https://fairdata.ai/api/mcp (Streamable HTTP, JSON-RPC 2.0)
Discovery: https://fairdata.ai/.well-known/mcp.json
Protocol versions: 2026-07-28 (current), 2025-11-25, 2025-06-18, 2025-03-26, 2024-11-05

Tools: get_ruleset, search_collections, get_collection, compare_collections,
list_collection_datasets, get_collection_manifest, explain_collection_qualification,
search_datasets_in_corpus, get_dataset, explain_dataset_qualification,
create_ingestion_plan, create_reproducibility_receipt, check_usage_constraints,
plus the record tools get_record, get_record_format, list_records,
get_ai_readiness, get_files, get_data_quality, find_similar_datasets,
search_datasets, get_linked_papers, assess_dataset.

check_usage_constraints returns evidence and unknowns, never a yes/no on whether
a use is permitted.

## Documentation

- https://fairdata.ai/developers — the developer portal
- https://fairdata.ai/developers/index.md — this documentation as Markdown
- https://fairdata.ai/api/openapi.json — OpenAPI 3.1
- https://fairdata.ai/schemas/collection-manifest-1.0.0.json
- https://fairdata.ai/schemas/fairdata-lock-1.0.0.json

## How to cite

Cite the dataset, not FAIRdata.ai — we describe and assess these records, we did
not create the data. If you want to record how a selection was made, cite the
collection id, its version and the manifest checksum.

## Part of

Infinite Researchers (https://infiniteresearchers.com). Sister projects:
OpenScience.ai, Preprints.ai, OpenAccess.ai.

Contact: hello@fairdata.ai


============================================================================
FULL DOCUMENTATION
============================================================================

# Start / Developer overview
Source: https://fairdata.ai/developers/start/overview

# Developer overview

FAIRdata.ai turns public academic datasets into evidence-checked, machine-readable
records, collections and ingestion recipes. This portal is for the people and
agents that consume them.

Everything here is a public read. There is no key, no sign-up and no quota
form. The API is rate limited to keep one client from starving the rest, and the
limit is reported in the response headers.

## The three things you can get

| You want | You call | You get |
| --- | --- | --- |
| A themed set of datasets | `GET /api/v1/collections` | Real aggregates: counts, score medians, licence composition, freshness |
| The reason a set qualifies | `GET /api/v1/collections/{id}/qualification` | Every criterion, its threshold, the observed value, the evidence and the evaluator |
| Something to pin a workflow to | `GET /api/v1/collections/{id}/manifest` | A versioned manifest with a checksum, validated against a published schema |

## Three access levels

Not every dataset supports every level, and the API says which applies rather
than implying the deepest one everywhere.

- **Level 1 — Metadata access.** You read our records and the source metadata. Always available.
- **Level 2 — File-manifest access.** You get verified source URLs, file metadata, any published checksums, licences and retrieval instructions. Available when file resolution has run.
- **Level 3 — Prepared record access.** You can load structured records directly. Available only where formats and rights allow, which is a subset.

## What qualification is, and is not

A collection is "Machine-Ready" when every member clears an automated evidence
gate on metadata quality, curation evidence, licence signal and file access.

That is a statement about *evidence we could check automatically*. It is not a
statement that the data are correct, unbiased, safe, free of personal
information, or legally available for your use. **Machine-readable does not
mean permission to train.** The [ruleset](/developers/concepts/qualification)
sets out exactly what was tested; the [responsible-use page](/developers/concepts/responsible-use)
sets out what was not.

## Where to go next

- [Five-minute quickstart](/developers/start/quickstart) — a working request in curl, Python or TypeScript
- [Choose an access method](/developers/start/access-methods) — REST, SDK, CLI, MCP or raw files
- [Load a collection](/developers/guides/load-a-collection) — the end-to-end path
- [REST API reference](/developers/reference/rest-api) — every endpoint and parameter
- [Connect an MCP client](/developers/guides/mcp-client) — for agents
- [Service status](/developers/operations/status) — is it us or you?

---

# Start / Five-minute quickstart
Source: https://fairdata.ai/developers/start/quickstart

# Five-minute quickstart

Six steps, from a clean environment to a pinned selection you can reproduce.
Every command on this page runs in our CI on each commit — if one of them stops
working, the build fails before you ever see it.

You will:

1. list collections
2. inspect one
3. fetch its versioned manifest
4. read the qualification evidence
5. load a safe representative resource
6. record what you used

## curl

```bash
# 1. List collections
curl -sS "https://fairdata.ai/api/v1/collections?limit=5&sort=smallest" | jq '.collections[] | {id, title, dataset_count}'

# 2. Inspect one (substitute an id from step 1)
COLLECTION=$(curl -sS "https://fairdata.ai/api/v1/collections?limit=1&sort=smallest" | jq -r '.collections[0].id')
curl -sS "https://fairdata.ai/api/v1/collections/$COLLECTION" | jq '.collection.aggregates.licence'

# 3. Pin the version and fetch the manifest
VERSION=$(curl -sS "https://fairdata.ai/api/v1/collections/$COLLECTION" | jq -r '.collection.version')
curl -sS "https://fairdata.ai/api/v1/collections/$COLLECTION/manifest?version=$VERSION" -o manifest.json

# 4. Read the evidence behind the verdict
curl -sS "https://fairdata.ai/api/v1/collections/$COLLECTION/qualification" \
  | jq '.criteria[] | {id, status, threshold: .threshold.description, observed: .observed.value}'

# 5. Load a representative resource — the flat CSV summary
curl -sS "https://fairdata.ai/api/v1/collections/$COLLECTION/datasets.csv" | head -5

# 6. Record what you used
curl -sS "https://fairdata.ai/api/v1/collections/$COLLECTION/receipt" -o fairdata.lock.json
jq '{collection_version, manifest_checksum, dataset_count}' fairdata.lock.json
```

## Python

No dependencies beyond the standard library, so this runs anywhere Python does.
If you would rather use the typed client, see the [Python SDK](/developers/reference/sdks).

```python
import gzip, json, urllib.request

BASE = "https://fairdata.ai/api/v1"

def get(path):
    # Ask for gzip. A manifest for a large collection is several megabytes
    # uncompressed, and reading that much through a CDN with the standard
    # library is where truncated reads come from. Compressed it is ~20x smaller.
    req = urllib.request.Request(BASE + path, headers={
        "User-Agent": "fairdata-quickstart",
        "Accept-Encoding": "gzip",
    })
    with urllib.request.urlopen(req, timeout=120) as r:
        body = r.read()
        if r.headers.get("Content-Encoding") == "gzip":
            body = gzip.decompress(body)
    return json.loads(body)

# 1. List collections, smallest first — the easiest place to start
index = get("/collections?limit=5&sort=smallest")
for c in index["collections"]:
    print(c["id"], c["dataset_count"], c["aggregates"]["licence"]["distinct_licences"])

# 2. Inspect one
cid = index["collections"][0]["id"]
detail = get(f"/collections/{cid}")
collection = detail["collection"]
print(collection["title"], "-", collection["definition"])
print("licence position:", collection["aggregates"]["licence"]["statement"])

# 3. Pin the version and fetch the manifest
version = collection["version"]
manifest = get(f"/collections/{cid}/manifest?version={version}")
print(manifest["dataset_count"], "datasets at version", manifest["version"])

# 4. Read the evidence
for c in detail["qualification"]["criteria"]:
    print(f"  {c['status']:<8} {c['label']}: {c['threshold']['description']} "
          f"(observed {c['observed']['value']})")

# 5. Load a representative resource
first = manifest["datasets"][0]
print("first dataset:", first["doi"], first["licence"]["identifier"])
if first["licence"]["identifier"] is None:
    print("  no licence resolved — do not assume reuse is permitted")

# 6. Record what you used
receipt = get(f"/collections/{cid}/receipt")
with open("fairdata.lock.json", "w") as fh:
    json.dump(receipt, fh, indent=2)
print("receipt written:", receipt["manifest_checksum"])
```

## TypeScript

Runs on Node 20+ with no dependencies. The typed client is in the
[TypeScript SDK](/developers/reference/sdks).

```typescript
const BASE = "https://fairdata.ai/api/v1";

async function get<T>(path: string): Promise<T> {
  const response = await fetch(BASE + path, { headers: { Accept: "application/json" } });
  if (!response.ok) {
    const problem = await response.json().catch(() => ({}));
    throw new Error(`${problem.code ?? response.status}: ${problem.detail ?? response.statusText}`);
  }
  return response.json() as Promise<T>;
}

// 1. List collections
const index: any = await get("/collections?limit=5&sort=smallest");
for (const c of index.collections) {
  console.log(c.id, c.dataset_count, c.aggregates.licence.distinct_licences);
}

// 2. Inspect one
const id = index.collections[0].id;
const detail: any = await get(`/collections/${id}`);
console.log(detail.collection.title, "-", detail.collection.aggregates.licence.statement);

// 3. Pin the version and fetch the manifest
const version = detail.collection.version;
const manifest: any = await get(`/collections/${id}/manifest?version=${version}`);
console.log(manifest.dataset_count, "datasets at version", manifest.version);

// 4. Read the evidence
for (const c of detail.qualification.criteria) {
  console.log(`  ${c.status}  ${c.label}: ${c.threshold.description}`);
}

// 5. A representative resource
const first = manifest.datasets[0];
console.log("first dataset:", first.doi, first.licence.identifier ?? "no licence resolved");

// 6. Record what you used
const receipt: any = await get(`/collections/${id}/receipt`);
await Bun?.write?.("fairdata.lock.json", JSON.stringify(receipt, null, 2))
  ?? (await import("node:fs/promises")).writeFile("fairdata.lock.json", JSON.stringify(receipt, null, 2));
console.log("receipt written:", receipt.manifest_checksum);
```

## What you now have

A `fairdata.lock.json` that names the exact collection version, the manifest
checksum and every dataset you selected. Re-request the manifest with that
`version` later: if the checksum still matches, nothing has changed. If the
version is no longer published, the API tells you so rather than silently
serving a different set — see [collection versions](/developers/concepts/versions).

## Before you use any of it

Read [How may it be used?](/developers/concepts/responsible-use). A resolved
licence is a signal from the source metadata, not permission, and not legal
advice.

---

# Start / Choose an access method
Source: https://fairdata.ai/developers/start/access-methods

# Choose an access method

All five read the same data through the same service layer, so they cannot
disagree with each other.

| Method | Best for | Start here |
| --- | --- | --- |
| REST API | Anything. No dependencies, no client to keep current. | [REST reference](/developers/reference/rest-api) |
| Python SDK | Notebooks, pipelines, typed access, checksum verification | [SDKs](/developers/reference/sdks) |
| TypeScript SDK | Web apps, Node services, edge functions | [SDKs](/developers/reference/sdks) |
| CLI | Shell scripts, CI jobs, quick inspection | [CLI](/developers/reference/cli) |
| MCP | Agents that need to reason about the data, not just fetch it | [MCP](/developers/reference/mcp) |

## Rules of thumb

- **Fetching one thing?** Use curl or fetch. There is no advantage to a client.
- **Building a reproducible pipeline?** Use an SDK and pin the version. The SDKs write the lockfile for you.
- **Working in a shell or CI?** Use the CLI; it has JSON and human output modes and meaningful exit codes.
- **Wiring up an agent?** Use MCP. The tools return structured output with schemas, and `check_usage_constraints` gives an agent the evidence it needs instead of a false yes/no.
- **Want the bytes?** Every route ends at the source repository. We resolve and describe file URLs; we do not re-host the data.

## Older routes

The pre-v1 collection routes (`/api/collections`, `/api/collections/{theme}.jsonl`,
`/api/collections/{theme}/croissant`, `/api/corpus`) are unchanged and still
served. They now carry a `Link` header pointing at their v1 successor. Nothing
is scheduled for removal — see the
[migration guide](/developers/reference/migration).

---

# Start / Authentication
Source: https://fairdata.ai/developers/start/authentication

# Authentication

**Public read endpoints need no authentication.** No key, no token, no header.
This is deliberate: the point of the service is that machines can read it.

## What that means in practice

- Send no `Authorization` header. One is not required and will not raise your limits.
- CORS is open for `GET` on the public read endpoints, so browser clients work without a proxy.
- Requests are rate limited per client IP. See [rate limits](/developers/reference/pagination-filtering-caching).
- Write operations (running an assessment, administrative endpoints) do require a bearer token and are not part of the public v1 surface.

## Identifying yourself

Not required, but appreciated and useful to you:

- Send a descriptive `User-Agent` (`my-lab-pipeline/2.1 (+https://example.org)`). If something goes wrong at scale, it is what lets us tell you rather than block you.
- Send `X-Request-Id` if you have one. We echo it back, and it appears in error responses, so a report can be matched to a specific request.

## Optional keys, later

The architecture leaves room for optional API keys that would raise quotas,
attach usage analytics, save queries, send notifications, and run bulk jobs.
None of that exists today, and **no existing public read will be moved behind a
key**. If keys arrive, they will be additive.

---

# Guides / Find a dataset or collection
Source: https://fairdata.ai/developers/guides/find-data

# Find a dataset or collection

## Browse collections

```bash
curl -sS "https://fairdata.ai/api/v1/collections?sort=smallest&limit=10"
```

Useful filters, all combinable:

| Parameter | Effect |
| --- | --- |
| `q` | Free text over collection title and definition |
| `repository` | Only collections containing datasets from that source repository |
| `licence` | Only collections containing that SPDX identifier |
| `modality` | `tabular`, `sequence`, `geospatial`, `image`, `text`, `document`, `audio_video`, `archive` |
| `min_datasets` | Exclude small collections |
| `min_fair` | Exclude collections whose FAIR median is below a value |
| `sort` | `datasets`, `smallest`, `fair`, `curation`, `licence-clarity`, `title` |

Sorting by `smallest` is the most useful default when you are starting: a
50-dataset collection is far easier to understand end to end than a 900-dataset one.

## Search within a collection

```bash
curl -sS "https://fairdata.ai/api/v1/collections/31-biological-sciences/datasets?q=soil&licence=CC0-1.0&limit=20"
```

## Search the whole qualifying corpus

```bash
curl -sS "https://fairdata.ai/api/v1/search?q=permafrost&min_fair=85"
```

`/api/v1/search` is a literal substring match on titles and DOIs, and it only
covers datasets that clear the gate. It says so in every response, under
`query.match` and `scope`. For meaning-based search across the whole registry —
including datasets that do not qualify — use `/api/search/semantic`.

## Resolve a specific DOI

DOIs contain slashes, so there are three documented ways in:

```bash
# Recommended: nothing to encode
curl -sS "https://fairdata.ai/api/v1/datasets/resolve?doi=10.5061/dryad.ffbg79crm"

# Opaque single segment: the DOI as unpadded URL-safe base64
curl -sS "https://fairdata.ai/api/v1/datasets/MTAuNTA2MS9kcnlhZC5mZmJnNzljcm0"

# Literal DOI on the path
curl -sS "https://fairdata.ai/api/v1/datasets/10.5061/dryad.ffbg79crm"
```

A DOI that is in the registry but does not clear the gate returns a `404` with
code `dataset_not_found` **and** a `registry_record_url` pointing at its
record. Not qualifying is not the same as not existing.

---

# Guides / Load a collection
Source: https://fairdata.ai/developers/guides/load-a-collection

# Load a collection

## The shape of the job

A FAIRdata collection is a **set of independent datasets**, each in its own
formats, from its own repository, under its own licence. It is not one corpus
with one schema. Any loader that pretends otherwise will silently mangle
something.

So the order is: manifest → licence decision per dataset → file list per
dataset → fetch → receipt.

## 1. Fetch the manifest, pinned

```python
import gzip, json, urllib.request

BASE = "https://fairdata.ai/api/v1"
CID = "31-biological-sciences"

def get(path):
    # Always ask for gzip: an eight-megabyte manifest read uncompressed through
    # a CDN is where truncated reads come from, and both SDKs do the same.
    request = urllib.request.Request(BASE + path, headers={"Accept-Encoding": "gzip"})
    with urllib.request.urlopen(request, timeout=180) as r:
        body = r.read()
        if r.headers.get("Content-Encoding") == "gzip":
            body = gzip.decompress(body)
    return json.loads(body)

version = get(f"/collections/{CID}")["collection"]["version"]
manifest = get(f"/collections/{CID}/manifest?version={version}&include_files=true")
```

`include_files=true` adds a per-file listing — URL, media type, byte size and any
published checksum. It is off by default because it is materially larger.

## 2. Decide per dataset, not per collection

```python
usable, skipped = [], []
for d in manifest["datasets"]:
    licence = d["licence"]["identifier"]
    if licence is None:
        skipped.append((d["doi"], "no licence resolved"))
        continue
    if not d["files"]["resolved_count"]:
        skipped.append((d["doi"], "no resolved file URL"))
        continue
    usable.append(d)

print(len(usable), "usable;", len(skipped), "skipped")
```

Keep `skipped`. A silent omission is a reproducibility bug, and the lockfile
schema has a field for exactly this.

## 3. Fetch files, with the obvious production manners

```python
import hashlib, time, urllib.error

def fetch(url, dest, attempts=3):
    for attempt in range(attempts):
        try:
            req = urllib.request.Request(url, headers={"User-Agent": "my-pipeline/1.0"})
            with urllib.request.urlopen(req, timeout=300) as r, open(dest, "wb") as fh:
                digest = hashlib.md5()
                while chunk := r.read(1 << 20):
                    fh.write(chunk); digest.update(chunk)
            return digest.hexdigest()
        except (urllib.error.URLError, TimeoutError):
            if attempt == attempts - 1:
                raise
            time.sleep(2 ** attempt)
```

## 4. Verify what can be verified

```python
import hashlib

for f in dataset["file_list"]:
    algorithm = f["checksum_algorithm"]          # "md5", "sha-256", … or None
    if not algorithm:
        # Nothing published. Record what you received rather than implying a
        # verification that did not happen.
        record(f, verified=None, note="source publishes no checksum")
        continue
    digest = hashlib.new(algorithm.replace("-", ""))
    for chunk in stream(f["url"]):
        digest.update(chunk)
    record(f, verified=digest.hexdigest() == f["checksum"])
```

Use the algorithm the file names. Repositories differ — Zenodo and Dataverse
publish md5, Dryad and Mendeley publish sha-256 — and verifying a sha-256 with
md5 fails for a reason that has nothing to do with the data.

## 5. Write the receipt

```python
receipt = get(f"/collections/{CID}/receipt?version={version}")
receipt["skipped"] = [{"doi": doi, "reason": why} for doi, why in skipped]
json.dump(receipt, open("fairdata.lock.json", "w"), indent=2)
```

## Or use the SDK

The Python SDK does all of the above, including retries, checksum handling and
the receipt:

```python
from fairdata import FairData

client = FairData()
collection = client.collections.get("31-biological-sciences")
receipt = collection.pull(version=collection.version, dest="./data", verify_checksums=True)
receipt.write("fairdata.lock.json")
```

---

# Guides / Pin a version and reproduce it later
Source: https://fairdata.ai/developers/guides/pin-and-reproduce

# Pin a version and reproduce it later

## The problem

Collections change. Datasets get assessed, files get resolved, licences get
corrected, and the membership moves. "I used the environmental sciences
collection" is not a reproducible statement.

## The mechanism

Every collection has a **content-addressed version**. The same version string
always denotes the same set of members, because the string is derived from the
members themselves.

```bash
VERSION=$(curl -sS "https://fairdata.ai/api/v1/collections/41-environmental-sciences" | jq -r .collection.version)
curl -sS "https://fairdata.ai/api/v1/collections/41-environmental-sciences/manifest?version=$VERSION" -o manifest.json
```

Pass `?version=` to any collection endpoint. If that version is still the
published one, you get it. If it is not, you get a `409` with code
`version_not_available` naming the current version — never a different set
served quietly under your pinned request.

## The receipt

```bash
curl -sS "https://fairdata.ai/api/v1/collections/41-environmental-sciences/receipt?version=$VERSION" -o fairdata.lock.json
```

`fairdata.lock.json` records the collection id, the exact version, the manifest
checksum, the ruleset version, the retrieval timestamp, every selected dataset
with its licence and file count, and everything you skipped with a reason. It
validates against a [published schema](https://fairdata.ai/schemas/fairdata-lock-1.0.0.json).

Commit it next to your code. It is the thing that makes "which data did this
result use?" answerable.

## Checking later whether anything changed

The checksum is in a response header, so the cheap check is a `HEAD` request —
no need to download the manifest at all. The largest collections run to several
megabytes, and you only need those bytes when the answer is "yes, it changed".

```python
import json, urllib.request

lock = json.load(open("fairdata.lock.json"))
url = f"https://fairdata.ai/api/v1/collections/{lock['collection_id']}/manifest"

request = urllib.request.Request(url, method="HEAD")
with urllib.request.urlopen(request, timeout=60) as response:
    current = response.headers["X-FAIRdata-Manifest-Checksum"]
    version = response.headers["X-FAIRdata-Collection-Version"]

if current == lock["manifest_checksum"]:
    print("unchanged; still collection version", version)
else:
    print("changed since", lock["retrieved_at"], "— now version", version)
```

The two headers answer different questions, and it is worth keeping them apart:

| Signal | Answers | Changes when |
| --- | --- | --- |
| `X-FAIRdata-Collection-Version` | Are these the same datasets? | Membership changes |
| `X-FAIRdata-Manifest-Checksum` | Is anything about them different? | Membership, a licence, a score or the ruleset changes |

Neither changes because we deployed. The build that produced a manifest is
recorded in it as `pipeline_version`, but it is deliberately excluded from the
checksum — otherwise every deploy would report a change that had not happened.

```bash
curl -sI "https://fairdata.ai/api/v1/collections/41-environmental-sciences/manifest" \
  | grep -i x-fairdata
```

### Seeing what changed

When it has changed and you want the difference, then fetch the manifest. Read
the whole body before parsing: an eight-megabyte response over a flaky
connection should fail cleanly rather than half-parse.

```python
import gzip, json, urllib.request

request = urllib.request.Request(url, headers={"Accept-Encoding": "gzip"})
with urllib.request.urlopen(request, timeout=180) as response:
    body = response.read()
    if response.headers.get("Content-Encoding") == "gzip":
        body = gzip.decompress(body)
current = json.loads(body)

was = {d["doi"] for d in lock["datasets"]}
now = {d["doi"] for d in current["datasets"]}
print("  added:  ", sorted(now - was)[:10])
print("  removed:", sorted(was - now)[:10])
```

## The honest limitation

We retain member lists for the **current version only**. Ask for a superseded
version and the API refuses, because it cannot honestly serve it. That is why
the manifest you downloaded — and the lockfile beside it — are the artefacts
that carry your reproducibility, not a promise that our server will remember.

If you need long-term reproducibility, archive `manifest.json` alongside
`fairdata.lock.json`. Between them they contain every DOI, licence, score and
file URL you used.

---

# Guides / Verify checksums and preserve provenance
Source: https://fairdata.ai/developers/guides/verify-and-provenance

# Verify checksums and preserve provenance

## What is verifiable

Two separate things, often confused:

1. **Did I receive the manifest FAIRdata.ai produced?** Always verifiable, via `manifest_checksum`.
2. **Did I receive the file the repository holds?** Verifiable only when the source repository publishes a checksum. Many do not.

## Verifying the manifest

The checksum is sha256 over the canonical JSON of the manifest's content —
sorted keys, no whitespace — excluding the three fields that describe the run
that served the response rather than the collection: `manifest_checksum` itself,
`retrieved_at` and `pipeline_version`.

Canonical JSON has one more rule, and skipping it is why a hand-rolled check
fails on a real manifest: an integral float is written as an integer. Python
renders `75.0` as `"75.0"` where JavaScript renders it as `"75"`, and manifests
are full of such numbers — every qualification threshold is one. Collapsing them
first is what lets a checksum computed in one language agree with another's.

```python
import hashlib, json

EXCLUDED = ("manifest_checksum", "retrieved_at", "pipeline_version")

def canonicalise(v):
    if isinstance(v, bool): return v
    if isinstance(v, float) and v.is_integer(): return int(v)
    if isinstance(v, dict): return {k: canonicalise(x) for k, x in v.items()}
    if isinstance(v, list): return [canonicalise(x) for x in v]
    return v

manifest = json.load(open("manifest.json"))
body = {k: v for k, v in manifest.items() if k not in EXCLUDED}
canonical = json.dumps(canonicalise(body), sort_keys=True, separators=(",", ":"), ensure_ascii=False)
computed = "sha256:" + hashlib.sha256(canonical.encode()).hexdigest()
assert computed == manifest["manifest_checksum"]
```

Or skip all of that and call `manifest.verify()` in either SDK, which implements
exactly this.

Because `retrieved_at` is excluded, two requests for the same collection version
return the same checksum; because `pipeline_version` is, that holds across our
deploys too. That is the property a lockfile depends on.

## Verifying files

```python
for f in dataset["file_list"]:
    if f["checksum_algorithm"] is None:
        # Record it as unverified rather than implying otherwise.
        record(f, verified=None, note="source publishes no checksum")
        continue
    got = hash_downloaded_file(f["url"], f["checksum_algorithm"])
    record(f, verified=(got == f["checksum"]))
```

The lockfile schema has `checksum_verified` as a **nullable** boolean for this
reason. `null` means "no checksum existed to verify against" — which is not the
same as `false`.

## Provenance to keep

Keep these with your outputs, at minimum:

| Field | Why |
| --- | --- |
| `collection_version` | Which member set you drew from |
| `manifest_checksum` | Proof of exactly which manifest |
| `ruleset.version` | Which criteria were in force |
| `retrieved_at` | When the observations were true |
| Per dataset: `doi`, `licence`, `assessed_at` | Attribution and the age of each assessment |
| `skipped` | What you did not take, and why |

## Where each fact came from

Every criterion in a qualification response carries its evidence, and the
evidence carries its type. The types matter:

- `source_metadata` — the depositor or repository asserted it
- `source_file_observation` — we observed it by listing or fetching the files
- `deterministic_calculation` — we computed it, reproducibly, from observed inputs
- `ai_assisted_inference` — a model contributed to the value
- `fairdata_enrichment` — we added it

Values marked `generated: true` were produced by us or by a model. They must not
be presented downstream as if the depositor supplied them. See
[evidence and provenance](/developers/concepts/evidence).

---

# Guides / Use a collection with pandas, PyArrow, Hugging Face or PyTorch
Source: https://fairdata.ai/developers/guides/dataframes

# Use a collection with pandas, PyArrow, Hugging Face or PyTorch

## Start with the honest bit

The **collection metadata** loads into any dataframe tool directly. The
**underlying research data** does not, because a collection spans many datasets
with different schemas, formats and licences. Anything that promises otherwise
is hiding a transformation from you.

So there are two jobs, and they need different tools.

## Job 1 — the metadata, as a table

### pandas

```python
import pandas as pd

df = pd.read_csv(
    "https://fairdata.ai/api/v1/collections/41-environmental-sciences/datasets.csv",
    comment="#",
)
print(df["licence"].value_counts(dropna=False))
print(df[df["licence"].isna()][["doi", "title"]])   # the ones you cannot use yet
```

The CSV is a **lossy** view: it drops per-criterion evidence and per-file detail.
Its first line says so. Use the manifest when those matter.

### PyArrow, keeping provenance attached

```python
import json, urllib.request
import pyarrow as pa, pyarrow.parquet as pq

with urllib.request.urlopen(
    "https://fairdata.ai/api/v1/collections/41-environmental-sciences/manifest"
) as r:
    manifest = json.load(r)

table = pa.table({
    "doi":      [d["doi"] for d in manifest["datasets"]],
    "licence":  [d["licence"]["identifier"] for d in manifest["datasets"]],
    "fair":     [d["scores"]["fair"] for d in manifest["datasets"]],
    "files":    [d["files"]["resolved_count"] for d in manifest["datasets"]],
})
table = table.replace_schema_metadata({
    "fairdata_version": manifest["version"],
    "fairdata_manifest_checksum": manifest["manifest_checksum"],
})
pq.write_table(table, "collection.parquet")
```

Putting the version and checksum in the Parquet metadata means the provenance
survives the file being copied somewhere else.

## Job 2 — the data files

### Hugging Face Datasets

`load_dataset` needs files of one format with a compatible schema. A collection
does not have that, so build the file list first and be explicit about what you
drop:

```python
import json, urllib.request
from datasets import load_dataset

with urllib.request.urlopen(
    "https://fairdata.ai/api/v1/collections/41-environmental-sciences/manifest?include_files=true"
) as r:
    manifest = json.load(r)

csv_files, skipped = [], []
for d in manifest["datasets"]:
    if d["licence"]["identifier"] is None:
        skipped.append((d["doi"], "no licence resolved")); continue
    urls = [f["url"] for f in d.get("file_list", [])
            if (f.get("media_type") or "").endswith(("csv", "ms-excel")) and f.get("url")]
    if not urls:
        skipped.append((d["doi"], "no CSV file resolved")); continue
    csv_files.extend(urls)

print(f"{len(csv_files)} CSV files; {len(skipped)} datasets skipped")
ds = load_dataset("csv", data_files=csv_files[:1])   # schemas differ: load separately
```

Do not concatenate CSVs from different datasets without checking their columns.
They are different studies.

### PyTorch

There is no single tensor shape across a research-data collection, so a
ready-made `Dataset` class cannot be correct for all of it. What is honest is a
`Dataset` over the **resolved file list**, with parsing left to you:

```python
from torch.utils.data import Dataset
import json, urllib.request

class FairDataFiles(Dataset):
    def __init__(self, manifest_url, media_type_prefix=None):
        with urllib.request.urlopen(manifest_url) as r:
            manifest = json.load(r)
        self.version = manifest["version"]
        self.files = [
            {**f, "doi": d["doi"], "licence": d["licence"]["identifier"]}
            for d in manifest["datasets"] if d["licence"]["identifier"]
            for f in d.get("file_list", [])
            if f.get("url") and (not media_type_prefix
                                 or (f.get("media_type") or "").startswith(media_type_prefix))
        ]

    def __len__(self): return len(self.files)
    def __getitem__(self, i): return self.files[i]
```

## Croissant

Where a Croissant-aware loader is what you want, every collection has one:

```bash
curl -sS "https://fairdata.ai/api/v1/collections/41-environmental-sciences/croissant" -o croissant.jsonld
```

It is a `schema.org/Dataset` whose `distribution` lists each member as a
`cr:FileObject` pointing at that dataset's own Croissant record, with its own
`cr:license`. See [Croissant and output formats](/developers/concepts/formats).

---

# Guides / Use a collection from JavaScript
Source: https://fairdata.ai/developers/guides/javascript

# Use a collection from JavaScript

## In the browser

CORS is open for `GET` on the public read endpoints, so no proxy is needed.

```javascript
const response = await fetch("https://fairdata.ai/api/v1/collections?limit=10&sort=smallest");
if (!response.ok) {
  const problem = await response.json();   // application/problem+json
  throw new Error(`${problem.code}: ${problem.detail}`);
}
const { collections, meta } = await response.json();
console.log(meta.collection_version, collections.length);
```

## Handling errors properly

Every documented JSON endpoint returns `application/problem+json` on failure —
never an HTML error page. Branch on `code`, not on the message:

```javascript
async function getJson(url) {
  const r = await fetch(url, { headers: { Accept: "application/json" } });
  if (r.ok) return r.json();
  const problem = await r.json().catch(() => ({}));
  const err = new Error(problem.detail ?? r.statusText);
  err.code = problem.code;           // e.g. "collection_not_found"
  err.requestId = problem.request_id; // quote this when reporting
  throw err;
}
```

## Streaming JSONL in Node

For large collections, stream rather than buffering the whole array:

```javascript
const response = await fetch(
  "https://fairdata.ai/api/v1/collections/31-biological-sciences/datasets.jsonl",
);
const reader = response.body.getReader();
const decoder = new TextDecoder();
let buffer = "";
let seen = 0;

while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  buffer += decoder.decode(value, { stream: true });
  const lines = buffer.split("\n");
  buffer = lines.pop();
  for (const line of lines) {
    if (!line) continue;
    const dataset = JSON.parse(line);
    if (dataset.licence) seen += 1;
  }
}
console.log(seen, "datasets with a resolved licence");
```

## Conditional requests

Collections change a few times a day. Cache the `ETag` and you will mostly get
`304 Not Modified` for free:

```javascript
const cache = new Map();

async function getCached(url) {
  const prior = cache.get(url);
  const r = await fetch(url, { headers: prior ? { "If-None-Match": prior.etag } : {} });
  if (r.status === 304) return prior.body;
  const body = await r.json();
  cache.set(url, { etag: r.headers.get("etag"), body });
  return body;
}
```

## Or use the SDK

```typescript
import { FairData } from "@fairdata/sdk";

const fairdata = new FairData();
const collection = await fairdata.collections.get("31-biological-sciences");
const manifest = await collection.manifest({ version: collection.version });
```

---

# Guides / Connect an MCP client
Source: https://fairdata.ai/developers/guides/mcp-client

# Connect an MCP client

The MCP server is at `https://fairdata.ai/api/mcp`, over Streamable HTTP. No
authentication.

## Claude Desktop

```json
{
  "mcpServers": {
    "fairdata": {
      "type": "http",
      "url": "https://fairdata.ai/api/mcp"
    }
  }
}
```

## Claude Code

```bash
claude mcp add --transport http fairdata https://fairdata.ai/api/mcp
```

## Any client, by hand

```bash
# Discover what the server supports, in one call
curl -sS https://fairdata.ai/api/mcp \
  -H 'Content-Type: application/json' \
  -H 'MCP-Protocol-Version: 2026-07-28' \
  -d '{"jsonrpc":"2.0","id":1,"method":"server/discover","params":{}}' | jq

# List the tools
curl -sS https://fairdata.ai/api/mcp \
  -H 'Content-Type: application/json' \
  -H 'MCP-Protocol-Version: 2026-07-28' \
  -d '{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}}' | jq '.result.tools[].name'

# Call one
curl -sS https://fairdata.ai/api/mcp \
  -H 'Content-Type: application/json' \
  -H 'MCP-Protocol-Version: 2026-07-28' \
  -d '{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{
        "name":"search_collections","arguments":{"limit":3,"sort":"smallest"}}}' | jq '.result.structuredContent'
```

Older clients that use the `initialize` handshake still work: the server speaks
`2026-07-28`, `2025-11-25`, `2025-06-18`, `2025-03-26` and `2024-11-05`, and
negotiates whichever the client asks for.

## Asking an agent to use it well

Two prompts ship with the server (`prompts/list`), but the shape that works is:

> Use fairdata. Find a small collection about soil chemistry, explain why it
> qualifies including anything unknown, then check the usage constraints before
> recommending anything. Report the limits it returns.

The tools an agent should reach for, in order:

1. `get_ruleset` — what "qualified" means here
2. `search_collections` — find candidates with their real aggregates
3. `explain_collection_qualification` — the criteria matrix and the exceptions
4. `check_usage_constraints` — licence signals, obligations and unknowns
5. `create_ingestion_plan` — a concrete plan with runnable code
6. `create_reproducibility_receipt` — the lockfile

## The question agents get wrong

"Can I train a commercial model on this?" has no yes/no answer from a licence
signal alone. `check_usage_constraints` returns
`determination: "not_determined_by_this_service"` and then gives what it does
know: the licence signals and where they came from, what those licences permit
and require, and every unknown — personal data, consent scope, whether the
repository's current terms match the recorded metadata. An agent that reports
those honestly is being useful; one that collapses them into "yes" is not.

## Verifying your setup

```bash
curl -sS https://fairdata.ai/api/mcp/health | jq
```

Reports the protocol versions, tool count, the collection version being served
and the ruleset in force. Full details in the [MCP reference](/developers/reference/mcp).

---

# Guides / Handle licences and cite data correctly
Source: https://fairdata.ai/developers/guides/licences-and-citation

# Handle licences and cite data correctly

## The rule

**Decide per dataset, never per collection.** A collection-level licence badge
would be wrong for any mixed set, and most sets are mixed.

```python
licence = dataset["licence"]["identifier"]
if licence is None:
    skip(dataset, "no licence resolved")     # absence is not permission
elif licence in {"CC0-1.0"}:
    use(dataset, attribution="courtesy")
else:
    use(dataset, attribution="required", terms=dataset["licence"]["url"])
```

## What each signal means

| Value | Meaning |
| --- | --- |
| `CC0-1.0` | Public-domain dedication. No licence conditions; citation still expected by scholarly norms. |
| `CC-BY-4.0` | Attribution, link to the licence, indicate changes. |
| `CC-BY` | Attribution required, but the version was not resolved — check the source record. |
| `MIT`, `Apache-2.0`, `BSD` | Software-style terms; notice must travel with redistribution. Apache adds a patent grant. |
| `null` | **No licence could be resolved.** The rights statements were missing, unrecognised, or contradicted each other. Nothing may be inferred. |

Restrictive licences (NC, ND, SA) never appear in the qualifying corpus: a
dataset carrying one fails the licence criterion and is excluded. Contradictory
rights statements — a permissive licence alongside a non-commercial claim —
also fail, because a conflict is uncertainty, not permission.

## Reading a collection's licence position

```bash
curl -sS "https://fairdata.ai/api/v1/collections/41-environmental-sciences" \
  | jq '.collection.aggregates.licence | {distinct_licences, mixed, without_explicit_licence, statement}'
```

`statement` is prose written from the actual composition, not a template. For a
mixed set it says so, and names the licences.

## What a licence signal is not

It is the licence **recorded in the source metadata at assessment time**,
normalised to an SPDX identifier. It is not:

- legal advice
- a check that the repository still shows those terms today
- a check for third-party rights inside the files
- a statement about personal data or the scope of consent
- permission to train a model

Those remain your checks. The source landing page is the authority on its own
terms.

## Citing data

Every manifest entry carries a `citation` string built only from fields we
actually hold, and `null` when we do not hold enough to build one:

```bash
curl -sS "https://fairdata.ai/api/v1/collections/41-environmental-sciences/manifest" \
  | jq -r '.datasets[] | select(.citation) | .citation' | head -3
```

Cite the **dataset**, not FAIRdata.ai. We describe and assess these records; we
did not create the data. If you want to say how you selected them, cite the
collection version and manifest checksum alongside:

> Datasets selected from FAIRdata.ai collection `41-environmental-sciences`,
> version `3ee97c2ecc9d81fb` (manifest checksum `sha256:…`), retrieved
> 2026-08-19.

---

# Concepts / What "FAIR for machines" means
Source: https://fairdata.ai/developers/concepts/fair-for-machines

# What "FAIR for machines" means

FAIR — Findable, Accessible, Interoperable, Reusable — was written for research
data management. Most of the effort behind it has gone into making data findable
and accessible *by people*: landing pages, DOIs, repository search.

The consumers have changed. Increasingly the thing that finds a dataset, decides
whether it can be used, and loads it is a program: a pipeline, a training job, an
agent. Those consumers need different things from the same principles.

| Principle | For a person | For a machine |
| --- | --- | --- |
| Findable | A search box and a landing page | A stable identifier, a queryable index, structured metadata |
| Accessible | A download button | A resolvable file URL, a media type, a size, a checksum |
| Interoperable | A README explaining the columns | A declared schema and a format something can actually parse |
| Reusable | A licence statement to read | A normalised licence identifier, and an honest gap when there is not one |

## What we do about it

We take public academic datasets and their metadata and produce records that a
machine can act on:

- **Structured assessment** of the deposited metadata across four FAIR frameworks, plus a separate curation-evidence measure
- **Normalised licence signals**, with contradictions failing closed rather than resolving optimistically
- **Resolved file URLs**, observed rather than assumed
- **Collections** with real aggregates, so a program can compare sets rather than guess
- **Versioned manifests** with checksums, so a selection is reproducible
- **Per-criterion evidence**, so any verdict can be interrogated rather than trusted

## The playful version

We describe this as feeding machines better data. The metaphor is useful — a
collection page has an ingredients list, a nutrition label, dietary restrictions
and serving instructions — but it stops at the door of anything that has to be
precise. The API reference, the error codes, the licence panel and the
methodology are literal, because those are the places where a cute phrase would
cost someone something.

## What it is not

"FAIR for machines" is a statement about **the shape and describability of the
data**. It says nothing about whether the data are correct, representative,
ethically collected, or legally available for a given use.

That separation is the whole point, and we keep those dimensions visibly apart
everywhere: FAIRness, metadata completeness, curation readiness, file
accessibility, licence and stated rights, scientific evidence, provenance,
responsible-use considerations, and machine-ingestion compatibility. Collapsing
them into one number would make the number useless and the omissions invisible.

See [FAIR versus model readiness](/developers/concepts/responsible-use) for the
distinction that matters most.

---

# Concepts / Collection qualification
Source: https://fairdata.ai/developers/concepts/qualification

# Collection qualification

## One ruleset, one place

Every threshold, criterion label and caveat lives in a single versioned module
and is served at:

```bash
curl -sS https://fairdata.ai/api/v1/qualification/ruleset
```

The website, the REST API, the manifests, the Croissant packaging, the MCP tools
and this documentation all read from it. Nothing restates a threshold. A test
fails the build if the copy bundled into the site drifts from the Python source,
and another fails if a threshold reappears as a hand-written string anywhere in
the backend.

## The criteria

A dataset is Machine-Ready when **all** of these pass:

| id | Tests | Threshold | Evidence origin |
| --- | --- | --- | --- |
| `fair-score-minimum` | Composite FAIR score of the deposited metadata | ≥ 75 / 100 | Deterministic calculation |
| `curation-readiness-minimum` | Curation-evidence proxy (RDA/WDS Data Fitness for Use) | ≥ 50 / 100 | AI-assisted inference |
| `permissive-licence-signal` | Rights resolve to one permissive licence, with no contradiction | CC0-1.0, CC-BY, CC-BY-4.0, MIT, Apache-2.0 or BSD | Source metadata |
| `declared-file-formats` | Source metadata declares at least one format | present | Source metadata |
| `resolved-file-url` | At least one direct file URL resolved at assessment time | ≥ 1 file | Source file observation |

Two things worth noticing. The FAIR score used is the **source record's own**
score — enriched projections produced by us do not qualify a record, because
that would be marking our own homework. And the curation measure is
**model-assisted**, which is why it is labelled as such everywhere it appears.

## Collection-level qualification

A collection qualifies when every member does. For each numeric criterion the
observed value reported for the collection is the **worst** value across its
members, not an average — an average could hide a failure, and a collection that
contains one failing dataset has not passed.

## Statuses

Unknown is never collapsed into pass or fail:

| Status | Meaning |
| --- | --- |
| `qualified` | Every criterion passed at the last assessment |
| `not_qualified` | At least one criterion failed on a value we observed |
| `provisional` | Everything passed, but at least one value rests on low-confidence evidence |
| `assessment_incomplete` | Nothing failed, but at least one value has not been assessed |
| `unknown` | Not assessed against the ruleset at all |
| `stale` | Everything passed, but the assessment is older than the freshness window (365 days) |

`assessment_incomplete` is the one that matters most in practice. "We checked and
it fell short" and "we have not checked" are different facts, and a system that
reports them identically is lying about one of them.

## Ruleset versioning

`RULESET_VERSION` is semantic:

- **patch** — wording only, no membership change
- **minor** — a new criterion no current member fails, or a relaxed threshold
- **major** — a tightened threshold, or a new criterion that removes members

Any change that alters membership must bump at least the minor version and be
recorded in the [changelog](/developers/operations/changelog). A ruleset change
that adds or removes datasets from a collection is a product change, and is
reported as one.

## Reading the evidence

```bash
curl -sS "https://fairdata.ai/api/v1/collections/41-environmental-sciences/qualification" \
  | jq '.criteria[] | {id, status, required: .threshold.description,
                       observed: .observed.value, evidence: .evidence[0].type}'
```

Each criterion returns: what was tested, the threshold, the observed value,
pass/fail/unknown, the evidence with its source and URL, when it was checked, the
evaluator and its version, and whether the result was calculated, observed,
publisher-supplied, inferred or AI-assisted.

## Overrides and corrections

There is no silent human override. If an assessment is wrong — a licence
misread, a file wrongly marked unresolvable — report it via
[support](/developers/operations/support) with the DOI and what you observed.
Corrections go through the pipeline and change the collection version, so they
are visible as a change rather than an edit.

---

# Concepts / Evidence and provenance
Source: https://fairdata.ai/developers/concepts/evidence

# Evidence and provenance

## The vocabulary

Every observed value carries an evidence type from a closed list:

| Type | Meaning | Trust it for |
| --- | --- | --- |
| `source_metadata` | The depositor or repository asserted it | What was claimed, not whether it is true |
| `source_file_observation` | We observed it by listing or fetching files | What was there when we looked |
| `deterministic_calculation` | We computed it from observed inputs, reproducibly | Consistency; re-running gives the same answer |
| `external_registry` | A third party (DataCite, OpenAlex, SPDX) | As much as you trust that registry |
| `fairdata_enrichment` | We added it | Our method, documented; not a depositor claim |
| `ai_assisted_inference` | A model contributed | An estimate, not an observation |
| `human_review` | A person checked it | A person's judgement, at that time |
| `depositor_assertion` | A claim we have not verified | Nothing beyond "it was claimed" |

Anything with `generated: true` — `ai_assisted_inference` and
`fairdata_enrichment` — was produced by us or by a model. The interface marks
those visibly, and downstream consumers must not present them as if the
depositor supplied them.

## What each evidence record carries

```json
{
  "type": "source_file_observation",
  "type_label": "Observed from the files themselves",
  "generated": false,
  "source": "FAIRdata.ai file resolution against the source repository",
  "source_url": "https://fairdata.ai/api/record/10.5061/dryad.ffbg79crm/files",
  "retrieved_at": "2026-05-29T10:13:23.171822+00:00",
  "evaluator": "fairdata-file-resolver",
  "evaluator_version": "1.0.0",
  "pipeline_version": "a1b2c3d"
}
```

Which lets you answer, for any fact: which rule was tested, what the threshold
was, what this dataset scored, whether it passed, what evidence supports it,
where the evidence came from, when it was checked, which evaluator and version
produced it, whether the result was deterministic or inferred, and what remains
uncertain.

## Where AI is involved

Two places, both labelled:

1. **Curation readiness** (`curation-readiness-minimum`) is a model-assisted assessment against the RDA/WDS Data Fitness for Use criteria. It is a proxy for how much curation evidence accompanies the data — not a human audit of integrity, ethics or bias, and we say so wherever the number appears.
2. **Discipline classification.** Field of Research codes come from the dataset's own deposited subjects where those carry a usable signal, and from confidence-gated machine classification otherwise. Every collection reports the split — how many labels came from deposited metadata, how many were machine-assigned, and how many datasets remain unclassified. Machine assignments carry a confidence value and can be wrong.

Metadata enrichments we generate are recorded with full provenance — model,
timestamp, confidence, derivation — and never merged into the source metadata as
though the depositor had supplied them.

## Unknown is a value

Throughout the API, `null` means "we do not know", not "zero" and not "no".

- A dataset with no resolved licence has `licence.identifier: null`, a warning saying so, and the licence criterion at `unknown` — not `fail`.
- A collection whose byte total includes a file of unknown size reports `size.complete: false`, so the total reads as a lower bound.
- A file with no published checksum has `checksum_algorithm: null`, and a lockfile records `checksum_verified: null` — which is not `false`.

If you build on this data, preserve the distinction. Collapsing unknown into
false is how a "no licence recorded" quietly becomes "not licensed", and then
"licensed" after someone flips the polarity.

---

# Concepts / Collection versions
Source: https://fairdata.ai/developers/concepts/versions

# Collection versions

## Content-addressed

A collection version is a hash over the qualifying corpus. The same version
string always denotes the same members. Two consequences:

- Comparing version strings tells you whether anything changed, with no diff needed.
- All collections share one version string, because they are all views over one snapshot.

## Pinning

Add `?version=` to any collection endpoint:

```bash
curl -sS "https://fairdata.ai/api/v1/collections/41-environmental-sciences/manifest?version=3ee97c2ecc9d81fb"
```

You get exactly that version, or a `409` with code `version_not_available`
naming the current one. There is no third outcome — a pinned request never
silently returns different data.

## What changes between versions

- A dataset is newly assessed and clears the gate → added
- A licence is corrected from permissive to restrictive → removed
- File resolution finds a working URL → added
- A file URL stops resolving → removed
- A score is recomputed across the threshold → added or removed
- The ruleset changes → potentially many changes at once, recorded in the [changelog](/developers/operations/changelog)

## Retention, stated honestly

We retain **member lists for the current version only**. The version history
endpoint lists which versions existed, when, and how many datasets each had —
but it cannot reconstruct a superseded member list, and it says so rather than
implying otherwise:

```bash
curl -sS https://fairdata.ai/api/v1/collections/versions | jq '.retention'
```

This is why your reproducibility lives in the artefacts you keep — the manifest
and `fairdata.lock.json` — rather than in a promise that our server will
remember. Between them they contain every DOI, licence, score and file URL you
used, plus the checksum that tells you whether the collection has moved since.

## Freshness, separately

Version is about *membership*. Freshness is about *how old the observations
are*. A dataset assessed more than 365 days ago is reported as `stale` even
though it still qualifies, and collections report how many stale members they
contain. Both matter, and neither substitutes for the other.

---

# Concepts / Manifests and lockfiles
Source: https://fairdata.ai/developers/concepts/manifests

# Manifests and lockfiles

Two artefacts, two jobs.

## The manifest — what we published

`GET /api/v1/collections/{id}/manifest`

The complete, versioned description of a collection: identity, version,
generation time, ruleset, collection-level qualification with every criterion,
licence composition, size, source repositories, and one entry per dataset with
its identifiers, creators, publisher, scores, licence, classification, formats,
file summary, provenance, citation, unknown fields and warnings.

`include_files=true` adds per-file URLs, media types, byte sizes and any
published checksums.

Every manifest validates against
[collection-manifest-1.0.0.json](https://fairdata.ai/schemas/collection-manifest-1.0.0.json),
and that validation runs in CI against real generated output — a published
schema that is not enforced is decoration.

### Size, and asking for compression

A manifest for a large collection runs to several megabytes — the biggest is
around eight. Send `Accept-Encoding: gzip` and it is roughly twenty times
smaller. This is not only about bandwidth: some standard-library HTTP clients
truncate an uncompressed response of that size read through a CDN, and a
truncated manifest is a failed parse rather than a silent problem. Both SDKs
request compression; the documented examples do too.

If you only need to know whether a collection has changed, a `HEAD` request
returns `X-FAIRdata-Manifest-Checksum` with no body at all.

### The checksum

`manifest_checksum` is sha256 over the canonical JSON of the manifest's content,
excluding three fields that describe *the run that served the response* rather
than the collection: `manifest_checksum` itself, `retrieved_at`, and
`pipeline_version`.

That last one matters. `pipeline_version` is the deployed build; leaving it in
meant every backend deploy changed every checksum while nothing about the data
moved, so every lockfile reported a change that had not happened. The checksum is
now stable across deploys and changes only when the collection's membership, a
licence, a score or the ruleset changes.

Use `collection_version` to ask "are these the same datasets?" and
`manifest_checksum` to ask "is anything about them different?".

## The lockfile — what you took

`GET /api/v1/collections/{id}/receipt` → `fairdata.lock.json`

Records the collection id and exact version, the manifest checksum and URL, the
ruleset, the retrieval timestamp, the client, every selected dataset with its
licence and file counts, and **everything skipped with a reason**.

Validates against
[fairdata-lock-1.0.0.json](https://fairdata.ai/schemas/fairdata-lock-1.0.0.json).

In the interface we call this the robot receipt. The file and every field name
are literal.

### Why the skipped list matters

A pipeline that quietly drops 40 datasets because their licence would not
resolve produces a result nobody can explain later. The schema has a required
`reason` on every skip for that reason.

## Which to keep

Both. The manifest says what was available; the lockfile says what you used.
Together they answer "which data produced this result, and could I get them
again?" — which is the question that matters when someone asks about your
figures a year later.

## Fields that are deliberately absent

The manifest does not invent values. Where we hold nothing, you get `null`, an
entry in the dataset's `unknown_fields`, and usually a warning explaining the
consequence. You will see this most often for:

- `licence.identifier` — nothing could be resolved
- `files.total_bytes_complete: false` — at least one file reports no size
- `checksum_algorithm: null` — the source publishes no checksum
- `citation: null` — we hold too few fields to build one honestly

---

# Concepts / Croissant and output formats
Source: https://fairdata.ai/developers/concepts/formats

# Croissant and output formats

## Collection formats

| Format | Endpoint | Use it for | It loses |
| --- | --- | --- | --- |
| Manifest JSON | `/manifest` | Anything reproducible. The canonical form. | Nothing |
| Manifest + files | `/manifest?include_files=true` | Downloading. Per-file URLs and checksums. | Nothing |
| JSONL | `/datasets.jsonl` | Streaming large collections | Collection-level qualification and licence summary |
| CSV | `/datasets.csv` | Spreadsheets, quick dataframes | Per-criterion evidence and per-file detail — the header says so |
| Croissant JSON-LD | `/croissant` | ML tooling that speaks Croissant | Per-criterion evidence |
| Receipt | `/receipt` | Recording what you consumed | Not a description of the collection |

Any format that loses licence, provenance or qualification information says so
where you get it, not in a footnote.

## Croissant

Each collection is a `schema.org/Dataset` conforming to Croissant 1.0, whose
`distribution` lists every member as a `cr:FileObject` pointing at that
dataset's own Croissant record, carrying that dataset's own `cr:license`.

The collection-level `license` field describes **the collection description**
(CC-BY-4.0), not its members — a single member licence would misrepresent a
mixed set. The composition is in `usageInfo`, in prose written from the actual
data.

```bash
curl -sS "https://fairdata.ai/api/v1/collections/41-environmental-sciences/croissant" | jq '{name, version, usageInfo, members: (.distribution | length)}'
```

## Dataset formats

Every record is available in six serialisations, unchanged from before v1:

```
GET /api/record/{doi}              FAIRdata.ai record
GET /api/record/{doi}/schema.org   schema.org/Dataset JSON-LD
GET /api/record/{doi}/croissant    Croissant 1.0
GET /api/record/{doi}/ro-crate     RO-Crate 1.2
GET /api/record/{doi}/fdp          FAIR Data Point DCAT3
GET /api/record/{doi}/datacite     Enriched DataCite JSON
GET /api/record/{doi}/ai-readiness GDS 4-pillar assessment
GET /api/record/{doi}/files        Resolved file listing
```

## Schemas

| Schema | URL |
| --- | --- |
| Collection manifest | [collection-manifest-1.0.0.json](https://fairdata.ai/schemas/collection-manifest-1.0.0.json) |
| Lockfile | [fairdata-lock-1.0.0.json](https://fairdata.ai/schemas/fairdata-lock-1.0.0.json) |
| Qualification ruleset | [ruleset-machine-ready.json](https://fairdata.ai/schemas/ruleset-machine-ready.json) |
| OpenAPI 3.1 | [/api/openapi.json](https://fairdata.ai/api/openapi.json) |

## What we do not add

We have not added linked-data serialisations for the sake of a longer feature
list. RO-Crate and FAIR Data Point exist on records because they are genuinely
used in research-data infrastructure. If a format would not be consumed by
anyone, its absence is deliberate.

---

# Concepts / FAIR versus model readiness, and responsible use
Source: https://fairdata.ai/developers/concepts/responsible-use

# FAIR versus model readiness, and responsible use

## The claim we do not make

**Machine-readable does not mean permission to train.**

A FAIR score is a measurement of metadata quality. It is not, and cannot be,
evidence that:

- the underlying data are scientifically correct
- the data are representative or unbiased
- the data are safe for every use
- the data are legally permitted for model training, or for your particular use
- the dataset has been peer reviewed by a person
- the data are free of personal, sensitive or identifiable information

Qualification tells you that a machine could find, read, and legally *appear* to
be allowed to use this — as far as an automated check of the deposited metadata
can tell. That is a genuinely useful filter. It is not a clearance.

## The dimensions we keep apart

Each of these is reported separately, and none of them substitutes for another:

| Dimension | Answers |
| --- | --- |
| FAIRness | How well does the metadata follow the FAIR principles? |
| Metadata completeness | Which descriptive fields are present? |
| Curation readiness | How much curation evidence accompanies the data? (model-assisted) |
| File accessibility | Can the files actually be fetched? |
| Licence and stated rights | Do the rights statements resolve to one unambiguous licence? |
| Scientific evidence | Is the science sound? **We do not assess this.** |
| Provenance | Is the origin of each fact recorded? |
| Responsible use | What must a human judge before using this? |
| Machine-ingestion compatibility | Can the formats be loaded without bespoke parsing? |

A single composite would make the strong signals meaningless and the missing
ones invisible.

## What is always unknown to us

For every dataset, regardless of score:

- whether the data contain personal, sensitive or identifiable information
- whether the consent obtained covers your intended use
- whether the licence in the metadata matches what the repository shows today
- whether third-party rights apply to material inside the files
- whether the data are accurate, representative or fit for your purpose
- whether laws applying to you impose additional restrictions

`check_usage_constraints` (MCP) returns exactly this list, plus the licence
signals we do hold, and `determination: "not_determined_by_this_service"`. It
does not return a boolean, because a boolean would be a claim we are not in a
position to make.

## What we ask of downstream consumers

If you build on this data:

1. **Carry the limits forward.** Every qualification response includes a `limits` array. Show it wherever you show a status.
2. **Do not collapse unknown.** `null` licence is not "unlicensed" and not "permitted".
3. **Decide per dataset.** Collection-level licence badges are wrong for mixed sets.
4. **Do not present generated values as depositor claims.** Evidence marked `generated: true` was produced by us or by a model.
5. **Record what you used.** A lockfile costs one request and answers most later questions.

## If we have got something wrong

An incorrect licence reading or a wrongly excluded dataset is a real problem,
and we would rather hear about it. Report it via
[support](/developers/operations/support) with the DOI and what the source
actually shows. Corrections go through the pipeline and change the collection
version, so they appear as a visible change rather than a quiet edit.

---

# Reference / REST API
Source: https://fairdata.ai/developers/reference/rest-api

# REST API

Base URL: `https://fairdata.ai/api/v1`. No authentication. All responses are
UTF-8 JSON with UTC ISO 8601 timestamps unless stated otherwise.

The machine-readable contract is the [OpenAPI document](https://fairdata.ai/api/openapi.json).
This page is the narrative version of the same thing.

## Service

### `GET /meta`

Capability discovery — the one call to make first. Returns the API version, the
current collection version, the full ruleset, every schema URL, the MCP endpoint,
the rate limit, and the complete error catalogue.

### `GET /health`

`status` is `ok`, `degraded` or `unavailable`, with a `checks` object reporting
the collection version and how old it is. Deliberately free of infrastructure
detail. `Cache-Control: no-store`.

### `GET /qualification/ruleset`

The Machine-Ready ruleset: criteria, thresholds, evidence types, statuses,
dimensions and limits. See [qualification](/developers/concepts/qualification).

## Collections

### `GET /collections`

| Parameter | Type | Notes |
| --- | --- | --- |
| `q` | string | Free text over title and definition |
| `repository` | string | Contains datasets from this repository |
| `licence` | string | Contains this SPDX identifier |
| `modality` | string | Contains this modality |
| `min_datasets` | integer | ≥ 0 |
| `min_fair` | number | 0–100, against the FAIR median |
| `status` | string | A qualification status |
| `sort` | string | `datasets` (default), `smallest`, `fair`, `curation`, `licence-clarity`, `title` |
| `limit` | integer | 1–500, default 50 |
| `cursor` | string | From `pagination.cursor` |

Returns `meta`, `collections`, `pagination`, `links`, `corpus` and `limits`.

### `GET /collections/{collection_id}`

Accepts the canonical slug (`31-biological-sciences`), the bare Field of Research
division code (`31`), the cross-cutting slugs (`cited-in-research`,
`unclassified`) and the legacy theme keys. Optional `version` pins.

### `GET /collections/{collection_id}/qualification`

The criteria matrix plus `dataset_exceptions` — members whose own evaluation is
not a clean pass.

### `GET /collections/{collection_id}/datasets`

| Parameter | Notes |
| --- | --- |
| `q` | Substring on title and DOI |
| `licence`, `repository`, `modality` | Exact match |
| `has_tabular` | boolean |
| `cited` | boolean; at least one linked publication |
| `min_fair`, `min_curation` | 0–100 |
| `for` | ANZSRC Field of Research code prefix (2, 4 or 6 digits) |
| `sort` | `citations` (default), `fair`, `curation`, `recent`, `title` |
| `limit`, `cursor` | Pagination |

Also returns `facets` — licence, repository and modality counts over the whole
collection, for building filter controls with real numbers.

### `GET /collections/{collection_id}/datasets.jsonl`

One dataset per line, streamed, no pagination.

### `GET /collections/{collection_id}/datasets.csv`

Flat summary. Lossy, and the first line says which fields it drops.

### `GET /collections/{collection_id}/manifest`

Optional `version` and `include_files`. Response headers carry
`X-FAIRdata-Manifest-Checksum` and `X-FAIRdata-Collection-Version`, so a
`HEAD` request answers "has this changed?" without downloading anything.

**Send `Accept-Encoding: gzip`.** The largest collections produce manifests of
several megabytes, and roughly twenty times less compressed. Some standard-library
HTTP clients — Python's `urllib` among them — truncate a response that size read
uncompressed through a CDN. Both SDKs request compression already.

### `GET /collections/{collection_id}/croissant`

Croissant 1.0 JSON-LD, served as `application/ld+json`.

### `GET /collections/{collection_id}/receipt`

`fairdata.lock.json`. Optional repeated `doi` restricts the selection; anything
that is not a member is listed under `skipped` with a reason. `Cache-Control: no-store`.

### `GET /collections/versions` and `GET /collections/{id}/versions`

Published versions, newest first, with a `retention` block stating that
superseded member lists are not retained.

## Datasets

### `GET /datasets/resolve?doi={doi}`

The recommended lookup. Accepts a bare DOI, `doi:10.…` or `https://doi.org/10.…`.

### `GET /datasets/{identifier}`

`identifier` may be the DOI as unpadded URL-safe base64, or the literal DOI
path. Both resolve to the same record.

### `GET /datasets/{identifier}/qualification`

The per-criterion evidence for one dataset.

## Search

### `GET /search?q=…`

Case-insensitive substring over titles and DOIs, restricted to the qualifying
corpus. Both facts are stated in the response, under `query.match` and `scope`.
For semantic search across the whole registry, use `/api/search/semantic`.

## Response envelope

```json
{
  "meta": {
    "api_version": "v1",
    "api_revision": "2026-08-19",
    "collection_version": "3ee97c2ecc9d81fb",
    "generated_at": "2026-08-19T20:49:14Z",
    "request_id": "…"
  },
  "…": "the resource",
  "limits": ["Machine-readable does not mean permission to train. …"]
}
```

## Legacy routes

Everything under `/api/` that existed before v1 still works unchanged, including
`/api/collections`, `/api/collections/{theme}`, `/api/collections/{theme}.jsonl`,
`/api/collections/{theme}/croissant`, `/api/corpus`, `/api/record/{doi}` and its
format variants, `/api/datasets`, and the badge, sitemap and passport routes.
Collection routes now carry a `Link` header naming their v1 successor. Nothing
is scheduled for removal — see the [migration guide](/developers/reference/migration).

---

# Reference / Errors
Source: https://fairdata.ai/developers/reference/errors

# Errors

Every documented JSON endpoint returns `application/problem+json` on failure
(RFC 9457). **A JSON endpoint never returns an HTML error page.**

```json
{
  "type": "https://fairdata.ai/errors/collection-not-found",
  "title": "Collection not found",
  "status": 404,
  "detail": "No collection matches '99-nope' in collection version 3ee97c2ecc9d81fb.",
  "code": "collection_not_found",
  "request_id": "8f3c…",
  "instance": "/api/v1/collections/99-nope",
  "parameter": "collection_id",
  "documentation_url": "https://fairdata.ai/developers/reference/errors"
}
```

**Branch on `code`.** It is stable. `title` and `detail` are for humans and may
be reworded without a version bump.

## The catalogue

| Code | Status | Means | Do |
| --- | --- | --- | --- |
| `collection_not_found` | 404 | The identifier matches no collection in the current snapshot | List `/collections` and use an `id` from the response |
| `dataset_not_found` | 404 | No qualifying dataset with that identifier | Check the DOI; the response includes `registry_record_url` if it exists in the wider registry |
| `version_not_available` | 409 | The pinned version is not the published one, and superseded member lists are not retained | Compare your `manifest_checksum` against the current manifest to see what changed |
| `invalid_parameter` | 422 | A parameter was outside its documented range or vocabulary | Check the OpenAPI document; `parameter` names the offender |
| `invalid_cursor` | 422 | The cursor was issued for a different query or an older collection version | Restart the listing and follow `links.next` |
| `snapshot_unavailable` | 503 | The published snapshot could not be read. **No empty result has been substituted.** | Retry after `Retry-After`; check [status](/developers/operations/status) |
| `rate_limited` | 429 | Too many requests in the window | Wait for `Retry-After`; see the rate-limit headers |
| `not_acceptable` | 406 | None of the requested media types is available | Request `application/json` |
| `internal_error` | 500 | An unexpected failure | Retry once; report the `request_id` |

The same list is served at `GET /api/v1/meta` under `errors`, so a client can
discover it without scraping this page.

## The important one

`snapshot_unavailable` is a **503, not an empty 200**. When the collection
snapshot cannot be read, the API refuses rather than reporting that there are no
collections. If you are building on this, preserve that distinction: an
interface that renders "no results" for a service failure is telling its users
something false.

## Request ids

Every response carries `X-Request-Id`. Send your own and we echo it. Quote it
when reporting a problem — it is what lets us find the specific request.

## Client patterns

```python
import urllib.error, urllib.request, json, time

def get(url, attempts=3):
    for attempt in range(attempts):
        try:
            with urllib.request.urlopen(url, timeout=60) as r:
                return json.load(r)
        except urllib.error.HTTPError as e:
            problem = json.load(e) if e.headers.get("content-type", "").startswith("application/problem") else {}
            code = problem.get("code")
            if code == "rate_limited" and attempt < attempts - 1:
                time.sleep(int(e.headers.get("Retry-After", 5))); continue
            if code == "snapshot_unavailable" and attempt < attempts - 1:
                time.sleep(int(e.headers.get("Retry-After", 60))); continue
            raise RuntimeError(f"{code}: {problem.get('detail')} (request {problem.get('request_id')})")
```

Retry `429` and `503`. Do not retry `404`, `409` or `422` — they will not
become true.

---

# Reference / Pagination, filtering, caching and rate limits
Source: https://fairdata.ai/developers/reference/pagination-filtering-caching

# Pagination, filtering, caching and rate limits

## Pagination

Cursor-based. The cursor encodes the offset **and** a fingerprint of the query
and the collection version it was issued against.

```json
{
  "pagination": { "limit": 50, "offset": 0, "returned": 50, "total": 955,
                  "has_more": true, "cursor": "eyJvIjo1MCwiZiI6…" },
  "links": { "self": "…", "next": "…", "first": "…" }
}
```

Follow `links.next` until it is `null`. Do not construct cursors.

If you change a filter or sort mid-listing, or the collection version moves, the
cursor is rejected with `invalid_cursor` — deliberately, because continuing
would page through a different set and silently drop or duplicate rows.

`limit` is 1–500, default 50.

## Filtering

All filters are AND-combined. Unknown values produce `invalid_parameter` with
the offending `parameter` named, rather than being ignored — an ignored filter
is a wrong answer that looks right.

Filter values are exact matches except `q`, which is a case-insensitive
substring, and `for`, which is a Field of Research code **prefix**, so `31`
matches `3101` and `310501`.

## Caching

| Header | Behaviour |
| --- | --- |
| `Cache-Control` | `public, max-age=300, stale-while-revalidate=600` on listings; 600s on manifests; `no-store` on receipts and health |
| `ETag` | Weak validator derived from the collection version and the query |
| `Last-Modified` | When the collection snapshot was generated |
| `Vary` | `Accept, Accept-Encoding, Origin` |

Send `If-None-Match` and you will get `304 Not Modified` when nothing has
changed — which, for a collection that regenerates a few times a day, is most of
the time.

```bash
ETAG=$(curl -sSI "https://fairdata.ai/api/v1/collections" | grep -i '^etag:' | cut -d' ' -f2- | tr -d '\r')
curl -sS -o /dev/null -w '%{http_code}\n' -H "If-None-Match: $ETAG" "https://fairdata.ai/api/v1/collections"
# 304
```

Compression is negotiated normally through `Accept-Encoding`.

## Rate limits

Public reads need no key, and the limit exists only to stop one client starving
the rest.

| Header | Meaning |
| --- | --- |
| `RateLimit-Limit` | Requests allowed in the window |
| `RateLimit-Remaining` | Requests left |
| `RateLimit-Reset` | Seconds until the window resets |
| `RateLimit-Policy` | e.g. `240;w=60` |

Currently **240 requests per 60 seconds per client IP, per worker process**. The
per-worker part is stated because it is true: the effective ceiling across a
multi-worker deployment is higher, and we would rather describe the mechanism
than claim a global quota we do not enforce.

Exceeding it returns `429` with code `rate_limited` and a retry hint. Back off
on `RateLimit-Remaining` approaching zero rather than waiting to be refused.

## CORS

`GET` on the public read endpoints is available to browser clients from any
origin. Credentialed requests are not, because these endpoints require no
credentials.

---

# Reference / Collection manifest schema
Source: https://fairdata.ai/developers/reference/manifest-schema

# Collection manifest schema

Canonical schema:
[collection-manifest-1.0.0.json](https://fairdata.ai/schemas/collection-manifest-1.0.0.json)
(JSON Schema 2020-12). Every generated manifest is validated against it in CI.

## Top level

| Field | Type | Notes |
| --- | --- | --- |
| `manifest_schema_version` | string | Semantic; `1.0.0` |
| `collection` | object | `id`, `legacy_theme`, `title`, `kind`, `description`, `purpose`, `url`, `for_code` |
| `version` | string \| null | The pinnable collection version |
| `generated_at` | date-time | When the snapshot was built |
| `retrieved_at` | date-time | When this response was produced. **Excluded from the checksum.** |
| `ruleset` | object | `id`, `version`, `uri`, `name` |
| `methodology_version` | string | Scoring framework version |
| `pipeline_version` | string | Backend commit that produced it. **Excluded from the checksum**, so a deploy does not look like a data change. |
| `qualification` | object | Collection-level status and full criteria matrix |
| `limits` | string[] | What qualification does not establish. **Carry these forward.** |
| `licence_summary` | object | Composition, mixed flag, unresolved count, prose statement |
| `dataset_count` | integer | |
| `size` | object | `bytes`, `complete`, `datasets_measured`, `datasets_total` |
| `source_repositories` | object | Repository → count |
| `datasets` | array | One entry per member |
| `manifest_checksum` | string | `sha256:…` |

## Each dataset

| Field | Notes |
| --- | --- |
| `id` | The DOI as unpadded URL-safe base64 |
| `doi`, `title`, `creators`, `publisher`, `publication_year` | Attribution |
| `source_repository`, `source_landing_page` | Where it lives |
| `assessed_at` | When we last assessed it |
| `qualification` | Status plus the full per-criterion matrix |
| `scores` | `fair`, `curation_readiness`, and the `basis` they were computed on |
| `licence` | `identifier` (nullable), `url`, `rights_statements`, `attribution_required` |
| `classification` | `for_codes` with method and confidence, `subdomains`, `modalities` |
| `formats` | Declared media types |
| `files` | `resolved_count`, `total_bytes`, `total_bytes_complete`, `with_checksum`, `media_types`, `listing_url` |
| `file_list` | Present only with `include_files=true` |
| `citation_count` | Linked publications in our index |
| `provenance` | Record URL, Croissant URL, pipeline version |
| `citation` | Built only from fields we hold; `null` otherwise |
| `unknown_fields` | Criteria whose value is unknown for this dataset |
| `warnings` | Plain-language consequences of what is missing |

## Null semantics

`null` means unknown, never zero and never false.

| You see | It means |
| --- | --- |
| `licence.identifier: null` | No licence resolved. Not "unlicensed", not "permitted". |
| `size.complete: false` | `bytes` is a lower bound. |
| `checksum_algorithm: null` | The source publishes no checksum. Downloads cannot be verified against it. |
| `citation: null` | Too few fields to build a citation honestly. |
| A criterion at `unknown` | Not assessed. Not a failure. |

## Lockfile

[fairdata-lock-1.0.0.json](https://fairdata.ai/schemas/fairdata-lock-1.0.0.json).

| Field | Notes |
| --- | --- |
| `lockfile_version` | `1.0.0` |
| `collection_id`, `collection_version` | What you drew from |
| `manifest_checksum`, `manifest_url` | How to detect change |
| `ruleset` | Which criteria were in force |
| `retrieved_at`, `client` | When, and by what |
| `datasets[]` | Per dataset: `doi`, `assessed_at`, `licence`, scores, file counts, and optionally per-file records with `checksum_verified` (nullable) |
| `skipped[]` | `doi`/`filename` and a **required** `reason` |
| `verification` | How to check the checksum later |

## Validating locally

```bash
pip install jsonschema requests
python - <<'PY'
import json, requests
from jsonschema import Draft202012Validator
schema = requests.get("https://fairdata.ai/schemas/collection-manifest-1.0.0.json").json()
manifest = requests.get("https://fairdata.ai/api/v1/collections/41-environmental-sciences/manifest").json()
errors = list(Draft202012Validator(schema).iter_errors(manifest))
print("valid" if not errors else errors[:3])
PY
```

---

# Reference / MCP tools and resources
Source: https://fairdata.ai/developers/reference/mcp

# MCP tools and resources

Endpoint: `https://fairdata.ai/api/mcp` (Streamable HTTP, JSON-RPC 2.0).
Discovery: `https://fairdata.ai/.well-known/mcp.json`. No authentication.

## Protocol

Speaks `2026-07-28` (current) and negotiates down to `2025-11-25`,
`2025-06-18`, `2025-03-26` and `2024-11-05`. Send the version in the
`MCP-Protocol-Version` header or in `params._meta`; handshake-era clients get it
from `initialize`. An unsupported version returns an error listing what is
supported, rather than failing opaquely.

`server/discover` is implemented, so one call returns supported versions,
capabilities, identity and instructions.

## Tools

Collection tools call the same service layer as the REST API — they cannot
disagree with it. Every tool declares an `outputSchema` and returns
`structuredContent` alongside a text rendering of the same JSON.

| Tool | Does |
| --- | --- |
| `get_ruleset` | The ruleset: criteria, thresholds, evidence types, statuses, limits |
| `search_collections` | Find collections, with their real aggregates |
| `get_collection` | One collection in full |
| `compare_collections` | 2–5 collections across separate dimensions, deliberately not one score |
| `list_collection_datasets` | Members, filtered and paginated |
| `get_collection_manifest` | The versioned manifest; large ones return a URL and checksum rather than inline |
| `explain_collection_qualification` | Criteria matrix plus dataset-level exceptions |
| `search_datasets_in_corpus` | Substring search over qualifying datasets |
| `get_dataset` | One qualifying dataset |
| `explain_dataset_qualification` | Per-criterion evidence for one dataset |
| `create_ingestion_plan` | Access level, ordered steps, cautions, runnable code |
| `create_reproducibility_receipt` | `fairdata.lock.json` |
| `check_usage_constraints` | Licence signals, obligations, unknowns — never a yes/no |

Legacy record tools are unchanged and still available: `get_record`,
`get_record_format`, `list_records`, `get_ai_readiness`, `get_files`,
`get_data_quality`, `find_similar_datasets`, `search_datasets` (semantic, whole
registry), `get_linked_papers`, `assess_dataset`, and `find_training_data`
(deprecated in favour of `search_collections`).

## The permission question

`check_usage_constraints` returns:

```json
{
  "determination": "not_determined_by_this_service",
  "licence_signals": [{ "licence": "CC-BY-4.0", "datasets": 368, "evidence_type": "source_metadata" }],
  "permits": ["CC-BY-4.0: Use, adaptation and redistribution, including commercially."],
  "requires": ["CC-BY-4.0: Attribution to the creators, a link to the licence, and an indication of any changes made."],
  "unknowns": ["Whether the data contain personal, sensitive or identifiable information.", "…"],
  "your_checks": ["Open the source landing page and read the rights statement as the repository currently states it.", "…"],
  "limits": ["Machine-readable does not mean permission to train. …"]
}
```

`determination` is a constant. There is no code path that returns a boolean,
because a licence signal is not a legal conclusion.

## Resources

| URI | Content |
| --- | --- |
| `fairdata://collections` | Every collection with aggregates |
| `fairdata://collections/{collection_id}` | One collection |
| `fairdata://collections/{collection_id}/manifest` | Its manifest |
| `fairdata://collections/{collection_id}/qualification` | Its criteria matrix |
| `fairdata://datasets/{doi}` | One qualifying dataset |
| `fairdata://datasets/{doi}/qualification` | Its criteria matrix |
| `fairdata://schemas/collection-manifest` | The manifest JSON Schema |
| `fairdata://schemas/fairdata-lock` | The lockfile JSON Schema |
| `fairdata://methodology/qualification` | The ruleset |
| `fairdata://records` | The wider record registry |

## Prompts

`evaluate_collection_for_use` and `prepare_reproducible_ingestion`. Both walk an
agent through keeping the dimensions separate and reporting unknowns rather than
smoothing them.

## Errors

Unknown tool or malformed request → JSON-RPC protocol error. Anything a model
could fix — a bad argument, an unknown collection, an unavailable pinned
version — comes back as a tool result with `isError: true` and a message saying
what to do instead.

Argument validation is strict: required arguments, unknown arguments, wrong
types, enum violations and array bounds are all rejected with an actionable
message.

## Health

```bash
curl -sS https://fairdata.ai/api/mcp/health
curl -sS https://fairdata.ai/api/mcp/tools     # human-readable listing
```

---

# Reference / SDKs
Source: https://fairdata.ai/developers/reference/sdks

# SDKs

Two first-party clients, both thin: they wrap the public API and add ergonomics,
never business logic. Qualification, licence normalisation and aggregation
happen server-side, so an SDK cannot drift from the API.

Both live in this repository under `clients/` and are built and tested in CI.
They are **not yet published** to PyPI or npm — see
[the release process](/developers/reference/cli#releases) for how that will work.

## Python

```bash
pip install ./clients/python        # from a checkout, until it is published
```

```python
from fairdata import FairData

client = FairData()                       # or FairData(base_url=..., timeout=..., user_agent=...)

# List and filter
collections = client.collections.list(sort="smallest", min_datasets=20)
for c in collections:
    print(c.id, c.dataset_count, c.licence_summary["distinct_licences"])

# One collection
collection = client.collections.get("41-environmental-sciences")
print(collection.qualification.status, collection.version)

# Iterate every member without thinking about pagination
for dataset in collection.datasets(licence="CC0-1.0"):
    print(dataset["doi"])

# Pin, then download with verification and a receipt
manifest = collection.manifest(version=collection.version, include_files=True)
receipt = collection.pull(dest="./data", version=collection.version,
                          verify_checksums=True, streaming=True)
receipt.write("fairdata.lock.json")
```

Provides typed-ish dataclasses, automatic pagination, sensible timeouts,
retries on `429`/`5xx` for safe requests only, `FairDataError` carrying the API
`code` and `request_id`, checksum verification, streaming downloads, and no
telemetry of any kind.

## TypeScript

```bash
npm install ./clients/typescript    # from a checkout, until it is published
```

```typescript
import { FairData } from "@fairdata/sdk";

const fairdata = new FairData();

const collections = await fairdata.collections.list({ sort: "smallest" });
const collection = await fairdata.collections.get("41-environmental-sciences");

const manifest = await collection.manifest({ version: collection.version });
console.log(manifest.dataset_count, manifest.manifest_checksum);

for await (const dataset of collection.datasets({ licence: "CC0-1.0" })) {
  console.log(dataset.doi);
}

const receipt = await collection.receipt({ version: collection.version });
await receipt.write("fairdata.lock.json");
```

Full TypeScript types, async iteration over pages, `AbortSignal` support,
`FairDataError` with `code` and `requestId`, and zero runtime dependencies.

## R and Julia

No full SDK — the API is simple enough that a client would add little — but
first-class, tested examples are maintained:

```r
library(httr2); library(jsonlite)
manifest <- request("https://fairdata.ai/api/v1/collections/41-environmental-sciences/manifest") |>
  req_perform() |> resp_body_json()
licensed <- Filter(function(d) !is.null(d$licence$identifier), manifest$datasets)
cat(length(licensed), "of", manifest$dataset_count, "carry a resolved licence\n")
```

```julia
using HTTP, JSON3
r = HTTP.get("https://fairdata.ai/api/v1/collections/41-environmental-sciences/manifest")
manifest = JSON3.read(r.body)
licensed = [d for d in manifest.datasets if d.licence.identifier !== nothing]
println(length(licensed), " of ", manifest.dataset_count, " carry a resolved licence")
```

## Versioning

Semantic versioning. The SDK major version is independent of the API version: a
1.x SDK targets API v1, and would move to 2.x only for a breaking change in the
SDK's own interface.

---

# Reference / CLI
Source: https://fairdata.ai/developers/reference/cli

# CLI

Ships with the Python package. No configuration, no credentials.

```bash
pip install ./clients/python
fairdata --help
```

## Commands

```bash
fairdata collections list [--sort smallest] [--limit 20] [--json]
fairdata collections inspect <id> [--json]
fairdata collections manifest <id> [--version V] [--include-files] [--out FILE]
fairdata collections pull <id> [--version V] [--out DIR] [--verify] [--metadata-only]
fairdata collections qualification <id> [--json]
fairdata datasets get --doi <doi> [--json]
fairdata datasets explain --doi <doi> [--json]
fairdata schema [manifest|lockfile|ruleset]
fairdata status
```

## Output modes

Human-readable tables by default; `--json` for machine consumption on stdout,
with diagnostics on stderr so a pipe stays clean. `--quiet` suppresses
progress output.

```bash
fairdata collections list --json | jq '.[] | select(.dataset_count < 100) | .id'
```

## Exit codes

| Code | Meaning |
| --- | --- |
| 0 | Success |
| 1 | Not found (collection or dataset) |
| 2 | Invalid usage or parameter |
| 3 | The pinned version is unavailable |
| 4 | Rate limited |
| 5 | Service unavailable — retry later |
| 6 | Checksum verification failed |

Distinct codes so a CI job can tell "this will never work" from "try again".

## Examples

```bash
# What is the smallest collection with clear licensing?
fairdata collections list --sort licence-clarity --json | jq -r '.[0].id'

# Pin a version and pull metadata only
fairdata collections pull 41-environmental-sciences --metadata-only --out ./data

# Check whether a collection has changed since your lockfile
fairdata collections manifest 41-environmental-sciences --json \
  | jq -r .manifest_checksum
jq -r .manifest_checksum fairdata.lock.json
```

## Releases

Packages are built and tested in CI on every commit. Publishing to PyPI and npm
requires credentials and an explicit release approval, and is a separate,
tagged workflow — see `docs/sdk-release-process.md` in the repository. Nothing is
published automatically from a merge.

---

# Reference / Versioning policy and migration
Source: https://fairdata.ai/developers/reference/migration

# Versioning policy and migration

## Five things carry versions

| Thing | Format | Changes when |
| --- | --- | --- |
| API version | `v1` in the path | A breaking change to a response shape |
| API revision | `2026-08-19` in `meta.api_revision` | An additive change |
| Collection version | Content hash | Membership changes |
| Ruleset version | Semantic | Criteria or thresholds change |
| Schema version | Semantic | Manifest or lockfile structure changes |

## What may change without notice

Additive changes only: new endpoints, new response fields, new enum values, new
optional parameters, reworded human-readable text (`title`, `detail`,
descriptions).

**Write clients that ignore unknown fields.** They will appear.

## What may not

Removing or renaming a field, changing a type, changing the meaning of a value,
tightening a default, or removing an endpoint. Any of those requires a new major
version served in parallel, with the old one kept for at least twelve months
from the announcement.

Machine-readable identifiers are stable: error `code` values, criterion `id`
values, qualification status values, collection `id` values.

## Deprecation

When something is deprecated it gets a `Deprecation` header, a `Sunset` header
if a removal date is set, a `Link` header pointing at the successor, an entry in
the [changelog](/developers/operations/changelog), and a migration guide. Nothing
is removed without all five.

## Migrating from the pre-v1 routes

The old routes still work. Nothing is scheduled for removal. They now carry a
`Link` header naming their v1 successor.

| Old | New | What changes |
| --- | --- | --- |
| `GET /api/collections` | `GET /api/v1/collections` | Real aggregates per collection; cursor pagination; slug ids instead of bare theme keys |
| `GET /api/collections/{theme}` | `GET /api/v1/collections/{id}` | Qualification object with evidence; the bare theme key still resolves |
| `GET /api/collections/{theme}.jsonl` | `GET /api/v1/collections/{id}/datasets.jsonl` | Richer per-dataset shape |
| `GET /api/collections/{theme}/croissant` | `GET /api/v1/collections/{id}/croissant` | `conformsTo`, per-member licences, `usageInfo` |
| `GET /api/corpus` | `GET /api/v1/search` + collection filters | Structured errors, cursor pagination |
| n/a | `GET /api/v1/collections/{id}/manifest` | New: versioned, checksummed, schema-validated |
| n/a | `GET /api/v1/collections/{id}/receipt` | New: the lockfile |

### Identifier mapping

Collection ids gained a human-readable slug. Both forms resolve:

```
31                       →  31-biological-sciences
41                       →  41-environmental-sciences
proven                   →  cited-in-research
unclassified             →  unclassified
genomics, climate, …     →  unchanged (legacy keyword collections)
```

### Field mapping

| Old | New |
| --- | --- |
| `ai_ready_count` | `dataset_count` |
| `theme` | `id` (slug) and `legacy_theme` (old key) |
| `label` | `title` |
| `license` | `licence` |
| `criteria` (string) | `qualification.criteria` (array with evidence) |
| `caveats` | `limits` |
| `snapshot_id` | `version` / `meta.collection_version` |

### A minimal migration

```diff
- r = requests.get("https://fairdata.ai/api/collections/41")
- datasets = r.json()["datasets"]
+ r = requests.get("https://fairdata.ai/api/v1/collections/41-environmental-sciences/datasets?limit=500")
+ datasets = r.json()["datasets"]
+ # or, for the full set with evidence and checksums:
+ manifest = requests.get("https://fairdata.ai/api/v1/collections/41-environmental-sciences/manifest").json()
```

---

# Operations / Service status
Source: https://fairdata.ai/developers/operations/status

# Service status

Live status is fetched from `GET /api/v1/health` and shown at the top of this
page. It reports the health of the data this API serves — deliberately without
naming hosts, dependencies or internal versions.

## What the states mean

| Status | Means | Do |
| --- | --- | --- |
| `ok` | The collection snapshot is readable and recent | Nothing |
| `degraded` | Readable, but more than six hours old | Usable; treat freshness-sensitive work with care |
| `unavailable` | The snapshot could not be read | Requests return `503` `snapshot_unavailable`. Retry with backoff. **You will not get an empty result instead.** |

## Indicators we hold ourselves to

| Indicator | Target |
| --- | --- |
| Public read availability | 99.5% monthly |
| `GET /api/v1/collections` latency | p95 under 800 ms warm |
| Manifest latency | p95 under 2 s |
| Collection freshness | Regenerated at least every 6 hours |
| Collection generation success | ≥ 95% of scheduled runs |
| Manifest schema validity | 100% — a manifest that fails validation is a build failure |
| Documentation example validity | 100% — every quickstart runs in CI |
| File-verification freshness | Resolution re-run within 90 days for qualifying datasets |

These are operational targets, not a contractual SLA. The service is free and
unauthenticated.

## Checking it yourself

```bash
curl -sS https://fairdata.ai/api/v1/health | jq
curl -sS https://fairdata.ai/api/mcp/health | jq
```

Both are safe to poll. Please do not poll more than once a minute; nothing
changes faster than that.

## When something is wrong

1. Check the status above.
2. Check [known issues](/developers/operations/known-issues).
3. If it is not there, [report it](/developers/operations/support) with the `request_id` from the response.

---

# Operations / Changelog
Source: https://fairdata.ai/developers/operations/changelog

# Changelog

Dates are the date of change. Ruleset changes that alter collection membership
are marked, because they are product changes rather than internal ones.

## 2026-08-19 — v1 API, manifests, receipts, MCP 2026-07-28

**Added**

- `/api/v1`: a stable, versioned public API with structured errors (RFC 9457), conditional GET, cursor pagination and rate-limit headers.
- Versioned **collection manifests** with a stable `manifest_checksum`, validated against a [published JSON Schema](https://fairdata.ai/schemas/collection-manifest-1.0.0.json).
- **Reproducibility receipts** (`fairdata.lock.json`) with their own [schema](https://fairdata.ai/schemas/fairdata-lock-1.0.0.json).
- `GET /api/v1/qualification/ruleset`: the Machine-Ready ruleset as data, now the single source of truth for every surface.
- Per-criterion **qualification evidence** for collections and datasets, including evaluator, version, evidence origin and timestamp.
- Collection **version history** with an explicit retention statement.
- Collection detail pages at `/collections/{id}`, with comparison, filters and shareable URL state.
- Python and TypeScript SDKs and a CLI, built and tested in CI.
- MCP: 13 collection tools with output schemas and structured content, resources, resource templates and prompts.

**Changed**

- MCP now speaks protocol `2026-07-28` and negotiates down to `2024-11-05`. Handshake-era clients are unaffected.
- MCP collection tools call the service layer directly instead of the public API over localhost, so an agent and an HTTP client cannot get different answers.
- Collection responses now derive `criteria` and `caveats` from the ruleset rather than restating them. **Wording changed; thresholds did not.**
- `/.well-known/mcp.json` now describes the transport, protocol versions, capabilities and resources.

**Fixed**

- `/.well-known/ai-plugin.json` advertised its OpenAPI document at `/api/docs/openapi.json`, which returns 404. It now points at `/api/openapi.json`.
- The public collection list omitted "declared file formats" from its stated criteria, although the gate had always applied it. The stated criteria are now generated from the ruleset and are complete.

**Compatibility**

- Every pre-v1 route is unchanged and still served. Collection routes now carry a `Link` header naming their v1 successor. Nothing is scheduled for removal.

**Fixed during rollout**

Four defects surfaced by deploying and then exercising the live service, rather
than by the test suite. Recorded here because a changelog that reports only the
clean parts is not much of a record.

- **Collections returned 503 for about eight minutes.** The gate selected `assessed_at` from a view that records its timestamp as `created_at`, so the query was rejected and both `/api/collections` and `/api/v1/collections` failed. Every test replaces the snapshot loader with a fixture, so none of them issued the query. There is now a test that reads the `.select(…)` calls out of the source and checks each column against the schema.
- **Large manifests truncated for standard-library HTTP clients.** The biggest manifest is 8 MB, and Python's `urllib` raises `IncompleteRead` on that through the CDN uncompressed — 419 KB compressed. Both SDKs already requested gzip; the documented examples did not, and now do.
- **`HEAD` on a manifest returned 405 from the origin.** The documented cheap change-check only worked when the CDN held a cached `GET`, because FastAPI does not add `HEAD` to a `GET` route the way plain Starlette does. It is now an explicit endpoint.
- **`manifest_checksum` included `pipeline_version`,** the deployed build, so every deploy told every lockfile the collection had changed when nothing about the data had. It is excluded now, alongside `retrieved_at`. Both SDKs were updated to match.

## 2026-08-20 — performance, coverage and checksums

**Fixed**

- **The repository scorecard and Atom feed returned 504.** They paged `datasets` 1,000 rows at a time and then fetched assessments 300 DOIs at a time — 155 sequential round trips for Zenodo's 35,458 datasets, about 28 seconds, past the gateway timeout. The same work inside Postgres measures ~90 ms and is now a single call.
- **Mendeley datasets could never qualify.** Mendeley was named as a supported repository and had no DOI pattern, so 210 datasets that cleared the score and licence gate could never have a file resolved. Its public API needs no key and publishes a sha-256 per file.
- **Dryad's checksums were being discarded.** Dryad publishes sha-256; a column named `md5` meant the resolver correctly declined to record it as an md5 and then dropped it. 3,507 of 4,873 Dryad files had no checksum while Dryad published one for every file. Manifests now report the real algorithm — telling a consumer "md5" for a sha-256 sends them to verify with the wrong function.
- **The file-resolution queue starved its own tail.** It took the first 1,000 DOIs in DOI order every night, so a run of failing DOIs at the head blocked everything behind them. Dryad had 3,577 gate-passing datasets waiting on files while its resolver answered in under a second. The window now rotates daily.
- **The registry dump returned an empty body.** `/api/catalog.jsonl` answered 200 with nothing in it, and the CDN cached that empty 200 for an hour. The origin was fine — 20.8 MB in 28 s — but it built the whole thing before yielding its first line: 36 sequential round trips, so nothing reached the proxy in time and the proxy forwarded nothing. It streams page by page now; time to first byte went from 27.9 s to 0.17 s. An unreachable store returns 503 rather than an empty 200, which a caller cannot tell apart from a registry holding no records.
- **Three live `ReferenceError`s on the admin page** (`pct`, `fairGrade`, `mapDbDataset`) and sixteen `target="_blank"` links without `rel="noreferrer"`. Found by turning on a linter the project had configured — for a framework it does not use — and never installed.

**Added**

- An [API playground](/developers/playground): send a real request to the public endpoints and read the status, headers and body. Fixed endpoint list, GET only, no credential field.
- Collection snapshot warming on startup, so the first request after a deploy no longer pays a ~28 s rebuild.
- A daily job that re-resolves file listings older than 90 days and fills checksums the source publishes.
- `checksum` and `checksum_algorithm` on every file record, alongside the retained `md5`.

## Ruleset history

### machine-ready 1.0.0 — 2026-08-19

First versioned release. Codifies the gate that had been applied since
collections launched, with no change to membership:

- `fair-score-minimum` ≥ 75
- `curation-readiness-minimum` ≥ 50
- `permissive-licence-signal` — one unambiguous permissive licence
- `declared-file-formats` — at least one
- `resolved-file-url` — at least one

Membership impact: **none**. The same datasets qualify before and after.

## Schema history

| Schema | Version | Date | Change |
| --- | --- | --- | --- |
| collection-manifest | 1.0.0 | 2026-08-19 | First release |
| fairdata-lock | 1.0.0 | 2026-08-19 | First release |

---

# Operations / Known issues and limitations
Source: https://fairdata.ai/developers/operations/known-issues

# Known issues and limitations

## Data

**Coverage is concentrated and not representative.** The registry is a growing
sample, heavily weighted towards a small number of repositories. `corpus.repositories`
on the collection index reports the actual distribution. Do not treat a
collection as a sample of a field.

**Discipline classification is partly machine-derived.** Field of Research codes
come from deposited subjects where those carry a signal, and from
confidence-gated machine classification otherwise. Every collection reports the
split. Machine assignments can be wrong.

**Curation readiness is model-assisted.** It is a proxy, not a human audit, and
it is labelled as such everywhere it appears.

**Checksum coverage is good but not universal.** 97.5% of resolved files
(264,844 of 271,735, measured 2026-08-20) carry a checksum published by the
source. The gap is concentrated: GBIF and PANGAEA publish none at all, and
Dryad sits at 28% while its backfill runs — it publishes sha-256 for every
file, which a column named `md5` used to discard. Where a source publishes
nothing, the manifest reports `checksum_algorithm: null` rather than implying
a verification that cannot happen.

**Every recorded algorithm currently reads `md5`.** The column was backfilled
from the existing md5 values, which is what the platform had. Real sha-256
values appear as each repository's files are re-resolved; until a given file
has been, its algorithm is md5 because that is what was actually stored.

**File resolution is point-in-time.** A URL that resolved at assessment time can
move or expire. Listings older than 90 days are re-resolved daily, oldest first,
but a URL can still go stale between checks — re-check before a large download.

## API

**Superseded collection versions are not retained.** Only the current version's
member list is held. A pinned request for an older version returns `409` rather
than a different set. Keep the manifest and lockfile you downloaded.

**Version history may be empty on a fresh deployment.** The endpoint says so
explicitly rather than implying only one version has existed.

**Rate limiting is per worker process.** The effective ceiling across a
multi-worker deployment is higher than the advertised number. The headers
describe the per-worker window honestly.

**`/api/v1/search` is substring matching, not semantic search.** It says so in
every response. Semantic search over the whole registry is at
`/api/search/semantic`.

## Integrations

**Hugging Face and PyTorch need a transformation.** A collection is many
datasets with different schemas, so `load_dataset` cannot take it directly. The
[dataframes guide](/developers/guides/dataframes) shows the honest path, and the
recipe tabs mark these two as needing a transformation rather than showing code
that would not run.

**SDKs are not yet published to PyPI or npm.** They are built and tested in CI
and installable from a checkout. A tag-driven release workflow is in place; the
remaining step is adding the registry credentials, which is deliberately a
human one.

## Site

**The registry and admin pages are unchanged by this work.** They predate the v1
surface and have not been migrated to it.

Found something not on this list? [Tell us](/developers/operations/support).

---

# Operations / Troubleshooting
Source: https://fairdata.ai/developers/operations/troubleshooting

# Troubleshooting

## "No collections found"

If you got a `200` with an empty array, your filters excluded everything. If you
got a `503` with code `snapshot_unavailable`, the service could not read the
snapshot — **that is not an empty result**, and you should retry rather than
report zero collections to your users.

```bash
curl -sS -w '\n%{http_code}\n' "https://fairdata.ai/api/v1/collections"
```

## 404 on a collection id

Ids changed shape in v1: `31` is now canonically `31-biological-sciences`. Both
resolve, so a 404 usually means a typo or a code that has no members in the
current snapshot. List them:

```bash
curl -sS "https://fairdata.ai/api/v1/collections?limit=100" | jq -r '.collections[].id'
```

## 404 on a DOI that definitely exists

`/api/v1/datasets/*` only covers datasets that clear the Machine-Ready gate. The
404 body includes `registry_record_url` — follow it to see the record and why it
does not qualify:

```bash
curl -sS "https://fairdata.ai/api/record/10.5281/zenodo.123456"
```

## 409 version_not_available

Your pinned version is no longer published. Compare checksums to see what moved:

```bash
jq -r .manifest_checksum fairdata.lock.json
curl -sS "https://fairdata.ai/api/v1/collections/<id>/manifest" | jq -r .manifest_checksum
```

## 422 invalid_cursor

You changed a filter or sort mid-listing, or the collection version moved.
Restart from the first page and follow `links.next`. Do not construct cursors.

## 429 rate_limited

240 requests per 60 seconds per IP per worker. Read `RateLimit-Remaining` and
slow down before you are refused. If you are running a bulk job, use the JSONL
stream — one request instead of hundreds:

```bash
curl -sS "https://fairdata.ai/api/v1/collections/<id>/datasets.jsonl"
```

## A DOI in a URL breaks my router

DOIs contain slashes. Use `GET /api/v1/datasets/resolve?doi=…`, or the base64url
identifier from `dataset.id`. Both are documented in
[find data](/developers/guides/find-data).

## An MCP client will not connect

```bash
curl -sS https://fairdata.ai/api/mcp -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","id":1,"method":"server/discover","params":{}}'
```

If that returns a result, the server is fine and the problem is client
configuration — check the URL has no trailing slash and the transport is HTTP,
not stdio. An unsupported protocol version comes back as an error listing what
is supported.

## Downloads fail or hang

File URLs point at source repositories, not at us, so their availability is
theirs. Set a generous timeout, retry with backoff, and re-fetch the file
listing — a URL that resolved at assessment time may have moved:

```bash
curl -sS "https://fairdata.ai/api/record/<doi>/files"
```

## Still stuck

[Report it](/developers/operations/support) with the `request_id` from the
response headers or the problem body.

---

# Operations / Support and reporting
Source: https://fairdata.ai/developers/operations/support

# Support and reporting

One address for everything: **hello@fairdata.ai**. Use a subject prefix so it
reaches the right place quickly.

## Report a data problem

**Subject:** `[data]` — a wrong licence, a wrong score, a dataset that should or
should not qualify, a broken file URL, a wrong discipline label.

Include:

- the DOI
- what we show (paste the relevant part of `/api/v1/datasets/resolve?doi=…`)
- what the source repository actually shows, ideally with a link
- the collection version, if it is about a collection

Licence errors are the highest priority thing on this list. If we have read a
licence wrongly, people may be relying on it.

## Report an API problem

**Subject:** `[api]`

Include:

- the full URL, including query string
- the `request_id` from the `X-Request-Id` header or the problem body
- the status code and the `code` field
- what you expected
- roughly when it happened, with a timezone

## Report a documentation problem

**Subject:** `[docs]` — or use the "Was this useful?" control at the foot of any
page, which sends the page slug and nothing else.

Include the page, what is wrong, and — if it is an example — the error you got.
Every quickstart runs in CI, so a broken example is a bug on our side and we
want to know.

## Report a security issue

**Do not open a public issue.** Email **security@fairdata.ai**, or see
[/.well-known/security.txt](https://fairdata.ai/.well-known/security.txt).

Include what you found, how to reproduce it, and what an attacker could do. We
will acknowledge within three working days. Please give us reasonable time
before disclosing.

## What to expect

This is a small project. Best effort, no guaranteed response time, and no
support contract. Data problems and security reports get looked at first.

## What we do not collect

Documentation feedback records the page and your rating. It does not record who
you are, what you typed elsewhere on the page, or anything from your clipboard.
Analytics record which developer steps were reached — never free-text queries,
DOIs you typed, code you pasted, or credentials. See `docs/measurement.md` in the
repository for the full list of events and the reasoning.