How AnimalWiki is built

The rest of this site is about animals. This page is about the machinery underneath it, for anyone who wants to poke at it, reuse it, or contribute to it.

The encyclopedia is a Git repository

There is no database and no CMS. Every taxon is a Markdown file, and the directory tree is the classification, so a page's path is its ancestry:

wiki/eukaryota/animalia/chordata/mammalia/carnivora/felidae/panthera/panthera-leo.md
     └ domain  └ kingdom └ phylum  └ class  └ order   └ family └ genus  └ species

That is 360,003 Markdown files. Every container rank has an index.md; every species is a leaf file beside its siblings. Browse the repository on GitHub and the links between them work; browse this website and the same files are rendered with search, a taxonomy explorer and coverage statistics.

Generated and human text share a file

Each page has a machine-written region between two markers, rebuilt from scratch whenever the taxonomic backbone is refreshed, and a region below it that belongs to whoever wrote it and is never touched by any tool. That separation is the whole trick: it is what lets a corpus this size be regenerated at will and still accept pull requests without ever clobbering someone's writing.

The backbone

A taxonomic backbone derived from the iNaturalist taxonomy export is committed to the repository as two compressed TSV files — one for the 109,745 taxa from domain down to genus, one for the 756,338 species. tools/generate_wiki.py turns the backbone into the Markdown tree; tools/build_site.py renders that tree into the static site you are reading. Both are standard-library Python. Rebuilding the backbone from the upstream export is the only step that needs a third-party package.

What the backbone does not contain

The iNaturalist taxonomy is built around organisms people have observed and identified, which makes it excellent where naturalists look and thin where they do not. Science has described roughly 1.5 million animal species; the export carries 766,806 at species rank, and 756,338 of those appear here. The other 10,468 are dropped for one of two reasons: nearly all of them sit under a superfamily or a suborder with no family, order or class node above them, and a page whose breadcrumb has a hole in it is worse than no page; a couple of hundred more are hybrids or duplicate binomials left over from synonym merges. That is about 1.4% of the source, so the filter is not where the missing animals went.

The shortfall against the literature is not evenly spread. Every mammal, bird, reptile, amphibian and fish is here — the vertebrate counts match the published totals, sometimes exceed them where this taxonomy splits what others lump. What is missing is invertebrate: around half the described insects, half the molluscs, and the great majority of nematodes and flatworms, which are described from specimens and museum collections rather than from photographs.

Closing that gap means a different backbone — Catalogue of Life or the GBIF backbone, both of which aggregate the taxonomic literature rather than observations. Neither carries the iNaturalist taxon ids that the photographs and common names on this site are keyed to, so it would be a merge rather than a swap.

Coverage tiers

Of the species that are here, articles are written in three tiers, because writing 756,338 of them is not a thing that happens:

Any index-tier species can be promoted to an article by writing one. COVERAGE.md is the normative statement of the policy.

Why not every page is a file

GitHub Pages refuses to publish a site over 1 GB, which is a smaller number of pages than this corpus contains. Species pages are therefore prerendered in priority order — anything a contributor has written first, then anything with curated data, then vertebrates, then the rest — until the budget runs out. Everything beyond it is still a real URL: GitHub serves 404.html, and the client rebuilds the page from the search index and the path. A page that acquires contributor prose moves to the front of the prerender queue automatically.

Search

The search index covers every taxon in the backbone, not just the prerendered ones. It is sharded by name prefix — two characters, split to three wherever a bucket gets dense — so typing into the box downloads a few tens of kilobytes rather than a multi-megabyte index. Scientific names and common names are indexed separately, which is why "lion" finds Panthera leo and not only genera that happen to begin with those letters.

Licensing

• Taxonomy: the iNaturalist taxonomy export, CC0-1.0, via the AWS Open Data programme.
• Photographs: iNaturalist Open Data, hotlinked rather than copied, each carrying its photographer and licence.
• Conservation categories: contributor-entered, each citing an IUCN Red List assessment.
• Article text: CC BY-SA 4.0. Tooling: MIT.

Contributing

Corrections and new writing are both welcome, and the Edit this page link in the margin of any article is the shortest path to either. For anything larger, the contributing guide covers the file format and the review process, and ARCHITECTURE.md explains why the project is shaped the way it is.