libre-bible-data/CONVENTIONS.md

# Libre Bible Data Conventions

LibreBible is the public Bible and study-resource data project. This technical repo, `libre-bible-data`, is the canonical source, normalization, and packaging repository for free-to-use Bible and Bible-study resources used by Libre Study, GracePress Bible Tooltip, and related projects.

## Repository Purpose

This repo should gather every Bible translation and study resource that we can legally redistribute and normalize into app-ready packages.

Priority order:

1. Free-to-use Bible translations tied to Strong's numbers, morphology, lemmas, or concordance data.
2. Free-to-use Bible translations with attached study notes.
3. Free-to-use lexicons, dictionaries, cross-reference sets, and translation helps.
4. Free-to-use commentaries that can be tied to specific verses, ranges, chapters, books, or biblical sections.
5. Free-to-use maps, media, timelines, outlines, and other study aids.

The long-term goal is a dynamic resource library that applications can search, install, update, and combine into a more complete Bible study experience.

## Local and Remote Workflow

- Local development happens in the workspace repo on drive `W:`.
- The Christ Unscripted Gitea remote is the remote backup, collaboration, and publishing copy.
- Do not treat resource work as durable until it is committed locally and pushed to Gitea.
- Every named implementation phase should bump the repo version and be recorded in Git both locally and remotely.
- Phase completion means the version bump, code/data/docs changes, and validation result are committed locally and pushed to Gitea.
- The normal flow is local import/check, generated package verification, local commit, push to Gitea, then publish generated catalogs or releases from the pushed state.
- The local repo and the Gitea repo are both intentional copies. Either should be enough to recover the project if the other system fails.
- Do not publish generated resources or release artifacts from a dirty worktree unless Jason explicitly asks for that exact operation.
- Keep source-control bookkeeping quiet unless Jason asks about it or a source-control problem affects the work.
- Use the actual project scripts instead of ad hoc equivalents.
- If a command starts failing because of PowerShell, Bash, WSL, quoting, heredocs, pipes, regexes, `$` variables, or nested shell layers, stop retrying the same command shape. Move the logic into a script file or use argv-style execution.
- Prefer PowerShell for normal Windows-local Node/import work in this repo. Use WSL Bash only when a tool or script is genuinely Linux-oriented.

## Publishing Model

- Gitea should hold source manifests, importer scripts, generated packages, tags, release history, and issue/roadmap discussion.
- Public Gitea host: `https://git.christit.com`.
- Remote URL:

```text
https://git.christit.com/libre-study/libre-bible-data.git
git@git.christit.com:libre-study/libre-bible-data.git
```

- A polished public front end should be generated from committed metadata, especially `packages/json/catalog.json` and per-resource package catalogs.
- Do not customize Gitea before proving the generated catalog/front-end approach. Gitea should remain the reliable Git and release system.
- Public catalog pages should show resource title, abbreviation, language, license, redistribution status, upstream source, last checked date, package checksums, counts, features, and download links.
- GracePress plugins, Libre Study, and any future apps should consume stable package URLs or release artifacts from the pushed repo/public catalog.
- Do not hand-maintain public catalog details that can be derived from manifests and generated package catalogs.
- Generated catalogs, package indexes, checksums, and format-specific outputs should be regenerated by scripts rather than hand-corrected after the fact.

## Legal Boundary

- "Free online" is not enough. Redistribution and format conversion must be allowed.
- Every committed text/resource package must have explicit license metadata.
- If redistribution is unclear, store source metadata and importer instructions only. Do not commit the resource content.
- Preserve upstream attribution and license notes in generated catalogs.
- Keep jurisdiction-specific restrictions visible, especially when a text is public domain in one country but restricted in another.

## Source Manifests

Every source resource needs a manifest in `sources/`.

Each manifest should record:

- Stable resource id.
- Resource type, such as `translation`, `study_notes`, `commentary`, `map`, `lexicon`, `dictionary`, `cross_reference`, or `timeline`.
- Human title and abbreviation.
- Language code, language name, script, and dialect when relevant.
- Translation date, edition date, source text basis, tradition, and public description when the resource is a Bible translation.
- Contributors and maintainers with their roles.
- Upstream provider.
- Upstream human URL.
- Upstream download URL.
- Upstream format.
- License name and redistribution status.
- Jurisdiction notes.
- Expected source checksum.
- Last checked timestamp.
- Importer name and version.
- Generated package paths.
- Structured features and attachment metadata such as `strongs`, `notes`, `morphology`, `commentary`, `maps`, or `cross-references`.

Bible translation manifests should be broad enough to describe editions that include more than plain verse text. A translation may include or later connect to Hebrew, Greek, or Aramaic concordance entries, morphology, lemma links, study notes, commentaries, maps, timelines, media, and cross-reference resources. Do not encode KJV-only assumptions into the manifest shape.

Attachment metadata should identify:

- Attachment resource type.
- Whether the attachment is embedded in the source package or external.
- Relationship type, such as `word-to-strongs`, `verse-to-note`, `range-to-commentary`, `place-to-map`, or `event-to-timeline`.
- Anchor types, such as translation, book, chapter, verse, verse range, word/token, Strong's number, lemma, topic, place, or timeline event.
- Languages and systems involved, such as Hebrew, Greek, Aramaic, Strong's, morphology, or source-language lemmas.
- Generated package path when the attachment is packaged.

## Metadata Truth

- Source manifests are the canonical metadata source for upstream provider, license, redistribution status, source checksum, importer, and generated package paths.
- Generated package catalogs must be produced from source manifests and package outputs, not manually corrected afterward.
- Build/import scripts should fail when required source, license, checksum, or package metadata is missing.
- The public catalog must not silently disagree with the source manifest.

## Machine Catalogs

- `packages/json/catalog.json` is the machine-readable index for ChristIT.com, Libre Study, GracePress Bible Tooltip, and future consumers.
- Per-resource package catalogs, such as `packages/json/<resource-id>/catalog.json`, should carry the full resource metadata needed to render a public detail page without scraping Markdown.
- Machine catalogs should include schema version, project name, resource id, resource type, title, abbreviation, language, script, canon, translation/edition metadata, contributors, features, attachments, source, license, display summary, package paths, counts, file checksums, and source check metadata.
- Markdown docs are a human-readable projection of the same metadata. They must not become a separate truth source.
- Public ChristIT.com pages and app/plugin package discovery should read machine catalogs first, then link to Markdown docs for explanation.

## Update Workflow

- Update checks must be mechanical and repeatable.
- The normal flow is:

```powershell
npm.cmd run check
npm.cmd run build
```

- A source is considered unchanged when the upstream artifact checksum matches the manifest.
- If a checksum changes, inspect the upstream resource, license, and content before accepting the change.
- Update the manifest and generated packages in the same commit when accepting an upstream change.
- Do not silently overwrite generated packages without recording the source checksum that produced them.
- Keep update checks short and decisive: check source checksum, rebuild intended packages, inspect the relevant generated catalog, and stop unless Jason asks for broader verification.

## Package Outputs

Generated packages should be app-friendly and stable.

Initial package targets:

- JSONL for simple streaming imports.
- SQLite-ready schemas for Libre Study and desktop apps.
- WordPress/plugin-ready packages for GracePress Bible Tooltip.

Future package targets may include:

- SWORD-compatible exports.
- OSIS/USFM normalized exports.
- Search indexes.
- Static web catalogs.

## Backup Copies

- Source control is the main recovery layer, but local `.old` backups are still useful before large, risky, or release-bound importer/schema changes.
- Put backups near the affected file or resource folder, not in random temp locations.
- Prefer backup filenames with timestamp plus short reason, for example `import-usfm.js.20260712-parser-change.old`.
- Do not create `.old` backups for every generated package refresh. Generated outputs should be reproducible from manifests and scripts.

## Changelogs

- Keep changelogs newest-first when changelogs are introduced.
- Use a `## Current` section when helpful, followed by dated release sections.
- Keep entries concrete and operational: source added, license verified, importer changed, package regenerated, checksum accepted, or behavior preserved.
- Mark reconstructed history clearly if it is built from old packages, upstream archives, or prior notes.

## Data Model Direction

Resources should be able to attach to:

- Whole translation.
- Book.
- Chapter.
- Verse.
- Verse range.
- Word/token.
- Strong's number.
- Lemma.
- Topic/tag.
- Map location.
- Timeline event.

Do not flatten everything into verse text. Keep links, notes, lemmas, references, and resource relationships queryable.

## Current First Resource

The first resource is KJV from eBible.org's `eng-kjv2006` USFM package. It is treated as the first proving ground for:

- Source manifest discipline.
- Checksum-based update checks.
- USFM import.
- Verse normalization.
- Strong's-link extraction.
- Generated package catalogs.