libre-bible-data/CONVENTIONS.md

Libre Bible Data Conventions

LibreBible is the public Bible and study-resource data project. This technical repo, libre-bible-data, is the canonical source, normalization, and packaging repository for free-to-use Bible and Bible-study resources used by Libre Study, GracePress Bible Tooltip, and related projects.

Repository Purpose

This repo should gather every Bible translation and study resource that we can legally redistribute and normalize into app-ready packages.

Priority order:

1. Free-to-use Bible translations tied to Strong's numbers, morphology, lemmas, or concordance data.
2. Free-to-use Bible translations with attached study notes.
3. Free-to-use lexicons, dictionaries, cross-reference sets, and translation helps.
4. Free-to-use commentaries that can be tied to specific verses, ranges, chapters, books, or biblical sections.
5. Free-to-use maps, media, timelines, outlines, and other study aids.

The long-term goal is a dynamic resource library that applications can search, install, update, and combine into a more complete Bible study experience.

Local and Remote Workflow

• Local development happens in the workspace repo on drive W:.
• The Christ Unscripted Gitea remote is the remote backup, collaboration, and publishing copy.
• Do not treat resource work as durable until it is committed locally and pushed to Gitea.
• Every named implementation phase should bump the repo version and be recorded in Git both locally and remotely.
• Phase completion means the version bump, code/data/docs changes, and validation result are committed locally and pushed to Gitea.
• The normal flow is local import/check, generated package verification, local commit, push to Gitea, then publish generated catalogs or releases from the pushed state.
• The local repo and the Gitea repo are both intentional copies. Either should be enough to recover the project if the other system fails.
• Do not publish generated resources or release artifacts from a dirty worktree unless Jason explicitly asks for that exact operation.
• Keep source-control bookkeeping quiet unless Jason asks about it or a source-control problem affects the work.
• Use the actual project scripts instead of ad hoc equivalents.
• If a command starts failing because of PowerShell, Bash, WSL, quoting, heredocs, pipes, regexes, $ variables, or nested shell layers, stop retrying the same command shape. Move the logic into a script file or use argv-style execution.
• Prefer PowerShell for normal Windows-local Node/import work in this repo. Use WSL Bash only when a tool or script is genuinely Linux-oriented.

Publishing Model

• Gitea should hold source manifests, importer scripts, generated packages, tags, release history, and issue/roadmap discussion.
• Public Gitea host: https://git.christit.com.
• Remote URL:

https://git.christit.com/libre-study/libre-bible-data.git
git@git.christit.com:libre-study/libre-bible-data.git

• A polished public front end should be generated from committed metadata, especially packages/json/catalog.json and per-resource package catalogs.
• Do not customize Gitea before proving the generated catalog/front-end approach. Gitea should remain the reliable Git and release system.
• Public catalog pages should show resource title, abbreviation, language, license, redistribution status, upstream source, last checked date, package checksums, counts, features, and download links.
• GracePress plugins, Libre Study, and any future apps should consume stable package URLs or release artifacts from the pushed repo/public catalog.
• Do not hand-maintain public catalog details that can be derived from manifests and generated package catalogs.
• Generated catalogs, package indexes, checksums, and format-specific outputs should be regenerated by scripts rather than hand-corrected after the fact.

Legal Boundary

• "Free online" is not enough. Redistribution and format conversion must be allowed.
• Every committed text/resource package must have explicit license metadata.
• If redistribution is unclear, store source metadata and importer instructions only. Do not commit the resource content.
• Preserve upstream attribution and license notes in generated catalogs.
• Keep jurisdiction-specific restrictions visible, especially when a text is public domain in one country but restricted in another.

Source Manifests

Every source resource needs a manifest in sources/.

Each manifest should record:

• Stable resource id.
• Resource type, such as translation, study_notes, commentary, map, lexicon, dictionary, cross_reference, or timeline.
• Human title and abbreviation.
• Language code, language name, script, and dialect when relevant.
• Translation date, edition date, source text basis, tradition, and public description when the resource is a Bible translation.
• Contributors and maintainers with their roles.
• Upstream provider.
• Upstream human URL.
• Upstream download URL.
• Upstream format.
• License name and redistribution status.
• Jurisdiction notes.
• Expected source checksum.
• Last checked timestamp.
• Importer name and version.
• Generated package paths.
• Structured features and attachment metadata such as strongs, notes, morphology, commentary, maps, or cross-references.

Bible translation manifests should be broad enough to describe editions that include more than plain verse text. A translation may include or later connect to Hebrew, Greek, or Aramaic concordance entries, morphology, lemma links, study notes, commentaries, maps, timelines, media, and cross-reference resources. Do not encode KJV-only assumptions into the manifest shape.

Attachment metadata should identify:

• Attachment resource type.
• Whether the attachment is embedded in the source package or external.
• Relationship type, such as word-to-strongs, verse-to-note, range-to-commentary, place-to-map, or event-to-timeline.
• Anchor types, such as translation, book, chapter, verse, verse range, word/token, Strong's number, lemma, topic, place, or timeline event.
• Languages and systems involved, such as Hebrew, Greek, Aramaic, Strong's, morphology, or source-language lemmas.
• Generated package path when the attachment is packaged.

Metadata Truth

• Source manifests are the canonical metadata source for upstream provider, license, redistribution status, source checksum, importer, and generated package paths.
• Generated package catalogs must be produced from source manifests and package outputs, not manually corrected afterward.
• Build/import scripts should fail when required source, license, checksum, or package metadata is missing.
• The public catalog must not silently disagree with the source manifest.

Update Workflow

• Update checks must be mechanical and repeatable.
• The normal flow is:

npm.cmd run check
npm.cmd run build

• A source is considered unchanged when the upstream artifact checksum matches the manifest.
• If a checksum changes, inspect the upstream resource, license, and content before accepting the change.
• Update the manifest and generated packages in the same commit when accepting an upstream change.
• Do not silently overwrite generated packages without recording the source checksum that produced them.
• Keep update checks short and decisive: check source checksum, rebuild intended packages, inspect the relevant generated catalog, and stop unless Jason asks for broader verification.

Package Outputs

Generated packages should be app-friendly and stable.

Initial package targets:

• JSONL for simple streaming imports.
• SQLite-ready schemas for Libre Study and desktop apps.
• WordPress/plugin-ready packages for GracePress Bible Tooltip.

Future package targets may include:

• SWORD-compatible exports.
• OSIS/USFM normalized exports.
• Search indexes.
• Static web catalogs.

Backup Copies

• Source control is the main recovery layer, but local .old backups are still useful before large, risky, or release-bound importer/schema changes.
• Put backups near the affected file or resource folder, not in random temp locations.
• Prefer backup filenames with timestamp plus short reason, for example import-usfm.js.20260712-parser-change.old.
• Do not create .old backups for every generated package refresh. Generated outputs should be reproducible from manifests and scripts.

Changelogs

• Keep changelogs newest-first when changelogs are introduced.
• Use a ## Current section when helpful, followed by dated release sections.
• Keep entries concrete and operational: source added, license verified, importer changed, package regenerated, checksum accepted, or behavior preserved.
• Mark reconstructed history clearly if it is built from old packages, upstream archives, or prior notes.

Data Model Direction

Resources should be able to attach to:

• Whole translation.
• Book.
• Chapter.
• Verse.
• Verse range.
• Word/token.
• Strong's number.
• Lemma.
• Topic/tag.
• Map location.
• Timeline event.

Do not flatten everything into verse text. Keep links, notes, lemmas, references, and resource relationships queryable.

Current First Resource

The first resource is KJV from eBible.org's eng-kjv2006 USFM package. It is treated as the first proving ground for:

• Source manifest discipline.
• Checksum-based update checks.
• USFM import.
• Verse normalization.
• Strong's-link extraction.
• Generated package catalogs.