Initial Bible data repository

CONVENTIONS.md

@@ -0,0 +1,160 @@
+# Libre Bible Data Conventions
+
+Libre Bible Data is the canonical source, normalization, and packaging repository for free-to-use Bible and Bible-study resources used by Libre Study, GracePress Bible Tooltip, and related projects.
+
+## Repository Purpose
+
+This repo should gather every Bible translation and study resource that we can legally redistribute and normalize into app-ready packages.
+
+Priority order:
+
+1. Free-to-use Bible translations tied to Strong's numbers, morphology, lemmas, or concordance data.
+2. Free-to-use Bible translations with attached study notes.
+3. Free-to-use lexicons, dictionaries, cross-reference sets, and translation helps.
+4. Free-to-use commentaries that can be tied to specific verses, ranges, chapters, books, or biblical sections.
+5. Free-to-use maps, media, timelines, outlines, and other study aids.
+
+The long-term goal is a dynamic resource library that applications can search, install, update, and combine into a more complete Bible study experience.
+
+## Local and Remote Workflow
+
+- Local development happens in the workspace repo on drive `W:`.
+- The Christ Unscripted Gitea remote is the remote backup, collaboration, and publishing copy.
+- Do not treat resource work as durable until it is committed locally and pushed to Gitea.
+- The normal flow is local import/check, generated package verification, local commit, push to Gitea, then publish generated catalogs or releases from the pushed state.
+- The local repo and the Gitea repo are both intentional copies. Either should be enough to recover the project if the other system fails.
+- Do not publish generated resources or release artifacts from a dirty worktree unless Jason explicitly asks for that exact operation.
+- Keep source-control bookkeeping quiet unless Jason asks about it or a source-control problem affects the work.
+- Use the actual project scripts instead of ad hoc equivalents.
+- If a command starts failing because of PowerShell, Bash, WSL, quoting, heredocs, pipes, regexes, `$` variables, or nested shell layers, stop retrying the same command shape. Move the logic into a script file or use argv-style execution.
+- Prefer PowerShell for normal Windows-local Node/import work in this repo. Use WSL Bash only when a tool or script is genuinely Linux-oriented.
+
+## Publishing Model
+
+- Gitea should hold source manifests, importer scripts, generated packages, tags, release history, and issue/roadmap discussion.
+- Public Gitea host: `https://git.christit.com`.
+- Planned remote URL pattern:
+
+```text
+https://git.christit.com/libre-study/libre-bible-data.git
+git@git.christit.com:libre-study/libre-bible-data.git
+```
+
+- A polished public front end should be generated from committed metadata, especially `packages/json/catalog.json` and per-resource package catalogs.
+- Do not customize Gitea before proving the generated catalog/front-end approach. Gitea should remain the reliable Git and release system.
+- Public catalog pages should show resource title, abbreviation, language, license, redistribution status, upstream source, last checked date, package checksums, counts, features, and download links.
+- GracePress plugins, Libre Study, and any future apps should consume stable package URLs or release artifacts from the pushed repo/public catalog.
+- Do not hand-maintain public catalog details that can be derived from manifests and generated package catalogs.
+- Generated catalogs, package indexes, checksums, and format-specific outputs should be regenerated by scripts rather than hand-corrected after the fact.
+
+## Legal Boundary
+
+- "Free online" is not enough. Redistribution and format conversion must be allowed.
+- Every committed text/resource package must have explicit license metadata.
+- If redistribution is unclear, store source metadata and importer instructions only. Do not commit the resource content.
+- Preserve upstream attribution and license notes in generated catalogs.
+- Keep jurisdiction-specific restrictions visible, especially when a text is public domain in one country but restricted in another.
+
+## Source Manifests
+
+Every source resource needs a manifest in `sources/`.
+
+Each manifest should record:
+
+- Stable resource id.
+- Human title and abbreviation.
+- Language.
+- Upstream provider.
+- Upstream human URL.
+- Upstream download URL.
+- Upstream format.
+- License name and redistribution status.
+- Jurisdiction notes.
+- Expected source checksum.
+- Last checked timestamp.
+- Importer name and version.
+- Generated package paths.
+- Feature flags such as `strongs`, `notes`, `morphology`, `commentary`, `maps`, or `cross-references`.
+
+## Metadata Truth
+
+- Source manifests are the canonical metadata source for upstream provider, license, redistribution status, source checksum, importer, and generated package paths.
+- Generated package catalogs must be produced from source manifests and package outputs, not manually corrected afterward.
+- Build/import scripts should fail when required source, license, checksum, or package metadata is missing.
+- The public catalog must not silently disagree with the source manifest.
+
+## Update Workflow
+
+- Update checks must be mechanical and repeatable.
+- The normal flow is:
+
+```powershell
+npm.cmd run check
+npm.cmd run build
+```
+
+- A source is considered unchanged when the upstream artifact checksum matches the manifest.
+- If a checksum changes, inspect the upstream resource, license, and content before accepting the change.
+- Update the manifest and generated packages in the same commit when accepting an upstream change.
+- Do not silently overwrite generated packages without recording the source checksum that produced them.
+- Keep update checks short and decisive: check source checksum, rebuild intended packages, inspect the relevant generated catalog, and stop unless Jason asks for broader verification.
+
+## Package Outputs
+
+Generated packages should be app-friendly and stable.
+
+Initial package targets:
+
+- JSONL for simple streaming imports.
+- SQLite-ready schemas for Libre Study and desktop apps.
+- WordPress/plugin-ready packages for GracePress Bible Tooltip.
+
+Future package targets may include:
+
+- SWORD-compatible exports.
+- OSIS/USFM normalized exports.
+- Search indexes.
+- Static web catalogs.
+
+## Backup Copies
+
+- Source control is the main recovery layer, but local `.old` backups are still useful before large, risky, or release-bound importer/schema changes.
+- Put backups near the affected file or resource folder, not in random temp locations.
+- Prefer backup filenames with timestamp plus short reason, for example `import-usfm.js.20260712-parser-change.old`.
+- Do not create `.old` backups for every generated package refresh. Generated outputs should be reproducible from manifests and scripts.
+
+## Changelogs
+
+- Keep changelogs newest-first when changelogs are introduced.
+- Use a `## Current` section when helpful, followed by dated release sections.
+- Keep entries concrete and operational: source added, license verified, importer changed, package regenerated, checksum accepted, or behavior preserved.
+- Mark reconstructed history clearly if it is built from old packages, upstream archives, or prior notes.
+
+## Data Model Direction
+
+Resources should be able to attach to:
+
+- Whole translation.
+- Book.
+- Chapter.
+- Verse.
+- Verse range.
+- Word/token.
+- Strong's number.
+- Lemma.
+- Topic/tag.
+- Map location.
+- Timeline event.
+
+Do not flatten everything into verse text. Keep links, notes, lemmas, references, and resource relationships queryable.
+
+## Current First Resource
+
+The first resource is KJV from eBible.org's `eng-kjv2006` USFM package. It is treated as the first proving ground for:
+
+- Source manifest discipline.
+- Checksum-based update checks.
+- USFM import.
+- Verse normalization.
+- Strong's-link extraction.
+- Generated package catalogs.