177 lines
9.2 KiB
Markdown
177 lines
9.2 KiB
Markdown
# Libre Bible Data Conventions
|
|
|
|
LibreBible is the public Bible and study-resource data project. This technical repo, `libre-bible-data`, is the canonical source, normalization, and packaging repository for free-to-use Bible and Bible-study resources used by Libre Study, GracePress Bible Tooltip, and related projects.
|
|
|
|
## Repository Purpose
|
|
|
|
This repo should gather every Bible translation and study resource that we can legally redistribute and normalize into app-ready packages.
|
|
|
|
Priority order:
|
|
|
|
1. Free-to-use Bible translations tied to Strong's numbers, morphology, lemmas, or concordance data.
|
|
2. Free-to-use Bible translations with attached study notes.
|
|
3. Free-to-use lexicons, dictionaries, cross-reference sets, and translation helps.
|
|
4. Free-to-use commentaries that can be tied to specific verses, ranges, chapters, books, or biblical sections.
|
|
5. Free-to-use maps, media, timelines, outlines, and other study aids.
|
|
|
|
The long-term goal is a dynamic resource library that applications can search, install, update, and combine into a more complete Bible study experience.
|
|
|
|
## Local and Remote Workflow
|
|
|
|
- Local development happens in the workspace repo on drive `W:`.
|
|
- The Christ Unscripted Gitea remote is the remote backup, collaboration, and publishing copy.
|
|
- Do not treat resource work as durable until it is committed locally and pushed to Gitea.
|
|
- Every named implementation phase should bump the repo version and be recorded in Git both locally and remotely.
|
|
- Phase completion means the version bump, code/data/docs changes, and validation result are committed locally and pushed to Gitea.
|
|
- The normal flow is local import/check, generated package verification, local commit, push to Gitea, then publish generated catalogs or releases from the pushed state.
|
|
- The local repo and the Gitea repo are both intentional copies. Either should be enough to recover the project if the other system fails.
|
|
- Do not publish generated resources or release artifacts from a dirty worktree unless Jason explicitly asks for that exact operation.
|
|
- Keep source-control bookkeeping quiet unless Jason asks about it or a source-control problem affects the work.
|
|
- Use the actual project scripts instead of ad hoc equivalents.
|
|
- If a command starts failing because of PowerShell, Bash, WSL, quoting, heredocs, pipes, regexes, `$` variables, or nested shell layers, stop retrying the same command shape. Move the logic into a script file or use argv-style execution.
|
|
- Prefer PowerShell for normal Windows-local Node/import work in this repo. Use WSL Bash only when a tool or script is genuinely Linux-oriented.
|
|
|
|
## Publishing Model
|
|
|
|
- Gitea should hold source manifests, importer scripts, generated packages, tags, release history, and issue/roadmap discussion.
|
|
- Public Gitea host: `https://git.christit.com`.
|
|
- Remote URL:
|
|
|
|
```text
|
|
https://git.christit.com/libre-study/libre-bible-data.git
|
|
git@git.christit.com:libre-study/libre-bible-data.git
|
|
```
|
|
|
|
- A polished public front end should be generated from committed metadata, especially `packages/json/catalog.json` and per-resource package catalogs.
|
|
- Do not customize Gitea before proving the generated catalog/front-end approach. Gitea should remain the reliable Git and release system.
|
|
- Public catalog pages should show resource title, abbreviation, language, license, redistribution status, upstream source, last checked date, package checksums, counts, features, and download links.
|
|
- GracePress plugins, Libre Study, and any future apps should consume stable package URLs or release artifacts from the pushed repo/public catalog.
|
|
- Do not hand-maintain public catalog details that can be derived from manifests and generated package catalogs.
|
|
- Generated catalogs, package indexes, checksums, and format-specific outputs should be regenerated by scripts rather than hand-corrected after the fact.
|
|
|
|
## Legal Boundary
|
|
|
|
- "Free online" is not enough. Redistribution and format conversion must be allowed.
|
|
- Every committed text/resource package must have explicit license metadata.
|
|
- If redistribution is unclear, store source metadata and importer instructions only. Do not commit the resource content.
|
|
- Preserve upstream attribution and license notes in generated catalogs.
|
|
- Keep jurisdiction-specific restrictions visible, especially when a text is public domain in one country but restricted in another.
|
|
|
|
## Source Manifests
|
|
|
|
Every source resource needs a manifest in `sources/`.
|
|
|
|
Each manifest should record:
|
|
|
|
- Stable resource id.
|
|
- Resource type, such as `translation`, `study_notes`, `commentary`, `map`, `lexicon`, `dictionary`, `cross_reference`, or `timeline`.
|
|
- Human title and abbreviation.
|
|
- Language code, language name, script, and dialect when relevant.
|
|
- Translation date, edition date, source text basis, tradition, and public description when the resource is a Bible translation.
|
|
- Contributors and maintainers with their roles.
|
|
- Upstream provider.
|
|
- Upstream human URL.
|
|
- Upstream download URL.
|
|
- Upstream format.
|
|
- License name and redistribution status.
|
|
- Jurisdiction notes.
|
|
- Expected source checksum.
|
|
- Last checked timestamp.
|
|
- Importer name and version.
|
|
- Generated package paths.
|
|
- Structured features and attachment metadata such as `strongs`, `notes`, `morphology`, `commentary`, `maps`, or `cross-references`.
|
|
|
|
Bible translation manifests should be broad enough to describe editions that include more than plain verse text. A translation may include or later connect to Hebrew, Greek, or Aramaic concordance entries, morphology, lemma links, study notes, commentaries, maps, timelines, media, and cross-reference resources. Do not encode KJV-only assumptions into the manifest shape.
|
|
|
|
Attachment metadata should identify:
|
|
|
|
- Attachment resource type.
|
|
- Whether the attachment is embedded in the source package or external.
|
|
- Relationship type, such as `word-to-strongs`, `verse-to-note`, `range-to-commentary`, `place-to-map`, or `event-to-timeline`.
|
|
- Anchor types, such as translation, book, chapter, verse, verse range, word/token, Strong's number, lemma, topic, place, or timeline event.
|
|
- Languages and systems involved, such as Hebrew, Greek, Aramaic, Strong's, morphology, or source-language lemmas.
|
|
- Generated package path when the attachment is packaged.
|
|
|
|
## Metadata Truth
|
|
|
|
- Source manifests are the canonical metadata source for upstream provider, license, redistribution status, source checksum, importer, and generated package paths.
|
|
- Generated package catalogs must be produced from source manifests and package outputs, not manually corrected afterward.
|
|
- Build/import scripts should fail when required source, license, checksum, or package metadata is missing.
|
|
- The public catalog must not silently disagree with the source manifest.
|
|
|
|
## Update Workflow
|
|
|
|
- Update checks must be mechanical and repeatable.
|
|
- The normal flow is:
|
|
|
|
```powershell
|
|
npm.cmd run check
|
|
npm.cmd run build
|
|
```
|
|
|
|
- A source is considered unchanged when the upstream artifact checksum matches the manifest.
|
|
- If a checksum changes, inspect the upstream resource, license, and content before accepting the change.
|
|
- Update the manifest and generated packages in the same commit when accepting an upstream change.
|
|
- Do not silently overwrite generated packages without recording the source checksum that produced them.
|
|
- Keep update checks short and decisive: check source checksum, rebuild intended packages, inspect the relevant generated catalog, and stop unless Jason asks for broader verification.
|
|
|
|
## Package Outputs
|
|
|
|
Generated packages should be app-friendly and stable.
|
|
|
|
Initial package targets:
|
|
|
|
- JSONL for simple streaming imports.
|
|
- SQLite-ready schemas for Libre Study and desktop apps.
|
|
- WordPress/plugin-ready packages for GracePress Bible Tooltip.
|
|
|
|
Future package targets may include:
|
|
|
|
- SWORD-compatible exports.
|
|
- OSIS/USFM normalized exports.
|
|
- Search indexes.
|
|
- Static web catalogs.
|
|
|
|
## Backup Copies
|
|
|
|
- Source control is the main recovery layer, but local `.old` backups are still useful before large, risky, or release-bound importer/schema changes.
|
|
- Put backups near the affected file or resource folder, not in random temp locations.
|
|
- Prefer backup filenames with timestamp plus short reason, for example `import-usfm.js.20260712-parser-change.old`.
|
|
- Do not create `.old` backups for every generated package refresh. Generated outputs should be reproducible from manifests and scripts.
|
|
|
|
## Changelogs
|
|
|
|
- Keep changelogs newest-first when changelogs are introduced.
|
|
- Use a `## Current` section when helpful, followed by dated release sections.
|
|
- Keep entries concrete and operational: source added, license verified, importer changed, package regenerated, checksum accepted, or behavior preserved.
|
|
- Mark reconstructed history clearly if it is built from old packages, upstream archives, or prior notes.
|
|
|
|
## Data Model Direction
|
|
|
|
Resources should be able to attach to:
|
|
|
|
- Whole translation.
|
|
- Book.
|
|
- Chapter.
|
|
- Verse.
|
|
- Verse range.
|
|
- Word/token.
|
|
- Strong's number.
|
|
- Lemma.
|
|
- Topic/tag.
|
|
- Map location.
|
|
- Timeline event.
|
|
|
|
Do not flatten everything into verse text. Keep links, notes, lemmas, references, and resource relationships queryable.
|
|
|
|
## Current First Resource
|
|
|
|
The first resource is KJV from eBible.org's `eng-kjv2006` USFM package. It is treated as the first proving ground for:
|
|
|
|
- Source manifest discipline.
|
|
- Checksum-based update checks.
|
|
- USFM import.
|
|
- Verse normalization.
|
|
- Strong's-link extraction.
|
|
- Generated package catalogs.
|