Initial Bible data repository

2026-07-11 21:36:31 -05:00
commit a7cbe14eae
14 changed files with 380611 additions and 0 deletions
@@ -0,0 +1,3 @@
+node_modules/
+cache/
+*.log
@@ -0,0 +1,160 @@
+# Libre Bible Data Conventions
+
+Libre Bible Data is the canonical source, normalization, and packaging repository for free-to-use Bible and Bible-study resources used by Libre Study, GracePress Bible Tooltip, and related projects.
+
+## Repository Purpose
+
+This repo should gather every Bible translation and study resource that we can legally redistribute and normalize into app-ready packages.
+
+Priority order:
+
+1. Free-to-use Bible translations tied to Strong's numbers, morphology, lemmas, or concordance data.
+2. Free-to-use Bible translations with attached study notes.
+3. Free-to-use lexicons, dictionaries, cross-reference sets, and translation helps.
+4. Free-to-use commentaries that can be tied to specific verses, ranges, chapters, books, or biblical sections.
+5. Free-to-use maps, media, timelines, outlines, and other study aids.
+
+The long-term goal is a dynamic resource library that applications can search, install, update, and combine into a more complete Bible study experience.
+
+## Local and Remote Workflow
+
+- Local development happens in the workspace repo on drive `W:`.
+- The Christ Unscripted Gitea remote is the remote backup, collaboration, and publishing copy.
+- Do not treat resource work as durable until it is committed locally and pushed to Gitea.
+- The normal flow is local import/check, generated package verification, local commit, push to Gitea, then publish generated catalogs or releases from the pushed state.
+- The local repo and the Gitea repo are both intentional copies. Either should be enough to recover the project if the other system fails.
+- Do not publish generated resources or release artifacts from a dirty worktree unless Jason explicitly asks for that exact operation.
+- Keep source-control bookkeeping quiet unless Jason asks about it or a source-control problem affects the work.
+- Use the actual project scripts instead of ad hoc equivalents.
+- If a command starts failing because of PowerShell, Bash, WSL, quoting, heredocs, pipes, regexes, `$` variables, or nested shell layers, stop retrying the same command shape. Move the logic into a script file or use argv-style execution.
+- Prefer PowerShell for normal Windows-local Node/import work in this repo. Use WSL Bash only when a tool or script is genuinely Linux-oriented.
+
+## Publishing Model
+
+- Gitea should hold source manifests, importer scripts, generated packages, tags, release history, and issue/roadmap discussion.
+- Public Gitea host: `https://git.christit.com`.
+- Planned remote URL pattern:
+
+```text
+https://git.christit.com/libre-study/libre-bible-data.git
+git@git.christit.com:libre-study/libre-bible-data.git
+```
+
+- A polished public front end should be generated from committed metadata, especially `packages/json/catalog.json` and per-resource package catalogs.
+- Do not customize Gitea before proving the generated catalog/front-end approach. Gitea should remain the reliable Git and release system.
+- Public catalog pages should show resource title, abbreviation, language, license, redistribution status, upstream source, last checked date, package checksums, counts, features, and download links.
+- GracePress plugins, Libre Study, and any future apps should consume stable package URLs or release artifacts from the pushed repo/public catalog.
+- Do not hand-maintain public catalog details that can be derived from manifests and generated package catalogs.
+- Generated catalogs, package indexes, checksums, and format-specific outputs should be regenerated by scripts rather than hand-corrected after the fact.
+
+## Legal Boundary
+
+- "Free online" is not enough. Redistribution and format conversion must be allowed.
+- Every committed text/resource package must have explicit license metadata.
+- If redistribution is unclear, store source metadata and importer instructions only. Do not commit the resource content.
+- Preserve upstream attribution and license notes in generated catalogs.
+- Keep jurisdiction-specific restrictions visible, especially when a text is public domain in one country but restricted in another.
+
+## Source Manifests
+
+Every source resource needs a manifest in `sources/`.
+
+Each manifest should record:
+
+- Stable resource id.
+- Human title and abbreviation.
+- Language.
+- Upstream provider.
+- Upstream human URL.
+- Upstream download URL.
+- Upstream format.
+- License name and redistribution status.
+- Jurisdiction notes.
+- Expected source checksum.
+- Last checked timestamp.
+- Importer name and version.
+- Generated package paths.
+- Feature flags such as `strongs`, `notes`, `morphology`, `commentary`, `maps`, or `cross-references`.
+
+## Metadata Truth
+
+- Source manifests are the canonical metadata source for upstream provider, license, redistribution status, source checksum, importer, and generated package paths.
+- Generated package catalogs must be produced from source manifests and package outputs, not manually corrected afterward.
+- Build/import scripts should fail when required source, license, checksum, or package metadata is missing.
+- The public catalog must not silently disagree with the source manifest.
+
+## Update Workflow
+
+- Update checks must be mechanical and repeatable.
+- The normal flow is:
+
+```powershell
+npm.cmd run check
+npm.cmd run build
+```
+
+- A source is considered unchanged when the upstream artifact checksum matches the manifest.
+- If a checksum changes, inspect the upstream resource, license, and content before accepting the change.
+- Update the manifest and generated packages in the same commit when accepting an upstream change.
+- Do not silently overwrite generated packages without recording the source checksum that produced them.
+- Keep update checks short and decisive: check source checksum, rebuild intended packages, inspect the relevant generated catalog, and stop unless Jason asks for broader verification.
+
+## Package Outputs
+
+Generated packages should be app-friendly and stable.
+
+Initial package targets:
+
+- JSONL for simple streaming imports.
+- SQLite-ready schemas for Libre Study and desktop apps.
+- WordPress/plugin-ready packages for GracePress Bible Tooltip.
+
+Future package targets may include:
+
+- SWORD-compatible exports.
+- OSIS/USFM normalized exports.
+- Search indexes.
+- Static web catalogs.
+
+## Backup Copies
+
+- Source control is the main recovery layer, but local `.old` backups are still useful before large, risky, or release-bound importer/schema changes.
+- Put backups near the affected file or resource folder, not in random temp locations.
+- Prefer backup filenames with timestamp plus short reason, for example `import-usfm.js.20260712-parser-change.old`.
+- Do not create `.old` backups for every generated package refresh. Generated outputs should be reproducible from manifests and scripts.
+
+## Changelogs
+
+- Keep changelogs newest-first when changelogs are introduced.
+- Use a `## Current` section when helpful, followed by dated release sections.
+- Keep entries concrete and operational: source added, license verified, importer changed, package regenerated, checksum accepted, or behavior preserved.
+- Mark reconstructed history clearly if it is built from old packages, upstream archives, or prior notes.
+
+## Data Model Direction
+
+Resources should be able to attach to:
+
+- Whole translation.
+- Book.
+- Chapter.
+- Verse.
+- Verse range.
+- Word/token.
+- Strong's number.
+- Lemma.
+- Topic/tag.
+- Map location.
+- Timeline event.
+
+Do not flatten everything into verse text. Keep links, notes, lemmas, references, and resource relationships queryable.
+
+## Current First Resource
+
+The first resource is KJV from eBible.org's `eng-kjv2006` USFM package. It is treated as the first proving ground for:
+
+- Source manifest discipline.
+- Checksum-based update checks.
+- USFM import.
+- Verse normalization.
+- Strong's-link extraction.
+- Generated package catalogs.
@@ -0,0 +1,31 @@
+# Libre Bible Data
+
+Libre Bible Data is the canonical source and packaging repo for Bible resources used by Libre Study, GracePress Bible Tooltip, and related tools.
+
+This repo does not treat third-party Bible texts as ours. Each resource starts with a source manifest that records upstream location, license status, source format, checksum, importer, generated package paths, and update-check metadata.
+
+## First Resource
+
+The first resource is `kjv-eng-kjv2006`, sourced from eBible.org's `eng-kjv2006` USFM package.
+
+The eBible listing identifies it as public domain, describes it as the 1769 King James Version with Strong's numbers added, and notes the United Kingdom Crown/Cambridge printing restriction does not affect public-domain status outside the UK. See `sources/kjv-eng-kjv2006.json`.
+
+## Workflow
+
+```powershell
+npm install
+npm run check
+npm run import:kjv
+```
+
+Generated packages land under `packages/json/<resource-id>/`.
+
+## Repo Rules
+
+- Every hosted text needs a source manifest.
+- Every source manifest needs license metadata.
+- If redistribution is not clearly allowed, do not commit generated text.
+- Generated packages must include checksums and importer metadata.
+- Update checks should be mechanical: fetch upstream, hash it, compare with the recorded checksum, and report drift.
+
+See `CONVENTIONS.md` for the durable project rules and resource priorities.
@@ -0,0 +1,30 @@
+# Resource Manifest
+
+Each `sources/*.json` file describes one upstream resource.
+
+Required fields:
+
+- `id`: Stable package id used by apps.
+- `title`: Human-readable title.
+- `abbreviation`: Short label.
+- `language`: BCP-47-ish language code.
+- `source.provider`: Upstream publisher or repository.
+- `source.url`: Human-facing upstream page.
+- `source.download_url`: Machine download URL used by importers.
+- `source.format`: Upstream source format, such as `usfm-zip`.
+- `license.name`: License or rights label.
+- `license.redistribution`: Whether this repo may redistribute normalized outputs.
+- `license.notes`: Practical rights notes.
+- `checks.expected_sha256`: Last accepted source archive checksum.
+- `checks.last_checked_at`: Last automated check timestamp.
+- `importer.name`: Script/importer id.
+- `importer.version`: Importer version.
+- `packages`: Generated package paths.
+
+Update flow:
+
+1. Download the upstream artifact to `cache/<resource-id>/`.
+2. Calculate SHA-256.
+3. Compare against `checks.expected_sha256`.
+4. If unchanged, update `checks.last_checked_at` only when intentionally accepting that metadata churn.
+5. If changed, inspect upstream release notes/license, regenerate packages, verify counts/checksums, then update the manifest in the same commit.
@@ -0,0 +1,24 @@
+{
+  "name": "libre-bible-data",
+  "version": "0.1.0",
+  "lockfileVersion": 3,
+  "requires": true,
+  "packages": {
+    "": {
+      "name": "libre-bible-data",
+      "version": "0.1.0",
+      "dependencies": {
+        "adm-zip": "^0.5.16"
+      }
+    },
+    "node_modules/adm-zip": {
+      "version": "0.5.18",
+      "resolved": "https://registry.npmjs.org/adm-zip/-/adm-zip-0.5.18.tgz",
+      "integrity": "sha512-ufJnssQGbxzLNS1Ho9bCtX4rQKCCvoVuDLHoJyc3F9dOGDB4BkWs2Ci0kv53lqocAEQ/Cbi+I2XCsNYGqVYqng==",
+      "license": "MIT",
+      "engines": {
+        "node": ">=12.0"
+      }
+    }
+  }
+}
@@ -0,0 +1,14 @@
+{
+  "name": "libre-bible-data",
+  "version": "0.1.0",
+  "private": true,
+  "type": "module",
+  "scripts": {
+    "check": "node scripts/check-sources.js",
+    "import:kjv": "node scripts/import-usfm.js sources/kjv-eng-kjv2006.json",
+    "build": "npm run check && npm run import:kjv"
+  },
+  "dependencies": {
+    "adm-zip": "^0.5.16"
+  }
+}
@@ -0,0 +1,27 @@
+{
+  "generated_at": "2026-07-12T02:30:18.202Z",
+  "resources": [
+    {
+      "id": "kjv-eng-kjv2006",
+      "title": "King James (Authorized) Version",
+      "abbreviation": "KJV",
+      "language": "eng",
+      "features": [
+        "strongs",
+        "usfm"
+      ],
+      "license": {
+        "name": "Public Domain",
+        "redistribution": true,
+        "jurisdiction_notes": "eBible states this work is firmly public domain outside the United Kingdom; UK printing/import may be restricted by Crown letters patent.",
+        "source_license_url": "https://ebible.org/find/show.php?id=eng-kjv2006"
+      },
+      "package_path": "packages/json/kjv-eng-kjv2006/catalog.json",
+      "counts": {
+        "books": 66,
+        "verses": 31102,
+        "strongs_links": 348884
+      }
+    }
+  ]
+}
@@ -0,0 +1,45 @@
+{
+  "id": "kjv-eng-kjv2006",
+  "title": "King James (Authorized) Version",
+  "abbreviation": "KJV",
+  "language": "eng",
+  "features": [
+    "strongs",
+    "usfm"
+  ],
+  "source": {
+    "provider": "eBible.org",
+    "url": "https://ebible.org/find/show.php?id=eng-kjv2006",
+    "download_url": "https://ebible.org/Scriptures/eng-kjv2006_usfm.zip",
+    "format": "usfm-zip",
+    "upstream_id": "eng-kjv2006",
+    "upstream_last_updated": "2026-05-16"
+  },
+  "license": {
+    "name": "Public Domain",
+    "redistribution": true,
+    "jurisdiction_notes": "eBible states this work is firmly public domain outside the United Kingdom; UK printing/import may be restricted by Crown letters patent.",
+    "source_license_url": "https://ebible.org/find/show.php?id=eng-kjv2006"
+  },
+  "importer": {
+    "name": "scripts/import-usfm.js",
+    "version": "0.1.0"
+  },
+  "generated_at": "2026-07-12T02:30:18.202Z",
+  "source_sha256": "4ea6952590d070bfa22985aded48a49581e31b568a60aa09e25f73462e700e7d",
+  "counts": {
+    "books": 66,
+    "verses": 31102,
+    "strongs_links": 348884
+  },
+  "files": {
+    "verses_jsonl": {
+      "path": "verses.jsonl",
+      "sha256": "e3102f854a99c4788ed259ec61cb93fea830845c0788c8c0929bc77e9db6ca10"
+    },
+    "strongs_jsonl": {
+      "path": "strongs-links.jsonl",
+      "sha256": "fa9702224776509275ea17fb245cb72150f1c262c4fbe44c7161e8e83866c779"
+    }
+  }
+}
@@ -0,0 +1,26 @@
+import { readdir } from 'node:fs/promises'
+import path from 'node:path'
+import { downloadFile, readJson, sha256File } from './lib.js'
+
+const root = process.cwd()
+const sourceDir = path.join(root, 'sources')
+const files = (await readdir(sourceDir)).filter((file) => file.endsWith('.json'))
+
+for (const file of files) {
+  const manifestPath = path.join(sourceDir, file)
+  const manifest = await readJson(manifestPath)
+  const cachePath = path.join(root, 'cache', manifest.id, path.basename(manifest.source.download_url))
+
+  await downloadFile(manifest.source.download_url, cachePath)
+  const actual = await sha256File(cachePath)
+  const expected = manifest.checks.expected_sha256?.toLowerCase()
+  const status = expected === actual ? 'unchanged' : 'changed'
+
+  console.log(`${manifest.id}: ${status}`)
+  console.log(`  expected: ${expected}`)
+  console.log(`  actual:   ${actual}`)
+
+  if (status === 'changed') {
+    process.exitCode = 1
+  }
+}
@@ -0,0 +1,159 @@
+import AdmZip from 'adm-zip'
+import { mkdir, readFile, rm, writeFile } from 'node:fs/promises'
+import path from 'node:path'
+import {
+  cleanUsfmText,
+  downloadFile,
+  exists,
+  normalizeStrong,
+  readJson,
+  sha256File,
+  writeJson,
+} from './lib.js'
+
+const manifestPath = process.argv[2]
+if (!manifestPath) {
+  throw new Error('Usage: node scripts/import-usfm.js sources/<manifest>.json')
+}
+
+const root = process.cwd()
+const manifest = await readJson(path.resolve(root, manifestPath))
+const archivePath = path.join(root, 'cache', manifest.id, path.basename(manifest.source.download_url))
+const extractDir = path.join(root, 'cache', manifest.id, 'usfm')
+const outputDir = path.join(root, 'packages', 'json', manifest.id)
+
+if (!(await exists(archivePath))) {
+  await downloadFile(manifest.source.download_url, archivePath)
+}
+
+const archiveSha = await sha256File(archivePath)
+if (archiveSha !== manifest.checks.expected_sha256.toLowerCase()) {
+  throw new Error(`Checksum mismatch for ${manifest.id}: ${archiveSha}`)
+}
+
+await rm(extractDir, { recursive: true, force: true })
+await mkdir(extractDir, { recursive: true })
+new AdmZip(archivePath).extractAllTo(extractDir, true)
+
+const zip = new AdmZip(archivePath)
+const usfmEntries = zip
+  .getEntries()
+  .filter((entry) => entry.entryName.toLowerCase().endsWith('.usfm'))
+  .sort((a, b) => a.entryName.localeCompare(b.entryName))
+
+const verses = []
+const strongsLinks = []
+let tokenId = 1
+
+for (const entry of usfmEntries) {
+  const filePath = path.join(extractDir, entry.entryName)
+  const usfm = await readFile(filePath, 'utf8')
+  const bookId = /\\id\s+([A-Z0-9]+)/.exec(usfm)?.[1]
+  const bookTitle = /\\toc2\s+(.+)/.exec(usfm)?.[1]?.trim() ?? bookId
+  let chapter = null
+
+  for (const line of usfm.split(/\r?\n/)) {
+    const chapterMatch = /^\\c\s+(\d+)/.exec(line)
+    if (chapterMatch) {
+      chapter = Number(chapterMatch[1])
+      continue
+    }
+
+    const verseMatch = /^\\v\s+(\d+)\s+([\s\S]+)/.exec(line)
+    if (!verseMatch || chapter === null || !bookId) {
+      continue
+    }
+
+    const verseNumber = Number(verseMatch[1])
+    const raw = verseMatch[2]
+    const verseId = `${bookId}.${chapter}.${verseNumber}`
+    const links = [...raw.matchAll(/\\\+?w\s+([^|\\]+)\|strong="([^"]+)"\\\+?w\*/g)]
+    const text = cleanUsfmText(raw.replace(/\\\+?w\s+([^|\\]+)\|strong="([^"]+)"\\\+?w\*/g, '$1'))
+
+    verses.push({
+      id: verseId,
+      translation_id: manifest.id,
+      book_id: bookId,
+      book: bookTitle,
+      chapter,
+      verse: verseNumber,
+      reference: `${bookTitle} ${chapter}:${verseNumber}`,
+      text,
+    })
+
+    for (const [index, link] of links.entries()) {
+      strongsLinks.push({
+        id: tokenId++,
+        verse_id: verseId,
+        translation_id: manifest.id,
+        book_id: bookId,
+        chapter,
+        verse: verseNumber,
+        position: index + 1,
+        surface: cleanUsfmText(link[1]),
+        strong_raw: link[2],
+        strong: normalizeStrong(link[2]),
+      })
+    }
+  }
+}
+
+await mkdir(outputDir, { recursive: true })
+await writeFile(
+  path.join(outputDir, 'verses.jsonl'),
+  `${verses.map((verse) => JSON.stringify(verse)).join('\n')}\n`,
+  'utf8',
+)
+await writeFile(
+  path.join(outputDir, 'strongs-links.jsonl'),
+  `${strongsLinks.map((link) => JSON.stringify(link)).join('\n')}\n`,
+  'utf8',
+)
+
+const catalog = {
+  id: manifest.id,
+  title: manifest.title,
+  abbreviation: manifest.abbreviation,
+  language: manifest.language,
+  features: manifest.features,
+  source: manifest.source,
+  license: manifest.license,
+  importer: manifest.importer,
+  generated_at: new Date().toISOString(),
+  source_sha256: archiveSha,
+  counts: {
+    books: new Set(verses.map((verse) => verse.book_id)).size,
+    verses: verses.length,
+    strongs_links: strongsLinks.length,
+  },
+  files: {
+    verses_jsonl: {
+      path: 'verses.jsonl',
+      sha256: await sha256File(path.join(outputDir, 'verses.jsonl')),
+    },
+    strongs_jsonl: {
+      path: 'strongs-links.jsonl',
+      sha256: await sha256File(path.join(outputDir, 'strongs-links.jsonl')),
+    },
+  },
+}
+
+await writeJson(path.join(outputDir, 'catalog.json'), catalog)
+await writeJson(path.join(root, 'packages', 'json', 'catalog.json'), {
+  generated_at: catalog.generated_at,
+  resources: [
+    {
+      id: catalog.id,
+      title: catalog.title,
+      abbreviation: catalog.abbreviation,
+      language: catalog.language,
+      features: catalog.features,
+      license: catalog.license,
+      package_path: `packages/json/${catalog.id}/catalog.json`,
+      counts: catalog.counts,
+    },
+  ],
+})
+
+console.log(`${manifest.id}: imported ${catalog.counts.verses} verses`)
+console.log(`${manifest.id}: imported ${catalog.counts.strongs_links} Strong's links`)
@@ -0,0 +1,71 @@
+import { createHash } from 'node:crypto'
+import { createWriteStream } from 'node:fs'
+import { mkdir, readFile, stat, writeFile } from 'node:fs/promises'
+import path from 'node:path'
+
+export async function readJson(filePath) {
+  return JSON.parse(await readFile(filePath, 'utf8'))
+}
+
+export async function writeJson(filePath, value) {
+  await mkdir(path.dirname(filePath), { recursive: true })
+  await writeFile(filePath, `${JSON.stringify(value, null, 2)}\n`, 'utf8')
+}
+
+export async function exists(filePath) {
+  try {
+    await stat(filePath)
+    return true
+  } catch {
+    return false
+  }
+}
+
+export async function downloadFile(url, filePath) {
+  await mkdir(path.dirname(filePath), { recursive: true })
+  const response = await fetch(url)
+  if (!response.ok || !response.body) {
+    throw new Error(`Download failed ${response.status} ${response.statusText}: ${url}`)
+  }
+
+  const output = createWriteStream(filePath)
+  await new Promise((resolve, reject) => {
+    response.body.pipeTo(
+      new WritableStream({
+        write(chunk) {
+          output.write(Buffer.from(chunk))
+        },
+        close() {
+          output.end(resolve)
+        },
+        abort(reason) {
+          output.destroy(reason)
+          reject(reason)
+        },
+      }),
+    ).catch(reject)
+  })
+}
+
+export async function sha256File(filePath) {
+  const data = await readFile(filePath)
+  return createHash('sha256').update(data).digest('hex')
+}
+
+export function normalizeStrong(value) {
+  const match = /^([GH])0*([0-9]+)$/i.exec(value)
+  return match ? `${match[1].toUpperCase()}${Number(match[2])}` : value.toUpperCase()
+}
+
+export function cleanUsfmText(input) {
+  return input
+    .replace(/\\f [\s\S]*?\\f\*/g, '')
+    .replace(/\\x [\s\S]*?\\x\*/g, '')
+    .replace(/\\add\s+([^\\]+)\\add\*/g, '$1')
+    .replace(/\\nd\s+([\s\S]*?)\\nd\*/g, '$1')
+    .replace(/\\[+\w]+\*/g, '')
+    .replace(/\\[a-z0-9+]+(?:\s+)?/gi, '')
+    .replace(/\s+/g, ' ')
+    .replace(/\s+([,.;:?!])/g, '$1')
+    .trim()
+}
@@ -0,0 +1,35 @@
+{
+  "id": "kjv-eng-kjv2006",
+  "title": "King James (Authorized) Version",
+  "abbreviation": "KJV",
+  "language": "eng",
+  "canon": "protocanon",
+  "features": ["strongs", "usfm"],
+  "source": {
+    "provider": "eBible.org",
+    "url": "https://ebible.org/find/show.php?id=eng-kjv2006",
+    "download_url": "https://ebible.org/Scriptures/eng-kjv2006_usfm.zip",
+    "format": "usfm-zip",
+    "upstream_id": "eng-kjv2006",
+    "upstream_last_updated": "2026-05-16"
+  },
+  "license": {
+    "name": "Public Domain",
+    "redistribution": true,
+    "jurisdiction_notes": "eBible states this work is firmly public domain outside the United Kingdom; UK printing/import may be restricted by Crown letters patent.",
+    "source_license_url": "https://ebible.org/find/show.php?id=eng-kjv2006"
+  },
+  "checks": {
+    "expected_sha256": "4ea6952590d070bfa22985aded48a49581e31b568a60aa09e25f73462e700e7d",
+    "last_checked_at": "2026-07-12T02:18:30Z"
+  },
+  "importer": {
+    "name": "scripts/import-usfm.js",
+    "version": "0.1.0"
+  },
+  "packages": {
+    "json_catalog": "packages/json/kjv-eng-kjv2006/catalog.json",
+    "verses_jsonl": "packages/json/kjv-eng-kjv2006/verses.jsonl",
+    "strongs_jsonl": "packages/json/kjv-eng-kjv2006/strongs-links.jsonl"
+  }
+}