Building ProjectSola: Converting 20 Years of Sermon Audio into Theological Essays

My church has over 20 years of audio and video sermons archived on its website. It is an incredibly rich library of theological teaching, but it suffers from a common problem: audio and video aren’t terribly consumable en masse. If you are looking for a specific topic, studying a particular passage of scripture, or wanting to quickly read through a sermon’s key takeaways, scrubbing through a 45-minute audio file is highly inefficient. There was little to no metadata surrounding these files, making two decades of valuable teaching practically invisible to search engines and readers alike.

I wanted to change that. My goal was to break down these audio archives into text, extract and index scripture coverage for deep cross-referencing, and rewrite the rough audio transcripts into highly readable, structured theological essays.

To achieve this, I built ProjectSola (hosted at sola.wescrockett.com)—a dual-repository system featuring a Python data processing pipeline on Unraid (projectSola) and a modern Astro static web frontend (projectSola-web).

Here is how I engineered the automated pipeline, management console, and static site to process over 1.8 million words of audio into structured digital essays.

By the Numbers: Archive Scope vs. Completed Output

To visualize the scale of converting two decades of church audio archives into structured digital text, here is the empirical breakdown:

The Architecture & Pipeline

ProjectSola is structured as a multi-stage Python data pipeline, managed by a visual Flask console running in my Unraid homelab server, feeding into an Astro static web frontend.

┌─────────────────┐      ┌───────────────────┐      ┌─────────────────────┐      ┌─────────────────────┐      ┌────────────────────┐
│  scrape_series  │ ──▶  │  process_sermons  │ ──▶  │ format_transcripts  │ ──▶  │ HTML-to-MD Convert  │ ──▶  │ projectSola-web    │
│                 │      │                   │      │                     │      │                     │      │ (Astro SSG)        │
│ Scrape series   │      │ Transcribe audio  │      │ AI formats essays   │      │ Generate Markdown   │      │ Deploy site, search│
│ page, build CSV │      │ via URL (Deepgram)│      │ Embed scripture     │      │ for content schema  │      │ & cross-reference  │
└─────────────────┘      └───────────────────┘      └─────────────────────┘      └─────────────────────┘      └────────────────────┘
         │                        │                          │                          │                            │
         ▼                        ▼                          ▼                          ▼                            ▼
  data/series/             data/transcripts/           data/essays/               data/markdown/               Astro Content        
  {slug}_Series.csv        {slug}/*.txt                {slug}/*.html              {slug}/*.md                  Collection          

Stage 1: Web Scraping (scrape_series.py)

The pipeline begins by targeting a sermon series page from the church website.

• Using a pool of 10 concurrent workers, the scraper parses the series index to gather all sermon links.
• For each sermon, it extracts metadata: the title, date, speaker, passage tags, and the raw MP3 audio URL.
• This metadata is saved into a normalized, version-controlled CSV catalog ({slug}_Series.csv).

Stage 2: URL-Based Audio Transcription (process_sermons.py)

Traditional transcription pipelines require downloading gigabytes of MP3 files, saving them to disk, and uploading them to an API. This is slow and bandwidth-heavy.

• Instead, ProjectSola utilizes Deepgram’s URL-based transcription.
• We send the audio URL directly to Deepgram’s API, and Deepgram streams and transcribes the audio straight from the source.
• Powered by 15 concurrent workers, this stage transcribes entire series in parallel, saving the raw transcripts as text files and updating the metadata CSV with their file paths.

Stage 3: AI Theological Editing & Scripture Integration (format_transcripts.py)

Raw speech-to-text transcripts are messy. They contain verbal filler, grammatical stumbles, and lack proper paragraph structure.

• We feed the raw transcripts into Gemini models with structured system prompts.
• Gemini formats the transcript into a polished theological essay. It preserves the speaker’s voice and theological depth while organizing the text into clear sections with headings and bullet points.
• Crucially, the model identifies every verbal scripture reference. Using a Bible utility layer, we look up the cited passages and programmatically insert the exact NASB scripture text directly into blockquotes inside the essay.

Stage 4: HTML-to-Markdown Conversion & Sync

To bridge the gap between output essays and modern web publishing:

• An automated HTML-to-Markdown converter translates generated essays into clean, version-controlled .md files formatted with frontmatter schemas.
• Over 640 formatted markdown essays (spanning series like Follow, 2 Corinthians, Hebrews, Peter, and more) are synced to the web content directory.
• The SyncManager handles path validation between Windows UNC share paths and local Unraid Linux mount paths (/mnt/user/...), providing granular metrics on new vs. updated files.

Modern Astro Frontend (projectSola-web)

To deliver a lightning-fast reader experience, I decoupled the web UI into a dedicated Astro SSG repository (projectSola-web).

Features & Optimizations:

• Static Site Generation (SSG): Pre-renders 640+ sermon essays into static HTML pages for instant page loads and optimal SEO.
• Normalized Slug Routing: Normalizes sermon URL slugs with lowercasing and dashed formatting to ensure robust static route generation.
• Scripture Cross-Reference System: Features cascading filters (Book → Chapter → Verse) and smart passage matching so users can search 20 years of sermons by exact scripture passage.
• Enhanced Media & UX: Replaced embedded audio players with direct source page and audio stream links, added custom scripture favicons, branded header logos, and crisp typographic hierarchy.

Unraid Management Console & Docker CI/CD (projectSola)

To oversee processing status and automate deployments, I built a Flask-based visual management console running in a Docker container on my Unraid server.

# Thread-safe job execution queue in Flask server
class ProcessingJob:
    def __init__(self, series_slug):
        self.series_slug = series_slug
        self.status = "queued"
        self.progress = 0
        self.logs = []

Infrastructure & Engineering Highlights:

• Real-Time Log Telemetry: Uses Server-Sent Events (SSE) to stream live console logs and progress updates to the UI, complete with auto-scroll position preservation.
• Docker & Unraid CI/CD: Built a GitHub Actions workflow with docker buildx and provenance: false to ensure clean container image builds without Unraid manifest resolution loops.
• Reverse Proxy Integration: Integrates SWAG (Secure Web Application Gateway) volume mounts in docker-compose.yml for zero-trust SSL reverse-proxy routing.

The Vision: Preserving Theological Heritage

ProjectSola has turned over 200+ hours of processed audio and 1.8M+ words into a searchable, readable digital library. By combining Python automation, Deepgram, Gemini AI, Docker on Unraid, and an Astro static frontend, we’ve unlocked two decades of deep biblical teaching for pastors, students, and readers worldwide.