framevideo-voiceover-ssml

FrameVideo Voiceover SSML

Generate narration scripts from existing project content (subtitles, transcripts, AI analysis) and enhance them with SSML-style markup for pronunciation fixes, pauses, and number reading control.

When To Use

• User wants narration generated from project subtitles or transcript
• Need to fix pronunciation of brand names, technical terms, or foreign words
• Need to control pause timing for dramatic effect or pacing
• Need to normalize how numbers, dates, or prices are spoken
• Expanding terse subtitles into natural narration
• Creating voiceover that stays aligned with visible text

Do NOT Use

• For simple text-to-speech without markup → use framevideo-media (tts) directly
• For digital human video synthesis → use chanjing-digital-human
• For choosing TTS voices or providers → use framevideo-media
• When the user provides a complete narration script ready for TTS

Quick Start

Minimal workflow:

# 1. Find source text (subtitle file, transcript, or analysis)
cat subtitles.txt
# Output: "Visit chanjing.ai for more info"
 
# 2. Add SSML markup for pronunciation
echo '<phoneme alphabet="ipa" ph="tʃæn.dʒɪŋ">Chanjing</phoneme> <break time="0.3s"/> Visit chanjing dot AI for more info.'

Core Concepts

Source Text Priority

Find the source text in this order:

1. Project subtitles/transcript — the primary source, already aligned with visible content
2. AI analysis text — scene descriptions or product copy extracted during storyboard
3. Existing narration draft — if the user provided one
4. Manual authoring — last resort when no source exists

Why subtitles first? They're already timed to the visual beats and match what viewers see on screen.

SSML-Style Markup

Three tags for delivery control:

Markup is authoring-layer only. Keep it in your script file, but strip it before sending to TTS providers that don't support SSML.

Fallback Strategy

Not all TTS providers support SSML tags:

• Kokoro (local): Does NOT support SSML — strip tags before generation
• Chanjing TTS: Supports phoneme, break, ttnumber — pass through unchanged
• ElevenLabs: Supports subset of SSML — check their docs

Always save both versions:

• script-marked.txt — original with SSML tags
• script-fallback.txt — clean text for non-SSML providers

Workflow

Step 1: Locate Source Text

Step 2: Normalize Into Speakable Script

Expand terse subtitles:

Keep alignment with visible text:

• If subtitle says "50% faster", narration should say "fifty percent faster" at the same timing
• If product name appears on screen as "FrameVideo", say "FrameVideo" not "frame video"

Text normalization rules:

• Convert & to "and"
• Spell out acronyms on first use: "TTS, or text to speech"
• Convert URLs: "chanjing.ai" → "chanjing dot AI"
• Convert symbols: $99 → "ninety-nine dollars"

Step 3: Add SSML Markup

Use sparingly. Only add markup where TTS misreads or pacing is critical.

Phoneme Tag — Pronunciation Fixes

Chinese names and brands:

English technical terms:

Product names:

When to use:

• Brand names mispronounced by TTS
• Technical jargon with non-standard pronunciation
• Foreign words in English narration
• Acronyms that should be spelled vs pronounced

Alphabets:

• ipa — International Phonetic Alphabet (universal)
• py — Pinyin for Mandarin Chinese
• x-sampa — Extended SAMPA (ASCII-safe IPA)

Break Tag — Pause Control

Dramatic pauses:

List separation:

Transition markers:

Timing guidelines:

• 0.2s — natural comma pause
• 0.3-0.5s — sentence boundary, breath
• 0.6-0.8s — dramatic pause, emphasis
• 1.0s+ — scene transition, major shift

When to use:

• Default TTS pacing feels rushed
• Need emphasis before key message
• Narration must sync with on-screen timing

When NOT to use:

• Every comma (let TTS handle natural pauses)
• Replacing proper sentence structure

Ttnumber Tag — Number Reading

Years:

Product codes:

Prices:

Dates:

When to use:

• Default TTS reads numbers awkwardly ("two thousand twenty-six" vs "twenty twenty-six")
• Product codes should be spelled ("A-X-three-hundred" not "ax three hundred")
• Currency needs natural phrasing
• Dates need cultural reading convention

Step 4: Choose TTS Provider

Decision tree:

Provider SSML support:

Step 5: Generate Audio

With SSML support (Chanjing):

Without SSML support (Kokoro):

Stripping SSML tags:

Step 6: Save Both Versions

File structure:

Why save both?

• Marked version preserves authoring intent
• Fallback version for re-generation with different TTS
• Future TTS providers may add SSML support

Step 7: Writeback to Project

Generate transcript from audio:

Use in composition:

Sync captions (see framevideo skill for caption patterns):

Patterns

Pattern 1: Subtitle Expansion

Input subtitles.txt:

Output script-marked.txt:

Pattern 2: Brand Name Consistency

DESIGN.md snippet:

Apply in script:

Pattern 3: Technical Demo Narration

For product walkthroughs:

Pattern 4: Multilingual Product Names

English narration with Chinese brand:

Pattern 5: Number-Heavy Content

Pricing and metrics:

Rules (Non-Negotiable)

1. Always save the marked script — even if TTS doesn't support SSML today
2. Fallback text must be natural — stripping tags shouldn't break grammar
3. Transcribe generated audio — don't assume timing matches script
4. Source text takes priority — expand subtitles, don't replace them
5. Markup is sparse — only where TTS fails or timing is critical

Avoid

• ❌ Adding SSML to every sentence — only fix actual problems
• ❌ Using <break> instead of rewriting awkward phrasing
• ❌ Assuming Kokoro supports SSML — it doesn't, strip tags first
• ❌ Hard-coding phonemes without testing TTS output — verify it helps
• ❌ Ignoring provider limitations — check what tags are supported
• ❌ Discarding the marked script after generation — keep for iteration

Integration

With framevideo-media

With chanjing-digital-human

With website-to-framevideo

Validation

1. Check SSML syntax:

2. Test fallback stripping:

3. A/B test TTS output:

4. Verify transcript alignment:

References

• references/workflow.md — step-by-step workflow details
• references/ssml.md — SSML tag specifications
• framevideo-media skill — TTS generation and transcription
• chanjing-digital-human skill — SSML-aware TTS via Chanjing platform
• framevideo skill — audio element conventions and caption timing
• SSML W3C spec: https://www.w3.org/TR/speech-synthesis/
• IPA chart: https://www.internationalphoneticalphabet.org/ipa-charts/