From Silence to Symphony: How We Built an AI-Powered Audio Mixing Engine with FFmpeg 8 and Intelligent Agents

Miguel Enes··5 min read·Engineering
aicontent-generationvizraphp

Building an AI-Powered Audio Mixing Engine

The code below is an illustrative sketch of the workflow, not a versioned production listing. The available source record does not establish which model release was used or when each component shipped.

Professional meditation audio requires precise management of frequencies, dynamics, and timing. Human audio engineers rely on years of experience to balance voice tracks over background music.

We automated the creation of studio-quality meditation mantras. We mix voice narrations with background music. We built a system that learns and adapts with every successful mix.

The Architecture: Multi-Agent System

We built a multi-agent system using the Vizra ADK. The system orchestrates specialized components:

┌─────────────────────────────────────────────────────────────┐
│                 MantraGenerationWorkflow                    │
├─────────────────────────────────────────────────────────────┤
│  ┌─────────────────┐    ┌──────────────────────────────┐   │
│  │ Content Finder  │ →  │  MantraGenerationAgent       │   │
│  │ (FindUnmixed)   │    │  (MantraGenerationAgent)     │   │
│  └─────────────────┘    └──────────────────────────────┘   │
│            ↓                           ↓                    │
│  ┌─────────────────────────────────────────────────────┐   │
│  │          IntelligentFFmpegMixerTool                 │   │
│  │  ┌───────────┐  ┌────────────┐  ┌────────────────┐  │   │
│  │  │ Model     │→ │ Vector RAG │→ │ FFmpeg 8 Engine│  │   │
│  │  │ Parameter │  │  Learning  │  │   Execution    │  │   │
│  │  │ Generator │  │   System   │  │                │  │   │
│  │  └───────────┘  └────────────┘  └────────────────┘  │   │
│  └─────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────┘

The MantraGenerationAgent

Our primary agent acts as an audio engineer. It coordinates the mixing process:

class MantraGenerationAgent extends BaseLlmAgent
{
    protected string $model = 'provider-model'; // illustrative placeholder
    protected ?float $temperature = 0.5;

    protected array $tools = [
        FindUnmixedContentTool::class,
        IntelligentFFmpegMixerTool::class,
        VectorMemoryTool::class,
    ];
}

The agent processes high-level instructions. It pairs voice and music content. It consults past successful mixes to adjust parameters based on time-of-day context.

The Intelligence Layer: Gemini and Audio Analysis

Audio Analysis

The system analyzes audio characteristics using FFmpeg probe commands before mixing:

private function analyzeAudio(string $path): array
{
    $command = sprintf(
        'ffprobe -v quiet -print_format json -show_format -show_streams %s',
        escapeshellarg($path)
    );

    return [
        'duration' => (float) ($info['format']['duration'] ?? 0),
        'bitrate' => (int) ($info['format']['bit_rate'] ?? 0) / 1000,
        'channels' => (int) ($info['streams'][0]['channels'] ?? 0),
        'sample_rate' => (int) ($info['streams'][0]['sample_rate'] ?? 44100),
    ];
}

Parameter Generation

We pass audio files and extracted metadata to a language model. It proposes FFmpeg parameters for the specific voice-music pairing. The example leaves the provider and model release unspecified:

$response = Prism::structured()
    ->using($provider, $model)
    ->withPrompt($prompt, [$voiceAttachment, $musicAttachment])
    ->withSchema($mixingParametersSchema)
    ->asStructured();

The Core Mix: FFmpeg 8 Filter Chains

Filter Construction

The system constructs FFmpeg filter chains based on AI recommendations:

private function buildFilters(array $params): string
{
    $filters = [];

    // Voice processing: normalize, gain, convert to stereo
    $filters[] = '[0:a]loudnorm=I=-18:LRA=10:TP=-1.5,volume=1.5dB,aformat=channel_layouts=stereo[voice_norm]';

    // Voice delay
    $delayMs = $params['voice_delay_seconds'] * 1000;
    $filters[] = "[voice_norm]adelay={$delayMs}|{$delayMs}[voice_for_mix]";

    // Music processing: normalize, EQ, gain
    $filters[] = '[1:a]loudnorm=I=-19:LRA=9:TP=-2,' .
                 'highpass=f=80,lowpass=f=12000,' .
                 'equalizer=f=4000:width_type=h:width=800:g=-3,' .
                 'volume=-1dB[music_proc]';

    // The mix with intelligent weighting
    $filters[] = '[music_proc][voice_for_mix]amix=inputs=2:normalize=0:dropout_transition=2:weights=1 1[mix_combined]';

    // Final mastering: fade-in and loudness normalization
    $filters[] = '[mix_combined]afade=t=in:ss=0:d=0.12,loudnorm=I=-15:LRA=8:TP=-2[final]';

    return implode(';', $filters);
}

Time-Context Mixing

The system adapts parameters based on the listening context. Morning mixes use a 3-4 second voice delay with gentle ducking. Night mixes use a 6-8 second delay with strong ducking.

Vector Memory and RAG

Storing Successful Mixes

We use every successful mix as training data. We store the parameters and metadata:

private function storeSuccessfulMix(
    array $params,
    array $voiceInfo,
    array $musicInfo,
    AgentContext $context
): void {
    $description = sprintf(
        'Successful audio mix: Voice duration %.1fs, Music duration %.1fs. ' .
        'Ducking: %s threshold, %.2f ratio, %dms attack, %dms release.',
        $voiceInfo['duration'],
        $musicInfo['duration'],
        $params['ducking_threshold'],
        $params['ducking_ratio'],
        $params['ducking_attack'],
        $params['ducking_release']
    );

    $agent->vector()->addDocument([
        'content' => $description,
        'metadata' => [
            'parameters' => $params,
            'duration_ratio' => $voiceInfo['duration'] / max($musicInfo['duration'], 1),
            'timestamp' => now()->toIso8601String(),
        ],
        'namespace' => 'audio_mixing',
    ]);
}

Retrieval-Augmented Generation

The system queries its vector memory for similar successful mixes before processing new content:

$similarMixes = $agent->rag()->search([
    'query' => "meditation mantra mixing with {$voice->label} and {$music->label}",
    'namespace' => 'audio_mixing',
    'limit' => 3,
    'threshold' => 0.7,
]);

A morning mantra inherits parameters from past morning mixes. Gemini refines the final settings for the specific content.

Quality Assurance

We use LLM judges to assess mix quality and score the output.

Voice Clarity Assertion

class VoiceClarityAssertion extends BaseAssertion
{
    protected function getPrompt(
        string $input,
        string $output,
        ?string $expected = null
    ): string {
        return <<<PROMPT
        Evaluate voice clarity in this meditation mantra mix:

        **Clarity Criteria:**
        1. Voice Prominence (30 points)
        2. Intelligibility (30 points)
        3. Frequency Clarity (20 points)
        4. Mix Balance (20 points)

        Voice should be effortlessly intelligible and feel natural,
        as if narrator is speaking directly to listener with gentle
        ambient music in background.
        PROMPT;
    }
}

Parameter Guardrails

We enforce strict parameter bounds to ensure technical viability:

private const SAFE_VOICE_NORMALIZATION = 'loudnorm=I=-18:LRA=10:TP=-1.5:dual_mono=true:linear=true';
private const SAFE_DUCKING_RATIO = 3;
private const SAFE_DUCKING_THRESHOLD_DB = -22;
private const MIN_VOICE_DELAY_SECONDS = 2;
private const MAX_VOICE_DELAY_SECONDS = 20;

private function enforceParameterSafety(
    array $params,
    array $voiceInfo,
    array $musicInfo
): array {
    $availableIntro = $musicDuration - $voiceDuration - self::MUSIC_TAIL_BUFFER_SECONDS;
    $params['voice_delay_seconds'] = max(
        self::MIN_VOICE_DELAY_SECONDS,
        min(self::MAX_VOICE_DELAY_SECONDS, (int) round($availableIntro))
    );

    $params['ducking_ratio'] = max(3, min(12, $params['ducking_ratio']));
    $params['voice_gain_db'] = max(-2.0, min(3.0, $params['voice_gain_db']));
    $params['music_gain_db'] = max(-4.0, min(1.0, $params['music_gain_db']));

    return $params;
}

Review before delivery

The pipeline can store parameters and audio characteristics from a successful mix, retrieve similar examples, and place proposed settings behind bounds before FFmpeg runs. A reviewer can then check the resulting mix. This description does not claim a measured processing time, pass rate, or intervention rate. Related on this site: Production AI workflows.

Related Posts