Resolving Choppy AI Voice Playback on Linux Clients by Tuning Audio Scenario Buffering

Issue Description

When integrating Conversational AI (ConvoAI) agents with an Agora RTC SDK client running in a Linux server environment, audio playback from the AI agent may sound choppy, stuttered, or suffer from audible lag. The same conversation stream plays clearly and smoothly when connected via Web or Mobile SDKs. Standard audio scenario toggles (such as switching between AUDIO_SCENARIO_AI_CLIENT and AUDIO_SCENARIO_AI_SERVER) may fail to resolve the stuttering on Linux.

Platform and SDK Context

• Operating System: Linux (Server/Headless Environment)

• SDK Component: Agora RTC C++ Native SDK for Linux / Server SDK (integrated with ConvoAI)

• Target Issue: Audio playback frame timing jitter and buffer underflow

Root Cause Analysis

This issue is driven by thread scheduling jitter and callback timing drift within the Linux operating system environment.

Under the Hood (Technical Mechanism): The SDK's PCM rendering pipeline expects the raw audio playback callback (onPlaybackAudioFrame / onPlayback) to fire at tight, predictable intervals (typically 10 ms per audio frame).

In Linux server environments—especially those without real-time kernel priority configuration—thread scheduling delays can cause the callback delivery rate to fluctuate. While minor millisecond timing jitter is normally tolerated, severe timing drift delays frame consumption. When the internal rendering ring buffer empties before the next frame is delivered, a Buffer Underflow occurs, manifesting as lost audio frames, pitch distortion, and choppy speech synthesis.

Solution and Resolution Steps

1. Switch Audio Scenario to Chorus Mode

   Update the audio scenario on the Linux client to AUDIO_SCENARIO_CHORUS (or equivalent enum in your SDK version). This scenario configures the SDK's internal Audio Processing Module (APM) with an enlarged playback jitter buffer, designed specifically to absorb timing irregularities in raw audio callbacks.

2. Re-initialize the Engine Session

   Ensure that the audio scenario is configured during engine initialization or before joining the channel. If modified at runtime, restart the client engine instance so the audio pipeline re-allocates its ring buffer with the new capacity.

3. Implement Client-Side Jitter Buffering for Custom Playback Logic

   If application uses raw PCM audio callbacks (IAudioFrameObserver) and renders sound via custom ALSA/PulseAudio drivers, implement a lightweight client-side FIFO ring buffer:

   • Store incoming PCM frames from the SDK callback into a local queue.

   • Consume frames at a fixed hardware clock rate to decouple OS thread scheduling from hardware playback.

4. Audit Callback Timing Consistency

   Log the delta timestamps between consecutive audio callbacks in your staging environment. Verify that frame arrival counts remain aligned over a 1-second sliding window, ensuring total frame throughput matches the target sample rate (e.g., 100 frames/sec for 10 ms frames).

Prevention and Best Practice

• Use High-Capacity Scenarios for Headless Linux: When running SDK instances on headless Linux servers where thread scheduling is non-deterministic, favor audio scenarios with larger jitter buffers (such as Chorus or Game Streaming) rather than ultra-low-latency real-time scenarios.

• Linux Process Priority: Consider elevating the execution priority of the Linux worker process (chrt or nice parameters) to grant real-time scheduling priority to the RTC audio processing thread.