How to use cloud recording with Conversational AI?

This document provides the configuration to resolve audio experience inconsistencies that may occur when integrating the Agora Conversational AI Agent (the Agent) with the Cloud Recording service.

Overview: Core Issues and Solutions

Two primary issues can affect the audio experience when the Agent and Cloud Recording are used in the same channel:

1. Audio Scenario Conflict: Causes the Agent's audio optimization to fail and may create a short break at the start of a recording.
2. Silence Stream Publishing: The Agent publishes an audio stream even when there is no speech, resulting in a silent segment at the beginning of the recording file.

The table below summarizes the root causes and the corresponding configuration solutions:

Issue	Symptom	Root Cause	Solution
Audio Scenario Incompatibility	The Agent's AI_Server scenario falls back to a default mode when a Cloud Recording user joins the channel. A short audio break may occur when recording starts.	The Cloud Recording client does not yet support the AI_Client audio scenario, preventing it from forming an optimized pair with the Agent's AI_Server.	Set the Agent's audio_scenario to "chorus".
Automatic Publishing During Silence	The recorded file begins with a period of silence.	By default, the Agent automatically publishes an audio stream upon joining(Except AI_Server scenario), even when silent. Cloud Recording immediately begins recording this silent stream.	Disable the Agent's publish_audio_auto_publishing parameter.

Configuration Steps and Examples

Apply the following configurations to your Agent's startup request body.

1. Set the Audio Scenario to "chorus"

This ensures a stable audio processing mode for the Agent in channels shared with Cloud Recording.

Add or modify the audio_scenario field within the Agent's startup parameters:

{
  "properties": {
    "parameters": {
      "audio_scenario": "chorus" // Use "chorus" instead of "AI_Server"
    }
  }
}

2. Disable Automatic Audio Stream Publishing

This configures the Agent to publish a stream only when it detects actual audio, preventing the recording of silent segments.

Add the following field within the rtc configuration:

{
  "properties": {
    "parameters": {
      "rtc": {
        "publish_audio_auto_publishing": false
      }
      // ... other parameters
    }
  }
}

Complete Configuration Example

Below is a consolidated example of an Agent startup request body incorporating the above optimizations:

{
    "name": "TestConvoAgent",
    "properties": {
        "channel": "{{AccessChannel}}",
        "token": "{{token}}",
        "agent_rtc_uid": "0",
        "remote_rtc_uids": [
            "*"
        ],
        "idle_timeout": 30,
        "advanced_features": {
            "enable_bhvs": true
        },
        "llm": {
            "style": "openai",
            "url": "https://api.openai.com/v1/chat/completions",
            "api_key": "{{OPENAI_TOKEN}}",
            "system_messages": [
                {
                    "role": "system",
                    "content": "You are a helpful chatbot."
                }
            ],
            "max_history": 10,
            "greeting_message": "How is your day? This is Jack from Agora Support",
            "failure_message": "Please hold on a second.",
            "silence_message": "Are you still there?",
            "input_modalities": ["text"],
            "output_modalities": ["text"],
            "params": {
                "model": "gpt-4o-mini",
                "max_tokens": 1024
            }
        },
        "tts": {
            "vendor": "elevenlabs",
            "params": {
                "key": "{{11labs}}",
                "model_id": "eleven_turbo_v2_5",
                "sample_rate": 24000,
                "voice_id": "{{11labsVoice}}"
            }
        },
        "turn_detection": {
            "silence_duration_ms": 640
        },
        "parameters": {
            "enable_dump": true,
            "enable_error_message": true,
            "audio_scenario": "chorus",
            "enable_delay": true,
            "rtc": {
                "publish_audio_auto_publishing": false
            }
        }
    }
}

Associated Operation: Starting Cloud Recording

After starting the Agent with the correct configuration, you can start the Cloud Recording service by calling the Start method. https://docs.agora.io/en/cloud-recording/reference/restful-api。Below is a consolidated example of a Cloud Recording startup request body:

{
	"cname":"{{AccessChannel}}",
	"uid":"{{RecordingUID}}",
	"clientRequest":{
		"token":"{{token}}",
		"recordingConfig":{
			"maxIdleTime":120,
			"streamTypes":0,
			"audioProfile":1,
			"channelType":1
		},
        "recordingFileConfig": {
            "avFileType": [
                "hls",
                "mp4"
            ]
        },
		"storageConfig":{
			"vendor":{{Vendor}},
			"region":{{Region}},
			"bucket":"{{Bucket}}",
			"accessKey":"{{AccessKey}}",
			"secretKey":"{{SecretKey}}"
		}	
	}
}