OpenVoice/demo_part2.ipynb

Cross-Lingual Voice Clone Demo

In [ ]:

import os
import torch
import se_extractor
from api import ToneColorConverter

Initialization

In [ ]:

ckpt_converter = 'checkpoints/converter'
device = 'cuda:0'
output_dir = 'outputs'

tone_color_converter = ToneColorConverter(f'{ckpt_converter}/config.json', device=device)
tone_color_converter.load_ckpt(f'{ckpt_converter}/checkpoint.pth')

os.makedirs(output_dir, exist_ok=True)

In this demo, we will use OpenAI TTS as the base speaker to produce multi-lingual speech audio. The users can flexibly change the base speaker according to their own needs. Please create a file named .env and place OpenAI key as OPENAI_API_KEY=xxx.

In [ ]:

from openai import OpenAI
from dotenv import load_dotenv

# Please create a file named .env and place your
# OpenAI key as OPENAI_API_KEY=xxx
load_dotenv() 

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))

response = client.audio.speech.create(
    model="tts-1",
    voice="nova",
    input="This audio will be used to extract the base speaker tone color embedding. " + \
        "Typically a very short audio should be sufficient, but increasing the audio " + \
        "length will also improve the output audio quality."
)

response.stream_to_file(f"{output_dir}/openai_source_output.mp3")

Obtain Tone Color Embedding

The source_se is the tone color embedding of the base speaker. It is an average for multiple sentences with multiple emotions of the base speaker. We directly provide the result here but the readers feel free to extract source_se by themselves.

In [ ]:

base_speaker = f"{output_dir}/openai_source_output.mp3"
source_se, audio_name = se_extractor.get_se(base_speaker, tone_color_converter)

reference_speaker = 'resources/example_reference.mp3'
target_se, audio_name = se_extractor.get_se(reference_speaker, tone_color_converter)

Inference

In [ ]:

# Run the base speaker tts
text = [
    "MyShell is a decentralized and comprehensive platform for discovering, creating, and staking AI-native apps.",
    "MyShell es una plataforma descentralizada y completa para descubrir, crear y apostar por aplicaciones nativas de IA.",
    "MyShell est une plateforme décentralisée et complète pour découvrir, créer et miser sur des applications natives d'IA.",
    "MyShell ist eine dezentralisierte und umfassende Plattform zum Entdecken, Erstellen und Staken von KI-nativen Apps.",
    "MyShell è una piattaforma decentralizzata e completa per scoprire, creare e scommettere su app native di intelligenza artificiale.",
    "MyShellは、AIネイティブアプリの発見、作成、およびステーキングのための分散型かつ包括的なプラットフォームです。",
    "MyShell — это децентрализованная и всеобъемлющая платформа для обнаружения, создания и стейкинга AI-ориентированных приложений.",
    "MyShell هي منصة لامركزية وشاملة لاكتشاف وإنشاء ورهان تطبيقات الذكاء الاصطناعي الأصلية.",
    "MyShell是一个去中心化且全面的平台，用于发现、创建和投资AI原生应用程序。",
    "MyShell एक विकेंद्रीकृत और व्यापक मंच है, जो AI-मूल ऐप्स की खोज, सृजन और स्टेकिंग के लिए है।",
    "MyShell é uma plataforma descentralizada e abrangente para descobrir, criar e apostar em aplicativos nativos de IA."
]
src_path = f'{output_dir}/tmp.wav'

for i, t in enumerate(text):

    response = client.audio.speech.create(
        model="tts-1",
        voice="alloy",
        input=t,
    )

    response.stream_to_file(src_path)

    save_path = f'{output_dir}/output_crosslingual_{i}.wav'

    # Run the tone color converter
    encode_message = "@MyShell"
    tone_color_converter.convert(
        audio_src_path=src_path, 
        src_se=source_se, 
        tgt_se=target_se, 
        output_path=save_path,
        message=encode_message)