OpenVoice/demo_part1.ipynb

Voice Style Control Demo

In [ ]:

import os
import torch
import se_extractor
from api import BaseSpeakerTTS, ToneColorConverter

Initialization

In [ ]:

ckpt_base = 'checkpoints/base_speakers/EN'
ckpt_converter = 'checkpoints/converter'
device = 'cuda:0'
output_dir = 'outputs'

base_speaker_tts = BaseSpeakerTTS(f'{ckpt_base}/config.json', device=device)
base_speaker_tts.load_ckpt(f'{ckpt_base}/checkpoint.pth')

tone_color_converter = ToneColorConverter(f'{ckpt_converter}/config.json', device=device)
tone_color_converter.load_ckpt(f'{ckpt_converter}/checkpoint.pth')

os.makedirs(output_dir, exist_ok=True)

Obtain Tone Color Embedding

The source_se is the tone color embedding of the base speaker. It is an average of multiple sentences generated by the base speaker. We directly provide the result here but the readers feel free to extract source_se by themselves.

In [ ]:

source_se = torch.load(f'{ckpt_base}/en_default_se.pth').to(device)

The reference_speaker.mp3 below points to the short audio clip of the reference whose voice we want to clone. We provide an example here. If you use your own reference speakers, please make sure each speaker has a unique filename. The se_extractor will save the targeted_se using the filename of the audio and will not automatically overwrite.

In [ ]:

reference_speaker = 'resources/example_reference.mp3'
target_se, audio_name = se_extractor.get_se(reference_speaker, tone_color_converter)

Inference

In [ ]:

save_path = f'{output_dir}/output_en_default.wav'

# Run the base speaker tts
text = "This audio is generated by open voice."
src_path = f'{output_dir}/tmp.wav'
base_speaker_tts.tts(text, src_path, speaker='default', language='English', speed=1.0)

# Run the tone color converter
encode_message = "@MyShell"
tone_color_converter.convert(
    audio_src_path=src_path, 
    src_se=source_se, 
    tgt_se=target_se, 
    output_path=save_path,
    message=encode_message)

Try with different styles and speed. The style can be controlled by the speaker parameter in the base_speaker_tts.tts method. Available choices: friendly, cheerful, excited, sad, angry, terrified, shouting, whispering. Note that the tone color embedding need to be updated. The speed can be controlled by the speed parameter. Let's try whispering with speed 0.9.

In [ ]:

source_se = torch.load(f'{ckpt_base}/en_style_se.pth').to(device)
save_path = f'{output_dir}/output_whispering.wav'

# Run the base speaker tts
text = "This audio is generated by open voice with a half-performance model."
src_path = f'{output_dir}/tmp.wav'
base_speaker_tts.tts(text, src_path, speaker='whispering', language='English', speed=0.9)

# Run the tone color converter
encode_message = "@MyShell"
tone_color_converter.convert(
    audio_src_path=src_path, 
    src_se=source_se, 
    tgt_se=target_se, 
    output_path=save_path,
    message=encode_message)

Try with different languages. OpenVoice can achieve multi-lingual voice cloning by simply replace the base speaker. We provide an example with a Chinese base speaker here and we encourage the readers to try demo_part2.ipynb for a detailed demo.

In [ ]:

ckpt_base = 'checkpoints/base_speakers/ZH'
base_speaker_tts = BaseSpeakerTTS(f'{ckpt_base}/config.json', device=device)
base_speaker_tts.load_ckpt(f'{ckpt_base}/checkpoint.pth')

source_se = torch.load(f'{ckpt_base}/zh_default_se.pth').to(device)
save_path = f'{output_dir}/output_chinese.wav'

# Run the base speaker tts
text = "今天天气真好，我们一起出去吃饭吧。"
src_path = f'{output_dir}/tmp.wav'
base_speaker_tts.tts(text, src_path, speaker='default', language='Chinese', speed=1.0)

# Run the tone color converter
encode_message = "@MyShell"
tone_color_converter.convert(
    audio_src_path=src_path, 
    src_se=source_se, 
    tgt_se=target_se, 
    output_path=save_path,
    message=encode_message)

Tech for good. For people who will deploy OpenVoice for public usage: We offer you the option to add watermark to avoid potential misuse. Please see the ToneColorConverter class. MyShell reserves the ability to detect whether an audio is generated by OpenVoice, no matter whether the watermark is added or not.