RenAI-Chat

:email: RenAI Chat

This project aims to create a fun interface to use conversational models in the form of a Visual Novel in Renpy. It’s using multiple AI models:

• Pygmalion conversational AI or other Huggingface models
• TTS Coqui-AI and Tortoise-TTS for Text to Speech
• OpenAI Whisper with microphone option for Speech to Text

Overview

:boom: Installation

Check out the wiki page.

:fire: Features

• Speak without scripted text using the latest chatbots
• Clone your character voice with a Text to Speech module using extracts of voiced dialogues
• Talk with your own voice thanks to Speech recognition

:eyeglasses: How to add chatbot models

• Check the wiki to setup your GPU and for pratical info about chatbots to use

• On Huggingface website go to the model page, for example https://huggingface.co/PygmalionAI/pygmalion-2.7b

• Follow the instructions to download the model like this on your Command Prompt or Powershell opened wherever you want:

git lfs install
git clone https://huggingface.co/PygmalionAI/pygmalion-2.7b

• Once the download is finished, put the folder created in chatbot_models in the RenAIChat folder. You will be able to choose it in your next login !

:microphone: Customize voice

For Your TTS model (worse but faster, ~5s per turn): You can change the voice used by placing extracts in the coquiai_audios folder. The longer the extract, the longer the TTS will take to generate the audio at each turn (~1min is good). It has to be .wav audio files, use an online converter if you have .mp3 files.

For Tortoise TTS model (better but slower, ~40s per turn): You can change the voice samples in tortoise_voices_ folder. Create your own character by adding a folder with the name of your character and put the audio samples in it. The samples must be around 10 seconds long and at the number of 5 maximum. Prefer .wav files too.

On CPU, it can take 10x more time to generate the voice (Tortoise TTS can have unexpected behaviour on CPU)

:paperclip: Customize character json

Put the json of your character in the folder char_json in the format:

{
"char_name":...,
"char_persona":...,
"char_greeting":...,
"world_scenario":...,
"example_dialogue":"<START>\nYou:... \nCharName:...\n..."
}

:camera: Customize game appearance

In your RenAIChat folder with the Renpy files (so the other zip file RenAIChat-version-pc/mac), add a transparent sprite of your character with the filename char.png and add a background named bg.png in the folder game/images. Don’t forget to resize your BG to 1920x1080 and your character to the size you want (without being more than 1920x1080).

Python installation

❓Installation

• Clone the repository or download the latest release (source code.zip)
• Go to the project folder with your favorite IDE
• Be sure to have Python installed (3.8 or 3.9), it is not tested and functional before 3.7 and after 3.10.

To setup all the libraries:

• Run these commands in a terminal opened within the project folder to install the packages:

  pip install -r requirements.txt
  

• If you have a CUDA compatible GPU, run this too:

  pip install --force torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu117
  

• If there was an error returned during the installation of the packages, delete the corresponding line in requirements.txt and dowload the package concerned manually
• To download TTS (with Coqui AI TTS), run these commands:

  git clone https://github.com/coqui-ai/TTS
  cd TTS
  pip install -e .
  cd ../
  

• To download TTS (with Tortoise-TTS), run these commands:

  git clone https://github.com/152334H/tortoise-tts-fast
  cd tortoise-tts-fast
  pip install -e .
  cd ../
  

• simpleaudio or other packages might need to install Visual Studio C++ Tools too (see tutorial here), for simpleaudio follow this
• If you want to use Huggingface models with int8, follow these intructions:
  • Download these 2 dll files from here. Move those files in your python packages folder, on Windows it is something like C:\Users\MyName\AppData\Local\Programs\Python\Python39\Lib\site-packages\bitsandbytes
  • Edit bitsandbytes\cuda_setup\main.py:
    • Change ct.cdll.LoadLibrary(binary_path) to ct.cdll.LoadLibrary(str(binary_path)) two times in the file.
    • Replace the this line if not torch.cuda.is_available(): return 'libsbitsandbytes_cpu.so', None, None, None, None with if torch.cuda.is_available(): return 'libbitsandbytes_cuda116.dll', None, None, None, None
• Run the script with python main.py
• For troubleshooting and other issues, don’t hesitate to submit an issue