UltraSinger
⚠️ This project is still under development!
UltraSinger is a tool to automatically create UltraStar.txt, midi and notes from music.
It automatically pitches UltraStar files, adding text and tapping to UltraStar files and creates separate UltraStar karaoke files.
It also can re-pitch current UltraStar files and calculates the possible in-game score.
Multiple AI models are used to extract text from the voice and to determine the pitch.
Please mention UltraSinger in your UltraStar.txt file if you use it. It helps others find this tool, and it helps this tool get improved and maintained.
You should only use it on Creative Commons licensed songs.
❤️ Support
There are many ways to support this project. Starring ⭐️ the repo is just one 🙏
You can also support this work on GitHub sponsors or Patreon or Buy Me a Coffee.
This will help me a lot to keep this project alive and improve it.
Table of Contents
💻 How to use this source code
Installation
- Install Python 3.12 (older/newer versions may have compatibility issues). Download
- Also download or install ffmpeg with PATH. Download
- Go to folder
install and run install script for your OS:
- Choose
GPU if you have an NVIDIA CUDA GPU.
- Choose
CPU if you don’t have an NVIDIA GPU or want CPU-only processing.
Run
- In root folder just run
run_on_windows.bat, run_on_linux.sh or run_on_mac.command to start the app.
- Now you can use the UltraSinger source code with
py UltraSinger.py [opt] [mode] [transcription] [pitcher] [extra]. See How to use for more information.
📖 How to use the App
Not all options working now!
UltraSinger.py [opt] [mode] [transcription] [pitcher] [extra]
[opt]
-h This help text.
-i Ultrastar.txt
audio/video like .mp3, .mp4, .wav, youtube link
-o Output folder
[mode]
## if INPUT is audio ##
default (Full Automatic Mode) - Creates all, depending on command line options
--interactive - Interactive Mode. All options are asked at runtime for easier configuration
# Single file creation selection is in progress, you currently getting all!
(-u Create ultrastar txt file) # In Progress
(-m Create midi file) # In Progress
(-s Create sheet file) # In Progress
## if INPUT is ultrastar.txt ##
default Creates all
[separation]
# Default is htdemucs
--demucs Model name htdemucs|htdemucs_ft|htdemucs_6s|hdemucs_mmi|mdx|mdx_extra|mdx_q|mdx_extra_q >> ((default) is htdemucs)
[transcription]
# Default is whisper
--whisper Multilingual model > tiny|base|small|medium|large-v1|large-v2|large-v3 >> ((default) is large-v2)
English-only model > tiny.en|base.en|small.en|medium.en
--whisper_align_model Use other languages model for Whisper provided from huggingface.co
--language Override the language detected by whisper, does not affect transcription but steps after transcription
--whisper_batch_size Reduce if low on GPU mem >> ((default) is 16)
--whisper_compute_type Change to "int8" if low on GPU mem (may reduce accuracy) >> ((default) is "float16" for cuda devices, "int8" for cpu)
--keep_numbers Numbers will be transcribed as numerics instead of as words
[extra]
--disable_hyphenation Disable word hyphenation. Hyphenation is enabled by default.
--disable_separation Disable track separation. Track separation is enabled by default.
--disable_karaoke Disable creation of karaoke style txt file. Karaoke is enabled by default.
--create_audio_chunks Enable creation of audio chunks. Audio chunks are disabled by default.
--keep_cache Keep cache folder after creation. Cache folder is removed by default.
--plot Enable creation of plots. Plots are disabled by default.
--quantize_to_key Quantize notes to detected musical key. Removes pitch slides and out-of-key notes. >> ((default) is enabled)
--format_version 0.3.0|1.0.0|1.1.0|1.2.0 >> ((default) is 1.2.0)
--musescore_path path to MuseScore executable
--keep_numbers Transcribe numbers as digits and not words
--ffmpeg Path to ffmpeg and ffprobe executable
[yt-dlp]
--cookiefile File name where cookies should be read from
[device]
--force_cpu Force all steps to be processed on CPU.
--force_whisper_cpu Only whisper will be forced to cpu
For standard use, you only need to use [opt]. All other options are optional.
Mode
default (Full Automatic Mode) - Creates all, depending on command line options
–interactive - Interactive Mode. All options are asked at runtime for easier configuration
--interactive
Audio / Video (full automatic)
Local file
-i "input/music.mp3"
Youtube
-i https://www.youtube.com/watch?v=YwNs1Z0qRY0
Note that if you run into a yt-dlp error such as Sign in to confirm you’re not a bot. This helps protect our community (yt-dlp issue) you can follow these steps:
- generate a cookies.txt file with yt-dlp
yt-dlp --cookies cookies.txt --cookies-from-browser firefox
- then pass the cookies.txt to UltraSinger
--cookiefile cookies.txt
UltraStar (re-pitch)
This re-pitch the audio and creates a new txt file.
-i "input/ultrastar.txt"
🗣 Transcriber
Keep in mind that while a larger model is more accurate, it also takes longer to transcribe.
Whisper
For the first test run, use the tiny, to be accurate use the large-v2 model.
-i XYZ --whisper large-v2
Whisper languages
Currently provided default language models are en, fr, de, es, it, ja, zh, nl, uk, pt.
If the language is not in this list, you need to find a phoneme-based ASR model from
🤗 huggingface model hub. It will download automatically.
Example for romanian:
-i XYZ --whisper_align_model "gigant/romanian-wav2vec2"
✍️ Hyphenation
Is on by default. Can also be deactivated if hyphenation does not produce
anything useful. Note that the word is simply split,
without paying attention to whether the separated word really
starts at the place or is heard. To disable:
-i XYZ --disable_hyphenation
👂 Pitcher
Pitching is done with the SwiftF0 model, which is faster and more accurate than CREPE.
SwiftF0 automatically detects pitch frequencies between 46.875 Hz (G1) and 2093.75 Hz (C7).
UltraSinger uses 60hz and 400hz
👄 Separation
The vocals are separated from the audio before they are passed to the models. If problems occur with this,
you have the option to disable this function; in which case the original audio file is used instead.
-i XYZ --disable_separation
🎵 Key Quantization
By default, UltraSinger quantizes all detected notes to the detected musical key of the song. This helps to:
- Remove pitch slides and vocal transitions between notes
- Correct out-of-key notes from pitch detection errors
If you want to keep the raw pitch detection without quantization, set quantize_to_key = False.
For most songs quantization should produces better results.
Sheet Music
For Sheet Music generation you need to have MuseScore installed on your system.
Or provide the path to the MuseScore executable.
-i XYZ --musescore_path "C:/Program Files/MuseScore 4/bin/MuseScore4.exe"
This defines the format version of the UltraStar.txt file. For more info see Official UltraStar format specification.
You can choose between different format versions. The default is 1.2.0.
0.3.0 is the first format version. Use this if you have an old UltraStar program and problems with the newer format.1.0.0 should be supported by the most UltraStar programs. Use this if you have problems with the newest format version1.1.0 is the current format version.1.2.0 is the upcoming format version. It is not finished yet.2.0.0 is the next format version. It is not finished yet.
-i XYZ --format_version 1.2.0
🏆 Ultrastar Score Calculation
The score that the singer in the audio would receive will be measured.
You get 2 scores, simple and accurate. You wonder where the difference is?
Ultrastar is not interested in pitch hights. As long as it is in the pitch range A-G you get one point.
This makes sense for the game, because otherwise men don’t get points for high female voices and women don’t get points
for low male voices. Accurate is the real tone specified in the txt. I had txt files where the pitch was in a range not
singable by humans, but you could still reach the 10k points in the game. The accuracy is important here, because from
this MIDI and sheet are created. And you also want to have accurate files
📟 Use GPU
With a GPU you can speed up the process. Also the quality of the transcription and pitching is better.
You need an NVIDIA CUDA device for this to work. Sorry, there is no CUDA device for macOS.
For GPU support on Windows and Linux, the installation script automatically installs PyTorch with CUDA support.
It is optional (but recommended) to install the CUDA driver for your GPU: see CUDA driver.
Also check your GPU CUDA support. See CUDA support
For manual installation, you can use:
uv pip install --index-url https://download.pytorch.org/whl/cu121 torch torchvision torchaudio
Crashes due to low VRAM
If something crashes because of low VRAM then use a smaller Whisper model.
Whisper needs more than 8GB VRAM in the large model!
You can also force CPU usage with the extra option --force_cpu.
📦 Containerized (Docker or Podman)
See container/README.md
评论