How to make a singing synth?

[CatsterCatster]

To sum it up, i've always dreamt about making my own vocal synth's for my group of UTAU's, except I don't know where to start???
If anyone could tell me at least one of these things, that would be nice. Thank you!

- Where to start
- What coding language
- What engine
- How to make a voicebank maker (similar to the ones like DeepVocal or VocalSharp)

Again, thank you very much!

 

[lIlI]

That's a huge question that might require consulting with professional programmers for answers! The truth is that every engine is likely made differently, over years of testing, trial and error. There may not be a conclusive solution to your questions, but exploring any developer documentation for programs like DeepVocal and Maghni AI or scholarly papers on vocal synthesis research could be a good place to start. Kanru Hua (of SynthV fame) sometimes tweets about the technical side of vocal synthesis, so you may glean some information there too. Good luck!

 

[Vector]

I have a background in computer science, and I don't even know where to start lol. It's a very cross-disciplinary subject. I'm sure there are prewritten audio processing libraries out there that can do the heavy lifting, but you still need to know what you want to do before you can find tools that can do it, so some background in music theory and how synthesizers work (additive/subtractive synthesis and the physics behind that) would be helpful. I know the term used for Vocaloid and UTAU is "concatenative synthesis," so papers about it will probably have that phrase.

A good place to start is probably looking at utsu: GitHub - titinko/utsu: Vocal synthesis frontend It's an open source project (in Java) attempting to have compatibility with UTAU, which is the simplest style of vocal synth, since it's mainly just pitch-shifting samples without any more complicated modeling like Vocaloid does. OpenUTAU is another project treading in similar territory. GitHub - stakira/OpenUtau: Open source UTAU successor

 

[CatsterCatster]

That's a huge question that might require consulting with professional programmers for answers! The truth is that every engine is likely made differently, over years of testing, trial and error. There may not be a conclusive solution to your questions, but exploring any developer documentation for programs like DeepVocal and Maghni AI or scholarly papers on vocal synthesis research could be a good place to start. Kanru Hua (of SynthV fame) sometimes tweets about the technical side of vocal synthesis, so you may glean some information there too. Good luck!

Thank you!

 

[CatsterCatster]

I have a background in computer science, and I don't even know where to start lol. It's a very cross-disciplinary subject. I'm sure there are prewritten audio processing libraries out there that can do the heavy lifting, but you still need to know what you want to do before you can find tools that can do it, so some background in music theory and how synthesizers work (additive/subtractive synthesis and the physics behind that) would be helpful. I know the term used for Vocaloid and UTAU is "concatenative synthesis," so papers about it will probably have that phrase.

A good place to start is probably looking at utsu: GitHub - titinko/utsu: Vocal synthesis frontend It's an open source project (in Java) attempting to have compatibility with UTAU, which is the simplest style of vocal synth, since it's mainly just pitch-shifting samples without any more complicated modeling like Vocaloid does. OpenUTAU is another project treading in similar territory. GitHub - stakira/OpenUtau: Open source UTAU successor

I'll go check it out, thanks!