• We're currently having issues with our e-mail system. Anything requiring e-mail validation (2FA, forgotten passwords, etc.) requires to be changed manually at the moment. Please reach out via the Contact Us form if you require any assistance.

Native integration of vocal synthesis in LMMS

SeleDreams

Passionate Fan
Jul 31, 2019
154
23
Hi, I'm posting this because I have a project of integrating natively in LMMS a vocal synthesis tool and I felt like it would be interesting to get some feedback and maybe some other devs interested in the project.

at the moment there are two main way to integrate it that were imagined, the first is to integrate it in an UTAU/VOCALOID kind of manner so by directly modifying lyrics in the midi editor, the second is to do it the alter/ego way

one of the issues I thought about in the alter/ego way to do things is the issue of synchronization to the song as the way it's done makes it hard for the engine to know what note corresponds to what lyric.

but one of the other issues with the vocaloid/utau way to do things is that it will require some modification of core lmms UI elements, and LMMS is planning a complete UI rewrite so it will require to rewrite the UI changes after the complete overhaul.

what do you think of this ? and if you have ideas about how you'd like the workflow to be, some sketches showing how it would look like would be very helpful !

the vocal synth would be new but would base its engine on existing open source technology, the first idea is to base the engine on WORLD as it is freely available under a GPL compatible licence, but more open source engines could get added in the future.
 

kozet

Conlanger
Jul 5, 2019
79
flirora.xyz
Seems like an interesting idea. I wouldn't hardcode the vocal synth functionality into LMMS's code but rather create a plugin. Perhaps you could propose the ability for LMMS's own plugin format to read lyrics to the devs. I wouldn't count on such a proposal going through to the mainstream plugin formats such as VST or LV2, but it seems more likely to be considered for one that's used by only one program.
 

SeleDreams

Passionate Fan
Jul 31, 2019
154
23
Seems like an interesting idea. I wouldn't hardcode the vocal synth functionality into LMMS's code but rather create a plugin. Perhaps you could propose the ability for LMMS's own plugin format to read lyrics to the devs. I wouldn't count on such a proposal going through to the mainstream plugin formats such as VST or LV2, but it seems more likely to be considered for one that's used by only one program.
Yeah I planned for the audio part to use the LMMS plugin format,the main thing that would get hard coded would mostly be having a midi editor that would inherit from the normal one but override some code to add some lyrics editing abilities to it on top and the audio plugins would be able to receive lyrics and stuff like this from the midi
by using overrides through inheritance it would allow to add functionality without modifying the core itself
 

Nezuh

Official Piko Husband
Apr 17, 2018
77
Argentina
www.youtube.com
That looks really really neat
I'd would like to have more to say, but I don't really understand much of the technical aspect of all this lol
But good luck! :D
 

SeleDreams

Passionate Fan
Jul 31, 2019
154
23
Progress, it is now able to call the resampler to generate the voice samples, the lyrics and the tone of the note is at the moment hardcoded but will be easy to adjust
 

SeleDreams

Passionate Fan
Jul 31, 2019
154
23
It now has the ability to generate an audio track that it actually plays back in LMMS, it still needs work though in order to make it actually convenient to use, moreover right now it doesn't allow to edit lyrics, it doesn't use the wavtool yet (so no crossfade between lyrics) and the sample track and midi track are separated so the audio goes through an other track
i need to fix these issues
 

SeleDreams

Passionate Fan
Jul 31, 2019
154
23
1614914385262.png
I started working on the UI, now the lyrics show up on the notes (not editable yet) and the pitch bend appears immediately (no automatic vibrato and portamento yet so everything must be hand made)
 
  • Like
Reactions: Nezuh and WyndReed

SeleDreams

Passionate Fan
Jul 31, 2019
154
23
Lyrics can now be edited, due to the way the notes are rendered i couldn't allow to edit the text directly "on" the note but it uses a popup that is just as fast as we can just press enter so it won't disturb the workflow
 

SeleDreams

Passionate Fan
Jul 31, 2019
154
23
So, big improvements, pitch bend is working, and notes can actually be played back in LMMS, it honestly is very fast to render so it renders in real time when drawing the notes
It also supports 3/4 time sigs (and probably others) which isn't normally supported by UTAU
 

SeleDreams

Passionate Fan
Jul 31, 2019
154
23
Progress update :
- Implemented silences so that gaps between notes will be taken in account as silences without the need of rest notes
- Implemented basics of the overlap variables so that note crossfades occur more smoothly based on the oto
- Went back to a single threaded system for the first versions of the utau support due to too many crashes, will be added back once stable enough (no impact on capabilities, more on the fact it will render a little bit more slowly)
 

Users Who Are Viewing This Thread (Users: 0, Guests: 1)