Native integration of vocal synthesis in LMMS

[SeleDreams]

Hi, I'm posting this because I have a project of integrating natively in LMMS a vocal synthesis tool and I felt like it would be interesting to get some feedback and maybe some other devs interested in the project.

at the moment there are two main way to integrate it that were imagined, the first is to integrate it in an UTAU/VOCALOID kind of manner so by directly modifying lyrics in the midi editor, the second is to do it the alter/ego way

one of the issues I thought about in the alter/ego way to do things is the issue of synchronization to the song as the way it's done makes it hard for the engine to know what note corresponds to what lyric.

but one of the other issues with the vocaloid/utau way to do things is that it will require some modification of core lmms UI elements, and LMMS is planning a complete UI rewrite so it will require to rewrite the UI changes after the complete overhaul.

what do you think of this ? and if you have ideas about how you'd like the workflow to be, some sketches showing how it would look like would be very helpful !

the vocal synth would be new but would base its engine on existing open source technology, the first idea is to base the engine on WORLD as it is freely available under a GPL compatible licence, but more open source engines could get added in the future.

 

[kozet]

Seems like an interesting idea. I wouldn't hardcode the vocal synth functionality into LMMS's code but rather create a plugin. Perhaps you could propose the ability for LMMS's own plugin format to read lyrics to the devs. I wouldn't count on such a proposal going through to the mainstream plugin formats such as VST or LV2, but it seems more likely to be considered for one that's used by only one program.

 

[SeleDreams]

Seems like an interesting idea. I wouldn't hardcode the vocal synth functionality into LMMS's code but rather create a plugin. Perhaps you could propose the ability for LMMS's own plugin format to read lyrics to the devs. I wouldn't count on such a proposal going through to the mainstream plugin formats such as VST or LV2, but it seems more likely to be considered for one that's used by only one program.

Yeah I planned for the audio part to use the LMMS plugin format,the main thing that would get hard coded would mostly be having a midi editor that would inherit from the normal one but override some code to add some lyrics editing abilities to it on top and the audio plugins would be able to receive lyrics and stuff like this from the midi
by using overrides through inheritance it would allow to add functionality without modifying the core itself

 

[SeleDreams]

It's been a while since I talked about it, I started to work on it recently, this is the repository SeleDreams/lmms at utau_integration (github.com)
atm i've setup the skeleton for me to start working on the code

 

[SeleDreams]

progress report
Voicebank parsing with oto.ini is integrated, there is no ability yet to produce sound but it's coming

 

[Nezuh]

That looks really really neat
I'd would like to have more to say, but I don't really understand much of the technical aspect of all this lol
But good luck! :D

 

[SeleDreams]

Progress, it is now able to call the resampler to generate the voice samples, the lyrics and the tone of the note is at the moment hardcoded but will be easy to adjust

 

[SeleDreams]

It now has the ability to generate an audio track that it actually plays back in LMMS, it still needs work though in order to make it actually convenient to use, moreover right now it doesn't allow to edit lyrics, it doesn't use the wavtool yet (so no crossfade between lyrics) and the sample track and midi track are separated so the audio goes through an other track
i need to fix these issues

 

[SeleDreams]

I started working on the UI, now the lyrics show up on the notes (not editable yet) and the pitch bend appears immediately (no automatic vibrato and portamento yet so everything must be hand made)

 

[SeleDreams]

Lyrics can now be edited, due to the way the notes are rendered i couldn't allow to edit the text directly "on" the note but it uses a popup that is just as fast as we can just press enter so it won't disturb the workflow

 

[SeleDreams]

So, big improvements, pitch bend is working, and notes can actually be played back in LMMS, it honestly is very fast to render so it renders in real time when drawing the notes
It also supports 3/4 time sigs (and probably others) which isn't normally supported by UTAU

 

[SeleDreams]

for those who are interested I am searching people who would provide the voice of the default voicebanks bundled with lmms

Microsoft Forms

forms.office.com

 

[SeleDreams]

Progress update :
- Implemented silences so that gaps between notes will be taken in account as silences without the need of rest notes
- Implemented basics of the overlap variables so that note crossfades occur more smoothly based on the oto
- Went back to a single threaded system for the first versions of the utau support due to too many crashes, will be added back once stable enough (no impact on capabilities, more on the fact it will render a little bit more slowly)