VOCALOID Vocaloid Malfunction in Performance

[henchmiku]

Hello everyone!

I hope everyone's in good health and otherwise keeping well. Some of you may remember me from a few months ago, but just to say again: I'm Jess, a research student in music looking into singing synthesis and how it's reconstructing our ideas about voice and what we can do with it.

I'm writing an article on Hatsune Miku's opera 'The End' at the moment, and I was thinking about this idea that Vocaloids (or any virtual singer tbh) is somehow perfect or limitless in performance, and human performers are much more prone to error. I'm coming round to the idea that this isn't true: technology can break down; code can screw up; things can malfunction unexpectedly. Of course, it's true that Vocaloid doesn't really have the capacity to be performed live/in-real time, and so that's obviously a major difference - there's more of a similarity to film which is edited to "perfection". I know there's a video of Miku "breaking down" at one MikuEXPO performance but I don't know when/where it happened so can't locate the video.

So..this brings me on to my two questions:

1) Have you witnessed (or even better, have a video of) a Vocaloid "breaking down" in live performance? What happened and how were things resolved?
2) If you're a singing synth user, have you ever had problems with the software which meant that the voice did weird things you didn't expect it to? (again recordings would be amazing if you have them).

Please get in touch if you can help me with these questions, and if you're happy to be interviewed over message/email and be part of this research.

Thanks everyone! Also, sorry if this isn't the right place to be posting...am happy to be redirected if so!

Jess

 

[EPROM]

–––

 

[Stardust]

Unfortunately I can't help with the first one. Like EPROM said I believe live performance vocals are per-rendered.
I can help a bit with the second one. A lot of voicebanks can have weird errors. A well-known one is Gumi English's odd pronunciations, as well as choppy transition between words or a word divided into three notes.
An example of the first one would be her pronunciation of tossing. I don't know if you can upload WAVs or any sort of sound formats, so I can't show you, nor do I have a youtube channel to upload to, but I was testing her a bit to get used to her phonemes and noticed she couldn't say the "toss" part of tossing correctly without phoneme editing. Mainly the letter T. Mispronounciations are common in voicebanks and usually easily fixable, but I've never encountered an issue where the voicebank could not pronounce it properly at all until this.
For the second, say you put in "memory", which ends up being divided into three different notes. She'll end up having a weird stop in the midst of saying this, causing it to sound unnatural.

Tonio can also crash an editor if the phoneme "@" is used on it's own. If used as part of the pronunciation of a word it's fine, but usage of a note with only that phoneme causes the crash.

Sonika is also another notoriously glitchy voicebank, but strangely I didn't have many issues using her when I did. So I don't know what examples I can show. I would recommend looking at the vocaloid wiki and seeing the voicebank notes for any vocaloid and it's releases to see if there are any other voicebanks with glitches or errors. Other vocal synth wikis you would want to look into would be SynthV and CeVIO.

I hope I was able to help.

Edit: Grammar and additional info

 

[static-oceans]

Oh this is a fun topic! Feel free to message me if you wanna have a 1 on 1 discussion about it! I can give you samples of what i can provide as well.

i'll reiterate a lot of what stardust already said. Firstly, though most vocaloids have a quirk about them, more often than not, with a little time and care, can be worked around.
The allure to vocaloid to me, is not that they can sound like humans, but that they can do things that human's can't, just like the difference between a synthesizer and a piano. You want to know if vocaloids do things you don't expect them to, and the short answer is yes. every time you sit down for a tuning session you're most likely going to run into something weird.

Gumi English has Transition problems. If you've heard any popular song using her, you may not have noticed, and that's because the producers put a lot of work into fixing them. with most of her vowel to vowel transitions, and some of her consonant to vowel transitions, there's a cutoff between them, making her sound choppy. i've heard different reasons as to why. People either say it's because she was coded wrong or that her voice provider recorded her vowels with glottal stops at the end. this video is meant to help teach you a couple of the work arounds of those problems, but it's also a great example.

Transition Problems aren't just a Gumi english problem, however, I've experienced chops with Yohioloid english, Both of Megurine Luka v2's voicebanks, Lily v2, and countless others. it's just something most vocaloid users just live with. transition problems can act in the opposite way as well, where, instead of completely cutting off the transition, the transition is too short and results in one of the phonemes being eaten up and becoming barely audible. a good example i have to deal with all the time, is with my fav, Avanna, who has problems with her [v] phoneme (which is a v sound). for example, if I make her sing the word "love" straight out of the dictionary phonetic spelling vocaloid gives you. she'll say "luh" instead, meaning that the transition between her [V] phoneme (the 'uh' sound, and her [v] phoneme is not configured properly.

With the Tonio [@] phoneme problem. Tonio will crash vocaloid if there is an @ phoneme at all. i've heard of people getting away with using it if it's used at a lower register, but it's a risky and unpredictable glitch. There are other phonemes that have an @ symbol in them, like [@r] (an 'er' sound), but it only applies to Tonio's solitary [@]

With Sonika, she has poor recordings, which i may get into later, which causes some of her issues, most of them are in the configuration though. I haven't learned how to use her yet, so i'd take everything i say abt her with a grain of salt as i don't know many of her fixes. There are some Phonemes that sonika will repeat. I forgot which ones, but it's happened to me multiple times so it's definitely a prominent issue. one of the other major problems that she has is that some of her phonemes will be at a higher pitch than they're supposed to, mostly with her consonants, so they'll often sound like they're spliced in like a YTP.

When it comes down to tone, and pronunciation, I think those are extremely subjective, and are chocked up to the recordings rather than a glitch, but i'll mention them anyway since it could be "a weird thing a voicebank does"

a lot of english voicebanks that were recorded with a native japanese voice provider have thick japanese accents. This can be fine in most cases, but in some, makes the vocaloid near incomprehensible. With the Kagamine Rin/Len english voicebanks for example, I haven't heard many examples of them where i could make out what they're saying at all for more than a few words. Now, i'm not knocking accented vocaloids at all, Macne Nana English is one of my fav voicebanks, and is a great example of how to intentionally implement a thick accent. With many voicebanks, it's unintentional, or at least poorly executed and results, imho, in the vocal sounding unnatural and alien. the way Rin and Len sing in english isn't how japanese people sing english. It's just the nature of vocal synthesizers and the way they're recorded i think. you CAN tell what accent they have, but often the way a person would actually sing with that accent irl would not come across the same.

Daina, an English vocaloid voiced by an American, has a pronunciation that i can only describe as "uwu speak" she has very slurry consonants but very harsh vowels that make her such an oddball.

jeez, i'm starting to get a little rambley. sorry for all the grammar mistakes, vvn is really hard to edit text in. but yes! feel free to message me if you wan't to talk about this more! it's a really cool topic.

and about your first question: the only thing that a live performing vocaloid can do glitch wise unintentionally is have the audio or video cut out.

 

[peaches2217]

Man, everyone here has given such amazing, detailed responses! This was a fun thread to read through.

The only contribution I can think to make that hasn’t already been mentioned... Piapro Studio often stutters horribly on first playback, though whether that’s a universal issue or just because my computer sucks I’m not sure.

​

 

[Scarlet Illusion]

The only contribution I can think to make that hasn’t already been mentioned... Piapro Studio often stutters horribly on first playback, though whether that’s a universal issue or just because my computer sucks I’m not sure.

I think it’s universal—it does that even on my powerful desktop. It’s incredibly irritating sometimes, lolol.

 

[Pewe]

Daina, an English vocaloid voiced by an American, has a pronunciation that i can only describe as "uwu speak" she has very slurry consonants but very harsh vowels that make her such an oddball.

oh God now I can’t un hear it

 

[Nezuh]

[...]I know there's a video of Miku "breaking down" at one MikuEXPO performance but I don't know when/where it happened so can't locate the video.[...]

Just for completion's sake, I would like to add that there were at least three malfunctions at live concerts.
The first and best known (I think), is when in the middle of Mikupa 2013, which was her longest concert with 39 songs, her voice was not heard at all (and you can even hear wire noises) during the usual mid-concert greeting (before Po Pi Po in this case). Luckily they fixed the problem right at the beginning of the song (I suppose they were different tracks) but nevertheless the guide track for the musicians could be heard for a few seconds.

The second was during 39 (San kyuu) at Miku Expo North America 2016.
Miku sang ahead for more than a minute.

The third is the oldest and most embarrassing imo, due to the moment it happened
At the end of the Mikupa 2012 Hong Kong concert, during the final seconds of Tell Your World (arguably the most emotional part and most "connection" with the audience), Miku suddenly begins to sing "Look this way, baby" (Kocchi muite, baby ), confusing everyone. Luckily the public went along nicely


There may be more at specific concerts without live broadcast, but I have no knowledge of them.
In addition to these, there are the mistakes on purpose. Like when Miku gets emotional and has to start "When the First Love Ends" twice, during the controversial Mikupa 2011.
(during the first minute)

 

[henchmiku]

Man, everyone here has given such amazing, detailed responses! This was a fun thread to read through.

The only contribution I can think to make that hasn’t already been mentioned... Piapro Studio often stutters horribly on first playback, though whether that’s a universal issue or just because my computer sucks I’m not sure.

​

Hahaa thanks for this! I actually really like that sound clip: I know some experimental musicians that would really dig it XD

 

[henchmiku]

Just for completion's sake, I would like to add that there were at least three malfunctions at live concerts.
The first and best known (I think), is when in the middle of Mikupa 2013, which was her longest concert with 39 songs, her voice was not heard at all (and you can even hear wire noises) during the usual mid-concert greeting (before Po Pi Po in this case). Luckily they fixed the problem right at the beginning of the song (I suppose they were different tracks) but nevertheless the guide track for the musicians could be heard for a few seconds.

The second was during 39 (San kyuu) at Miku Expo North America 2016.
Miku sang ahead for more than a minute.

The third is the oldest and most embarrassing imo, due to the moment it happened
At the end of the Mikupa 2012 Hong Kong concert, during the final seconds of Tell Your World (arguably the most emotional part and most "connection" with the audience), Miku suddenly begins to sing "Look this way, baby" (Kocchi muite, baby ), confusing everyone. Luckily the public went along nicely


There may be more at specific concerts without live broadcast, but I have no knowledge of them.
In addition to these, there are the mistakes on purpose. Like when Miku gets emotional and has to start "When the First Love Ends" twice, during the controversial Mikupa 2011.
(during the first minute)

Amazing, thank you. This was exactly what I was looking for. I guess the audience are cheering for Miku to metaphorically "pull herself together" ? Interestingly, in the first video it just sort of looks like she's mouthing for the audience to shout louder and when they are pumped up enough, then she starts to sing.

 

[henchmiku]

Oh this is a fun topic! Feel free to message me if you wanna have a 1 on 1 discussion about it! I can give you samples of what i can provide as well.

i'll reiterate a lot of what stardust already said. Firstly, though most vocaloids have a quirk about them, more often than not, with a little time and care, can be worked around.
The allure to vocaloid to me, is not that they can sound like humans, but that they can do things that human's can't, just like the difference between a synthesizer and a piano. You want to know if vocaloids do things you don't expect them to, and the short answer is yes. every time you sit down for a tuning session you're most likely going to run into something weird.

Gumi English has Transition problems. If you've heard any popular song using her, you may not have noticed, and that's because the producers put a lot of work into fixing them. with most of her vowel to vowel transitions, and some of her consonant to vowel transitions, there's a cutoff between them, making her sound choppy. i've heard different reasons as to why. People either say it's because she was coded wrong or that her voice provider recorded her vowels with glottal stops at the end. this video is meant to help teach you a couple of the work arounds of those problems, but it's also a great example.

Transition Problems aren't just a Gumi english problem, however, I've experienced chops with Yohioloid english, Both of Megurine Luka v2's voicebanks, Lily v2, and countless others. it's just something most vocaloid users just live with. transition problems can act in the opposite way as well, where, instead of completely cutting off the transition, the transition is too short and results in one of the phonemes being eaten up and becoming barely audible. a good example i have to deal with all the time, is with my fav, Avanna, who has problems with her [v] phoneme (which is a v sound). for example, if I make her sing the word "love" straight out of the dictionary phonetic spelling vocaloid gives you. she'll say "luh" instead, meaning that the transition between her [V] phoneme (the 'uh' sound, and her [v] phoneme is not configured properly.

With the Tonio [@] phoneme problem. Tonio will crash vocaloid if there is an @ phoneme at all. i've heard of people getting away with using it if it's used at a lower register, but it's a risky and unpredictable glitch. There are other phonemes that have an @ symbol in them, like [@r] (an 'er' sound), but it only applies to Tonio's solitary [@]

With Sonika, she has poor recordings, which i may get into later, which causes some of her issues, most of them are in the configuration though. I haven't learned how to use her yet, so i'd take everything i say abt her with a grain of salt as i don't know many of her fixes. There are some Phonemes that sonika will repeat. I forgot which ones, but it's happened to me multiple times so it's definitely a prominent issue. one of the other major problems that she has is that some of her phonemes will be at a higher pitch than they're supposed to, mostly with her consonants, so they'll often sound like they're spliced in like a YTP.

When it comes down to tone, and pronunciation, I think those are extremely subjective, and are chocked up to the recordings rather than a glitch, but i'll mention them anyway since it could be "a weird thing a voicebank does"

a lot of english voicebanks that were recorded with a native japanese voice provider have thick japanese accents. This can be fine in most cases, but in some, makes the vocaloid near incomprehensible. With the Kagamine Rin/Len english voicebanks for example, I haven't heard many examples of them where i could make out what they're saying at all for more than a few words. Now, i'm not knocking accented vocaloids at all, Macne Nana English is one of my fav voicebanks, and is a great example of how to intentionally implement a thick accent. With many voicebanks, it's unintentional, or at least poorly executed and results, imho, in the vocal sounding unnatural and alien. the way Rin and Len sing in english isn't how japanese people sing english. It's just the nature of vocal synthesizers and the way they're recorded i think. you CAN tell what accent they have, but often the way a person would actually sing with that accent irl would not come across the same.

Daina, an English vocaloid voiced by an American, has a pronunciation that i can only describe as "uwu speak" she has very slurry consonants but very harsh vowels that make her such an oddball.

jeez, i'm starting to get a little rambley. sorry for all the grammar mistakes, vvn is really hard to edit text in. but yes! feel free to message me if you wan't to talk about this more! it's a really cool topic.

and about your first question: the only thing that a live performing vocaloid can do glitch wise unintentionally is have the audio or video cut out.

Amazing info, thank you! It's intriguing that despite these technical hitches, users will still push through with a particular voicebank because of that voice's quality/timbre/popularity...even accent, as you point out. I wish I understood code so I could understand why this glitches happen.

 

[static-oceans]

Here's a good example of Gumi's problems with a vsqx I was working on a long time ago. The first example is Avanna, the second is Gumi. the vsqx is exactly the same, i just switched the singers.
https://clyp.it/fyoi0sis