Media preservation thread

[Grzesiek11]

I will be posting about my archival projects and their status here. The thread will probably mostly be just me - it could've been in the journals section, but I would like to encourage discussion if anyone has anything to say, so I'm making it in the community section instead. While Thoughts on Vocal Synth Music Preservation was about preservation, this will just be my notes, hopefully with some input from others.

So far, this list is incomplete. I plan on improving upon it when I have the time and will.

Projects

ニコニコ

The home of this culture we all know and love. I wouldn't consider it endangered as a whole, though the recent-ish ransomware attack was concerning (we lost some non-video data in it), and it might also not be doing the best financially. What is a real risk though is individual videos and user accounts getting removed.

Scope:

• All from tags: VOCALOID殿堂入り, VOCALOID伝説入り, VOCALOID神話入り, UTAU殿堂入り, UTAU伝説入り, SynthesizerV殿堂入り, SynthesizerV伝説入り, CeVIO殿堂入り伝, CeVIO伝説入り伝, NEUTRINO殿堂入り, NEUTRINO伝説入り, VOICEROID殿堂入り入, VOICEROID伝説入り.
• All from users: 鼻そうめんP / かんざきひろ, daniwell, MARETU, ギガ, kemu, 名無し, Mitchie M, Omoi, ドミノ･ピザチャンネル, cosMo＠暴走P, ゴジマジＰ, GYARI（ココアシガレットP）, samfree, 阿修, 神様うさぎ, 23.exe, ピノキオピー, doriko (this assortment of users is quite random, I mostly just put in whoever I remember or feel like is at risk of deletion).
• Early works: entirety of the tag VOCALOID from most of 2009 and before (missing some of December 2009 due to ニコニコ being annoying to work with).
• Random videos I feel are historically significant, or important to me or others.
• Suggestions on expanding the scope in any way are welcome.

Status: Indefinitely ongoing. New videos within scope are saved once in a while.

FC2WEB

Free Japanese web host used by many vocalo-P. It was supposed to cease operation on 2025-06-30, but so far hasn't.

Scope:

• All sites found linked in: UTAU Fandom wiki, UTAU Wikidot wiki, Vocaloid Fandom wiki, Vocaloid Lyrics Wiki, UTAU @wiki wiki, Hatsune Miku @wiki wiki, VocaDB, descriptions of videos from the ニコニコ project.
• Certain sites found in other places.
• Any new site submissions are welcome, even if you think they might be duplicate. Please submit any URL with domain ending with .web.fc2.com. .blog[n].fc2.com sites are not endangered right now but you may also submit them for future use.

Status: Ongoing. Everything that was planned to be included is done, but there are definitely sites not yet saved, and the host is still up.

Tools

• yt-dlp - Multi-site video downloader.
  Personally recommend it for YouTube, but not really niconico (though it usually works). niconico support is not a priority for yt-dlp and it had longstanding issues with the site in the past. Has mass download capabilities, though they can be lacking in some areas.
• nndownload - niconico video downloader.
  This is what I use for niconico, though it doesn't have many mass download capabilities. See nnscrape for that.
• gallery-dl - Multi-site image downloader.
  Can be used for Pixiv and Twitter. Haven't tried it myself yet.
• belweder - Accurate web downloader meant as a modern and hackable replacement for httrack.
• nnscrape - niconico mass downloader based on nndownload.
• piascrape - Piapro mass downloader.
• rclone - Frontend for a lot of cloud storage providers.
  Useful for getting files off public Google Drive links with some hassle, even though it's not really designed for that.

 

[MagicalMiku]

Media preservation is always good, and I like to save not only the videos (from both niconico and youtube), but also artworks of my favorite artists from sites like piapro, pixiv and twitter. I bought various external usb hard drives (WD Book 5TB, 6TB, 8TB) just for that hehe
I like to keep track of what to save in different .txt files
and it feels good, really good to save what I love. Because is not rare that some videos, some songs or some artworks and webpages are not available anymore after some time.

Here are the tools I use:
1: yt-dlp: to save videos and entire channels (use it together with ffmpeg to have lots of formats saving options available).
https://github.com/yt-dlp/yt-dlp
https://ffmpeg.org/
2: gallery-dl: if the pixiv/twitter accounts have many many pictures, this tool is very good.
https://github.com/mikf/gallery-dl
3: for piapro artworks and songs, I enjoy to save them manually, I like to spend some time during the week doing that, it is relaxing and nice for me to see all the beautiful works from the past years. I completed saving all Miku piapro artworks and songs from piapro. Next planned are all the piapro artworks/songs contest and the other piapro characters artworks.
4: to save webpages, I usually save them both "Save as.." in html with Firefox and also print them as PDF, it works very well.
5: I also like to save the trials and the full installers of all the Vocaloid I own
6: I use VocaDB as a wiki reference for producers' official pages/accounts:
https://vocadb.net/
7: I always save an offline copy of my Miku apps on Android (export as APK files, save into my pc) and all my Steam and GOG games.
8: for Project Diva Mega39s (PC/Switch) extra songs and modules (made by users, not official), I check the gamebanana pages:
https://gamebanana.com/games/16522
https://gamebanana.com/games/7886
9: all the CD albums and DVD/Blu-Ray concerts I have, I always make a copy of them into my pc. The CD albums I save both in FLAC and MP3 with freac, the DVD/Blu-Ray I make a MP4 video with Handbrake with custom made presets for good balance of quality/size:
https://www.freac.org/
https://handbrake.fr/
10: for media tag editor, I use Mp3tag, and MediaInfo for checking all the codec information of a media file:
https://www.mp3tag.de/en/
https://mediaarea.net/en/MediaInfo

 

[Blue Of Mind]

Don't forget about the Wayback Machine! Admittedly, the Internet Archive in general is now under threat from copyright holders, but the WM is still a godsend for archiving old pages (so long as a robots.txt script isn't on a site).

I've tried my best to save songs and post them to YouTube in the past, but obviously I can't keep up with every last corner of the Vocaloid fandom (and every so often, I've had copyright enforcements and tangles mess with my channel).

 

[Grzesiek11]

YouTube is seemingly getting rid of SRV3 subtitles (which is most of the ones with custom styling). This means songs like Aishite Aishite Aishite are going to have mostly-plain subtitles soon.

Yesterday I wrote a tool and started a project for these songs, while I'm busy with schoolwork, I wanted to at least do something. If you know people who have used styled subtitles, please let me know, either here or in DMs, include links (can be entire channels or playlists) if possible. I'm also doing some J-music people that aren't strictly vocaloid producers, like Nanawoakari or Eve, so feel free to submit songs like that too.

I think there are already people saving Vtuber stuff, supposedly they use styled subs a lot too.

Source

 

[Bookworm2]

YouTube is seemingly getting rid of VTT3 subtitles (which is most of the ones with custom styling). This means songs like Aishite Aishite Aishite are going to have mostly-plain subtitles soon.

Yesterday I wrote a tool and started a project for these songs, while I'm busy with schoolwork, I wanted to at least do something. If you know people who have used styled subtitles, please let me know, either here or in DMs, include links (can be entire channels or playlists) if possible. I'm also doing some J-music people that aren't strictly vocaloid producers, like Nanawoakari or Eve, so feel free to submit songs like that too.

I think there are already people saving Vtuber stuff, supposedly they use styled subs a lot too.

Source

Oh no! If I'm not mistaken, I believe Copycat by CircusP uses custom subtitles for coloring and other effects, such as Karaoke for the lyrics. Here's the link:

 

[Grzesiek11]

Yeah! I've got Circus covered. Actually, my bad on not saying who I got covered...

Here's a list by YouTube handles:

sheeno3rin, kikuo_sound, manbo-p, HatsuneMiku, GYARI, VocaCircus, JamiePaigeIRL, FLAVORFOLEY, NAKISO, pinocchiop, 23dtmJP, LonePi, shikiura

Also, the following non-vocaloid channels:

ooo0eve0ooo, nanaoakariSMEJ, takayan_gorizal

Kesera contributed their own subtitles, which they made for a bunch of producers. Also, I'm working on getting every song tagged stylized captions on VocaDB (507 entries), thanks to the suggestion of Mafuyu from Vocaloid Lyrics Wiki.

 

[MagicalMiku]

Good findings!
Also, I think with yt-dlp tool there is a command line option to save all the videos of a channel that have that subtitle format (choosing to save the subtitle as hardcoded in the video, or as a single subtitle file). If not, you could suggest the option in their github page:
https://github.com/yt-dlp/yt-dlp
https://github.com/yt-dlp/yt-dlp?tab=readme-ov-file#subtitle-options

 

[Grzesiek11]

Good findings!
Also, I think with yt-dlp tool there is a command line option to save all the videos of a channel that have that subtitle format (choosing to save the subtitle as hardcoded in the video, or as a single subtitle file). If not, you could suggest the option in their github page:
https://github.com/yt-dlp/yt-dlp
https://github.com/yt-dlp/yt-dlp?tab=readme-ov-file#subtitle-options

My tool is actually based on yt-dlp and requires it to be installed. The issue with using yt-dlp directly is that it cannot save all subtitle formats at once (and I'd rather do that, rather than waiting until they get rid of/break other formats as well, especially as it requires little additional space and effort), this feature was suggested already years ago, but has no progress.

You can write a naive script to do so, but that is very slow and causes ratelimiting, as it sends multiple requests per video. What I did is an approach that is a little less naive, but works much better.

To only save SRV3, you can do the following:

yt-dlp --write-subs --sub-langs all,-live_chat --sub-format srv3 --skip-download urlhere

(Plus the usual --cookies and now-usual JS runtime stuff.)