Thursday, August 26, 2010

Why are our releases are released with 180MB XviD AVI?

I was asked that by people on the NyaaTorrents channel when I was obtaining my trusted account status when I mentioned I was the encoder.  Back then, I told them I used rather old-fashioned encoding techniques, and they suggested I use h.264 in Matroska with softsubs (as you may know, we hardsub our subtitles).

I had to leave before getting more suggestions because I was in a hurry to get something done, but today is a good opportunity to explain my thinking.

Probably the first thing to explain is the difference between a "container" and a "video codec".

A container is the file format/structure that hold the video, audio, and potentially subtitle data.  For example, AVI, Matroska(MKV), and WMV are containers.

The video codec is how the video data is represented in the container.  h.264, DivX, and XviD are examples of codecs.

What complicates the issue is that a lot of times the container and the codecs get confused with each other, but I'll try not to confuse the issue.

The reason why I chose XviD for the video codec and AVI as the container is for a simple reason. 

COMPATIBILITY WITHOUT EXCESSIVE HASSLES.

To illustrate this, let's talk first about the various popular operating systems and their respective strengths and weaknesses for video playback.

1.  Windows 2000, XP, Vista, 7

This one is easy.  As long as you didn't go all crazy and installed 10 different codec packs, you're golden for playing just about everything.  That includes h.264, XviD, DivX, WMV, MP4, RealMedia, QT, and every exotic combination of softsub/container, not to mention hardsubs.  There is one fly in the ointment, which I'll talk about later.

2.  Linux

If you're using hardsubs, this is almost as easy as Windows.  mplayer, xine, or any other player that uses the ffmpeg libraries will play almost everything.  Unfortunately, this is where WMV falls flat.  ffmpeg tends to decode WMVs rather poorly (what you'll see is subtly splotchy video that is just noticeable to be annoying).  You can work around this if you installed a 32-bit distribution by using binary codecs, but if you're running a 64-bit distribution, you're stuck with ffmpeg.

If you're using softsubs, Linux also falls flat for anything more complex than regular SRT/SSA.  None of your fancy softsubs that you generated using Aegisub will show up correctly.

3.  Mac OS X

The situation is more dire.  You can forget about softsubs.  MKVs and WMVs are at best troublematic and at most crashprone.  (You need to install Flip4Mac to play WMVs in Mac OS X.)  MKVs in particular do not behave well in Mac OS X because of the weird buffering it does in Quicktime Player, and it is not immediately obvious how to switch audio/subtitle streams in an MKV file.

However, AVI with a DivX variant as the codec plays well as long as you have the Perion package.  (Perion is the Mac OS X equivalent of ffdshow in Windows.)

Now, the next thing is "why did I pick XviD over DivX or h.264?"

First thing, DivX sucks.  You can ask any encoder in the anime fansubbing community and they'll give you oodles of reasons why.  I don't need to rehash them here.

Now, it comes down to XviD versus h.264.  What will happen is that I will get some e-mail from someone saying, "I've got a computer with a Pentium 166 MMX processor/PowerPC G3 and your h.264 file plays like crap!"  XviD does not have such stringent CPU requirements.

Therefore, to make sure everyone can play the files, that's why I use the AVI container and the XviD codec.  For those of you wanting softsubs and MKVs, maybe in another couple of years when Apple starts natively supporting MKVs correctly and does softsubs correctly, then I'll use it.

Now, the objections: What about VLC? What about MPlayer Extended (for Mac OS X)?

VLC, in my opinion, is only good for playing Video CDs and taking frame captures.  It's clunky enough to hard crash a Windows 2000 install in a virtual machine, and it has insanely high CPU usage.  It tries to do too much with the wrong tools for the job.

MPlayer Extended on Mac OS X is okay, until you try to stress it.  Once stressed, the child mplayer processes it spawns mysteriously dies for no reason, requiring you to kill it manually using Activity Monitor or using the Terminal (hey, kill -9 does wonders!).

So, that's my rant on encoding.  It's very opinionated, so take it with a grain of salt.

10 comments:

  1. you're wrong about linux, mplayer plays fine 'your fancy softsubs' if you use the ass engine by parsing '-ass' on commandline, otherwise (imho wrong/outdated) default to simple text rendering. my recomendation is add 'ass=yes' to $HOME/.mplayer/config

    ReplyDelete
  2. One thing I have always enjoyed about hardsubs is say I am uploading a short clip to YouTube or something because it is a funny scene, or perhaps making an AMV it translates well with Windows Movie maker.

    MKVs do not do that. Soft subs do not translate and I do not know how to put them into WMM. Plus, when it tries to, it starts creating errors with some kind of CD maker that I need to cancel each time.

    ReplyDelete
  3. nanashi, I ended up looking up that option. I have to admit, mplayer has gone a long way. The one thing that hinders that is Linux' font management is arcane even to experienced users.

    Tyciol, I almost forgot about Windows Media Maker. Programs of their ilk (Final Cut Pro, iMovie, Adobe AfterEffects, Adobe Premiere) are NOT suitable for putting in subtitles of any appreciable amount.

    ReplyDelete
  4. What gets me about encoding is the people that DO use the H264 format with MKV. Often times the H264 encode is 100MBs BIGGER than the Xvid encode at the same size or 300MBs for the size up. It just shows me how shoddy the encoders are, as h264 is the BETTER codec for crunching with quality and thus the file size should be MUCH smaller. (I mean if SeikiS's Porfy no Tabi can get a 1280x720x24min encode crunched down to 170MBs in h264 and it looks like Blu-Ray quality... why can't these people.)
    It's not the codec you use, but how you use it.

    It's also good to see a fansubber state that they are using XVid DUE to the compatibility reasons. Often times people assume that everyone has the same computer, same operating system, etc. Often times I have to tweak the settings in MPC to get the audio or subtitles to appear because someone wanted to try some new fangled technique. I never can seems to get the sliding color karaoke to work on those MKVs so the subs don't even look like how they were intended. In my mind, it should "Just Work". With hard subs, it "Just Work"s.

    Also, thanks for the Licca :D I was stalled at ep 4, for a few years now :3 Can't wait for Doll Isamu (that why I'm watching it...)

    ReplyDelete
  5. I've encoded with h.264 before, so I know what it's capable of doing. It's a wonderful standard and the work that goes into the codec is incredible.

    When it comes to the day when I can play h.264 videos easily on the little PowerMac G3 sitting next to my encoding computer, then I'll probably jump to that.

    I don't really worry that much about filesizes per se. I worry more if someone can't play the file because of the file's special requirements.

    ReplyDelete
  6. I can't help but ask: Is the source really that blurry? I don't care about XviD or hardsubs, but I have massive eyestrain when watching these videos.

    I also wouldn't mind even if you released XviD files double the size they are now if there was a video clarity improvement.

    ReplyDelete
  7. Yeah, the raws aren't exactly the best. I got these raws around 2005 or so. The fact that they're WMV raws doesn't help matters.

    For episode 13, I actually did experiment with an increase in size, and it didn't gain me much. I may try again once we hit the episodes with the third ending song.

    Episodes 40-52 look considerably better than the earlier episodes, though.

    ReplyDelete
  8. I'm late to this discussion, but here are a few thoughts:

    1. The output devices for video playback have changed rather quickly and dramatically. Many/most are now looking to enjoy their favorite videos on large screens also, instead of only their pc or laptop. Softsubs can be enlarged or reduced for best playback. Hardsubs that looked fine on my laptop are often unreadably small on my big screen TV.

    2. The mac/pc chiasm may be a bridge too far to strive for. Not long ago Mac decided to even exclude flash players on their ipads and iphones, even though flash dominates YouTube. They may relent, but clearly Jobs is looking to take his products in a different direction. But flash and quicktime are both doomed. Today’s discussion is 480p or 720p or 1080p. Again, larger screens have rendered lower resolutions untenable.

    Fun fact: I can buy a very very large screen LCD home TV for less than half of what I paid for a very small screen CRT some 6 years ago. But it was a really good professional CRT. But it is now collecting dust in my garage.


    But fansubbing should be about doing what you enjoy, so do whatever that is and thanks for sharing.

    ReplyDelete
  9. Ahh... Please please please, seriously consider using softsubs (.ass in mkv) for future releases. There are zero problems with softsubs on linux (using mplayer) or on Mac (using mplayer extended). The required fonts are automatically used by mplayer if they are embedded in the .mkv, so font management is not an issue at all nowadays.

    With softsubs, you can change the font to anything you like. You can turn them off if you want to practice Japanese. You can take the files and translate the show into another language. You can even make a hardsubbed file from softsubbed file, if you need it for some obsolete hardware. But if you have hardsubs to begin with, you can do nothing...

    You do excellent job bringing us wonderful old shows, not available anywhere else! Thanks a lot for that! But, hardsubs... this make me sad... :(

    ReplyDelete
  10. Rereading this page was valuable, I quickly already forgot there was a difference between codecs and containers, lol. I bet I'd remember it better if I tried subbing something.

    It makes me wonder in regard to Anon's comment, if there would be a way if a group had a soft-subbed release to easily turn it into a hard-sub version on people's computers, which would be valuable for people who want to add something with subtitles to YT (I know WMM sucks for encoding, what I meant was something that already had hardsubs, I have no idea how you could actually add subs to anything using that).

    Based on Mav's rant, I am sort of raging now at the giant files I have tolerated from some groups... makes me think "I should get Licca's work, and maybe by the time I go back to look at other group's they will have aquired their superior compression techniques!"

    Besides Licca, I was wondering if you guys knew the names of any other groups who also use these superior compression techniques for small high-quality visuals for files?

    ReplyDelete