I was asked that by people on the NyaaTorrents channel when I was obtaining my trusted account status when I mentioned I was the encoder. Back then, I told them I used rather old-fashioned encoding techniques, and they suggested I use h.264 in Matroska with softsubs (as you may know, we hardsub our subtitles).
I had to leave before getting more suggestions because I was in a hurry to get something done, but today is a good opportunity to explain my thinking.
Probably the first thing to explain is the difference between a "container" and a "video codec".
A container is the file format/structure that hold the video, audio, and potentially subtitle data. For example, AVI, Matroska(MKV), and WMV are containers.
The video codec is how the video data is represented in the container. h.264, DivX, and XviD are examples of codecs.
What complicates the issue is that a lot of times the container and the codecs get confused with each other, but I'll try not to confuse the issue.
The reason why I chose XviD for the video codec and AVI as the container is for a simple reason.
COMPATIBILITY WITHOUT EXCESSIVE HASSLES.
To illustrate this, let's talk first about the various popular operating systems and their respective strengths and weaknesses for video playback.
1. Windows 2000, XP, Vista, 7
This one is easy. As long as you didn't go all crazy and installed 10 different codec packs, you're golden for playing just about everything. That includes h.264, XviD, DivX, WMV, MP4, RealMedia, QT, and every exotic combination of softsub/container, not to mention hardsubs. There is one fly in the ointment, which I'll talk about later.
If you're using hardsubs, this is almost as easy as Windows. mplayer, xine, or any other player that uses the ffmpeg libraries will play almost everything. Unfortunately, this is where WMV falls flat. ffmpeg tends to decode WMVs rather poorly (what you'll see is subtly splotchy video that is just noticeable to be annoying). You can work around this if you installed a 32-bit distribution by using binary codecs, but if you're running a 64-bit distribution, you're stuck with ffmpeg.
If you're using softsubs, Linux also falls flat for anything more complex than regular SRT/SSA. None of your fancy softsubs that you generated using Aegisub will show up correctly.
3. Mac OS X
The situation is more dire. You can forget about softsubs. MKVs and WMVs are at best troublematic and at most crashprone. (You need to install Flip4Mac to play WMVs in Mac OS X.) MKVs in particular do not behave well in Mac OS X because of the weird buffering it does in Quicktime Player, and it is not immediately obvious how to switch audio/subtitle streams in an MKV file.
However, AVI with a DivX variant as the codec plays well as long as you have the Perion package. (Perion is the Mac OS X equivalent of ffdshow in Windows.)
Now, the next thing is "why did I pick XviD over DivX or h.264?"
First thing, DivX sucks. You can ask any encoder in the anime fansubbing community and they'll give you oodles of reasons why. I don't need to rehash them here.
Now, it comes down to XviD versus h.264. What will happen is that I will get some e-mail from someone saying, "I've got a computer with a Pentium 166 MMX processor/PowerPC G3 and your h.264 file plays like crap!" XviD does not have such stringent CPU requirements.
Therefore, to make sure everyone can play the files, that's why I use the AVI container and the XviD codec. For those of you wanting softsubs and MKVs, maybe in another couple of years when Apple starts natively supporting MKVs correctly and does softsubs correctly, then I'll use it.
Now, the objections: What about VLC? What about MPlayer Extended (for Mac OS X)?
VLC, in my opinion, is only good for playing Video CDs and taking frame captures. It's clunky enough to hard crash a Windows 2000 install in a virtual machine, and it has insanely high CPU usage. It tries to do too much with the wrong tools for the job.
MPlayer Extended on Mac OS X is okay, until you try to stress it. Once stressed, the child mplayer processes it spawns mysteriously dies for no reason, requiring you to kill it manually using Activity Monitor or using the Terminal (hey, kill -9 does wonders!).
So, that's my rant on encoding. It's very opinionated, so take it with a grain of salt.