Google Code-In is the most innovative open-source program I’ve ever seen.  It provided a way for students who had never done open source — or never even done programming — to get involved in open source work.   It made it easy for people who weren’t sure of their ability, who didn’t know whether they could do open source, to get involved and realize that yes, they too could do amazing work — whether code useful to millions of people, documentation to make the code useful, translations to make it accessible, and more.  Hundreds of students had a great experience, learned new things, and many stayed around in open source projects afterwards because they enjoyed it so much!

x264 benefitted greatly from Google Code-In.  Most of the high bit depth assembly code was written through GCI — literally man-weeks of work by an professional developer, done by high-schoolers who had never written assembly before!  Furthermore, we got loads of bugs fixed in ffmpeg/libav, a regression test tool, and more.  And best of all, we gained a new developer: Daniel Kang, who is now a student at MIT, an x264 and libav developer, and has gotten paid work applying the skills he learned in Google Code-In!

Some students in GCI complained about the system being “unfair”.  Task difficulties were inconsistent and there were many ways to game the system to get lots of points.  Some people complained about Daniel — he was completing a staggering number of tasks, so they must be too easy.  Yet many of the other students considered these tasks too hard.  I mean, I’m asking high school students to write hundreds of lines of complicated assembly code in one of the world’s most complicated instruction sets, and optimize it to meet extremely strict code-review standards!  Of course, there may have been valid complaints about other projects: I did hear from many students talking about gaming the system and finding the easiest, most “profitable” tasks.  Though, with the payout capped at $500, the only prize for gaming the system is a high rank on the points list.

According to people at the session, in an effort to make GCI more “fair”, Google has decided to change the system.  There are two big changes they’re making.

Firstly, Google is requiring projects to submit tasks on only two dates: the start, and the halfway point.  But in Google Code-In, we certainly had no idea at the start what types of tasks would be the most popular — or new ideas that came up over time.  Often students would come up with ideas for tasks, which we could then add!  A waterfall-style plan-everything-in-advance model does not work for real-world coding.  The halfway point addition may solve this somewhat, but this is still going to dramatically reduce the number of ideas that can be proposed as tasks.

Secondly, Google is requiring projects to submit at least 5 tasks of each category just to apply.  Quality assurance, translation, documentation, coding, outreach, training, user interface, and research.  For large projects like Gnome, this is easy: they can certainly come up with 5 for each on such a large, general project.  But often for a small, focused project, some of these are completely irrelevant.  This rules out a huge number of smaller projects that just don’t have relevant work in all these categories.  x264 may be saved here: as we work under the Videolan umbrella, we’ll likely be able to fudge enough tasks from Videolan to cover the gaps.  But for hundreds of other organizations, they are going to be out of luck.  It would make more sense to require, say, 5 out of 8 of the categories, to allow some flexibility, while still encouraging interesting non-coding tasks.

For example, what’s “user interface” for a software library with a stable API, say, a libc?  Can you make 5 tasks out of it that are actually useful?

If x264 applied on its own, could you come up with 5 real, meaningful tasks in each category for it?  It might be possible, but it’d require a lot of stretching.

How many smaller or more-focused projects do you think are going to give up and not apply because of this?

Is GCI supposed to be something for everyone, or just or Gnome, KDE, and other megaprojects?

(And yes, I’ll get around to approving the queued comments and writing more real posts.  Eventually!  I promise!)

(Yes, it’s just a menu. But good things start small!)

Over the past couple of months, we’ve gotten an enormous response; over 40 companies have inquired about licensing, with more contacting us every day.  Due to the sheer volume of interest, we’ve partnered with CoreCodec, the creators of the free Matroska container format and developers of CoreAVC, to make x264 as widely available as possible in the world of commercial software as it is in the world of open source.  All of this is already filtering back to benefiting x264 users, with many bugs being reported by commercial licensees as well as some code contributed.

Today, we announce the first commercial consumer encoding software to switch to x264: Pegasys Inc.’s TMPGEnc.  Expect many more to follow: with x264 now available commercially as well as freely, there are few excuses left to use any other H.264 encoder.  Vendors of overpriced, underpowered proprietary competitors should begin looking for new jobs.

(Pegasys press release: English, Japanese)

I still don’t trust them, but it’s possible it’s merely bad research (and thus being unaware of prior art) as opposed to anything malicious.  Furthermore, word from within their office suggests they’re quite possibly being honest: supposedly the development team does not read x264 code at all.  So this might just all be very bad luck.

Regardless, the patent is still complete tripe, and should never have been filed.

Most importantly, stop harassing the guy whose name is on the patent (Lars): he’s just a programmer, not the management or lawyers responsible for filing the patent.  This is stupid and unnecessary.  I’ve removed the original post because of this; it can be found here for those who want to read it.

Appendix: the details of the patent:

I figure I’ll go over the exact correspondence between the patent and my code here.

1. A method for calculating run and level representations of quantized transform coefficients representing pixel values included in a block of a video picture, the method comprising:

Translation: It’s a run-level coder.

packing, at a video processing apparatus, each quantized transform coefficients in a value interval [Max, Min] by setting all quantized transform coefficients greater than Max equal to Max, and all quantized transform coefficients less than Min equal to Min

The quantized coefficients are clipped to a certain valid range to allow them to be packed into bytes (they start as 16-bit values).

reordering, at the video processing apparatus, the quantized transform ID coefficients according to a predefined order depending on respective positions in the block resulting in an array C of reordered quantized transform coefficients

This is the zigzag pattern used in H.264 (and most formats) for reordering DCT coefficients.  In x264, this is done before the run-level coder ste.

masking, at the video processing apparatus, C by generating an array M containing ones in positions corresponding to positions of C having non-zero values, and zeros in positions corresponding to positions of C having zero values

This is creating a bitmask based on the coefficient values, the pmovmskb step.

is generating, at the video processing apparatus, for each position containing a one in M, a run and a level representation by setting the level value equal to an occurring value in a corresponding position of C; and setting, at the video processing apparatus, for each position containing a one in M₅ the run value equal to the number of proceeding positions relative to a current position in M since a previous occurrence of one in M.

This is the process of creating run/level values from the bitmask.

Now into the detailed claims:

2. The method according to Claim 1, wherein the masking further includes, creating an array C from C where positions corresponding to positions of nonzero values in C are filled with ones, and positions corresponding to positions of zero values in C are filled with zeros, and creating M from C by extracting the most significant bit from values in respective position of C and inserting the bits in corresponding positions in M.

They’re extracting the most significant bit of the values to create a bitmask.  This is exactly what the pmovmskb in my algorithm does.

3. The method according to Claim 2, wherein the creating of the array C is executed by a C++ function PCMPGTB, and the creating of M from C is executed by a C++ function PMOVMSKB.

And here they use pcmpgtb (they call it a C++ function for some reason, but it’s a SSE instruction) to do the clipping of the input values.   This is exactly the same method I used in decimate_score.  They also use pmovmskb as mentioned.

4. The method according to Claim 1 , wherein the generating of the run and level representation further includes determining positions containing non-zero values in C by corresponding positions containing ones in M.

5. The method according to Claim 4, wherein the determining of positions containing non-zero values in C is executed by a C++ function BSF.

Here they iterate over the bitmask of transform coefficients using a “BSF” function to find runs, which is exactly what I did.  Of course, BSF isn’t a function, it’s an x86 instruction.

6. The method according to Claim 1 , wherein Max is 256 and Min is 0.

This is almost surely a typo or mistake of some sort.  They mean the Max should be 255, not 256: 256 doesn’t fit in a uint8_t.

7. The method according to Claim 1 , wherein the predefined order follows a zigzag path of transform coefficient positions in the block starting in an upper left corner heading towards a lower right corner.

This is a description of the typical DCT zigzag pattern (like in H.264, MPEG-2, Theora, etc).

Everything after this part is just repeating itself with the phrase “an apparatus” added in order to make the USPTO listen to them.

Do: contact the project using their preferred medium of communication.

Most open source projects use public methods of communication, such as mailing lists and IRC.  It’s not the end of the world if you mistakenly make contact with the wrong people or via the wrong medium, but be prepared to switch to the correct one once informed!  You may not be experienced using whatever form of communication the project uses, but if you refuse to communicate through proper channels, they will likely not be as inclined to assist you.  Larger open source projects are often much like companies in that they have different parts to their organization with different roles.  Don’t assume that everyone is a major developer!

If you don’t know what to do, a good bet is often to just ask someone.

Don’t: contact only one person.

Open source projects are a communal effort.  Major contributions are looked over by multiple developers and are often discussed by the community as a whole.  Yet many companies tend to contact only a single person in lieu of dealing with the project proper.  This has many flaws: to begin with, it forces a single developer (who isn’t paid by you) to act as your liaison, adding yet another layer between what you want and the people you want to talk to.  Contribution to open source projects should not be a game of telephone.

Of course, there are exceptions to this: sometimes a single developer is in charge of the entirety of some particular aspect of a project that you intend to contribute to, in which case this might not be so bad.

Do: make clear exactly what it is you are contributing.

Are you contributing code?  Development resources?  Money?  API documentation?  Make it as clear as possible, from the start!  How developers react, which developers get involved, and their expectations will depend heavily on what they think you are providing.  Make sure their expectations match reality.  Great confusion can result when they do not.

This also applies in the reverse — if there’s something you need from the project, such as support or assistance with development of your patch, make that explicitly clear.

Don’t: code dump.

Code does not have intrinsic value: it is only useful as part of a working, living project.  Most projects react very negatively to large “dumps” of code without associated human resources.  That is, they expect you to work with them to finalize the code until it is ready to be committed.  Of course, it’s better to work with the project from the start: this avoids the situation of writing 50,000 lines of code independently and then finding that half of it needs to be rewritten.  Or, worse, writing an enormous amount of code only to find it completely unnecessary.

Of course, the reverse option — keeping such code to yourself — is often even more costly, as it forces you to maintain the code instead of the official developers.

Do: ignore trolls.

As mentioned above, many projects use public communication methods — which, of course, allow anyone to communicate, by nature of being public.  Not everyone on a project’s IRC or mailing list is necessarily qualified to officially represent the project.  It is not too uncommon for a prospective corporate contributor to be turned off by the uninviting words of someone who isn’t even involved in the project due to assuming that they were.  Make sure you’re dealing with the right people before making conclusions.

Don’t: disappear.

If you are going to try to be involved in a project, you need to stay in contact.  We’ve had all too many companies who simply disappear after the initial introduction.  Some tell us that we’ll need an NDA, then never provide it or send status updates.  You may know why you’re not in contact — political issues at the company, product launch crunches, a nice vacation to the Bahamas — but we don’t!  If you disappear, we will assume that you gave up.

Above all, don’t assume that being at a large successful company makes you immune to these problems.  If anything, these problems seem to be the most common at the largest companies.  I didn’t name any names in this post, but practically every single one of these rules has been violated at some point by companies looking to contribute to x264.  In the larger scale of open source, these problems happen constantly.  Don’t fall into the same traps that many other companies have.

If you’re an open source developer reading this post, remember it next time you see a company acting seemingly nonsensically in an attempt to contribute: it’s quite possible they just don’t know what to do.  And just because they’re doing it wrong doesn’t mean that it isn’t your responsibility to try to help them do it right.

JPEG is a very old lossy image format.  By today’s standards, it’s awful compression-wise: practically every video format since the days of MPEG-2 has been able to tie or beat JPEG at its own game.  The reasons people haven’t switched to something more modern practically always boil down to a simple one — it’s just not worth the hassle.  Even if JPEG can be beaten by a factor of 2, convincing the entire world to change image formats after 20 years is nigh impossible.  Furthermore, JPEG is fast, simple, and practically guaranteed to be free of any intellectual property worries.  It’s been tried before: JPEG-2000 first, then Microsoft’s JPEG XR, both tried to unseat JPEG.  Neither got much of anywhere.

Now Google is trying to dump yet another image format on us, “WebP”.  But really, it’s just a VP8 intra frame.  There are some obvious practical problems with this new image format in comparison to JPEG; it doesn’t even support all of JPEG’s features, let alone many of the much-wanted features JPEG was missing (alpha channel support, lossless support).  It only supports 4:2:0 chroma subsampling, while JPEG can handle 4:2:2 and 4:4:4.  Google doesn’t seem interested in adding any of these features either.

But let’s get to the meat and see how these encoders stack up on compressing still images.  As I explained in my original analysis, VP8 has the advantage of H.264′s intra prediction, which is one of the primary reasons why H.264 has such an advantage in intra compression.  It only has i4x4 and i16x16 modes, not i8x8, so it’s not quite as fancy as H.264′s, but it comes close.

The test files are all around 155KB; download them for the exact filesizes.  For all three, I did a binary search of quality levels to get the file sizes close.  For x264, I encoded with --tune stillimage --preset placebo.  For libvpx, I encoded with --best.  For JPEG, I encoded with ffmpeg, then applied jpgcrush, a lossless jpeg compressor.  I suspect there are better JPEG encoders out there than ffmpeg; if you have one, feel free to test it and post the results.  The source image is the 200th frame of Parkjoy, from derf’s page (fun fact: this video was shot here!  More info on the video here.).

Files: (x264 [154KB], vp8 [155KB], jpg [156KB])

Results (decoded to PNG): (x264, vp8, jpg)

This seems rather embarrassing for libvpx.  Personally I think VP8 looks by far the worst of the bunch, despite JPEG’s blocking.  What’s going on here?  VP8 certainly has better entropy coding than JPEG does (by far!).  It has better intra prediction (JPEG has just DC prediction).  How could VP8 look worse?  Let’s investigate.

VP8 uses a 4×4 transform, which tends to blur and lose more detail than JPEG’s 8×8 transform.  But that alone certainly isn’t enough to create such a dramatic difference.  Let’s investigate a hypothesis — that the problem is that libvpx is optimizing for PSNR and ignoring psychovisual considerations when encoding the image… I’ll encode with --tune psnr --preset placebo in x264, turning off all psy optimizations.  

Files: (x264, optimized for PSNR [154KB]) [Note for the technical people: because adaptive quantization is off, to get the filesize on target I had to use a CQM here.]

Results (decoded to PNG): (x264, optimized for PSNR)

What a blur!  Only somewhat better than VP8, and still worse than JPEG.  And that’s using the same encoder and the same level of analysis — the only thing done differently is dropping the psy optimizations.  Thus we come back to the conclusion I’ve made over and over on this blog — the encoder matters more than the video format, and good psy optimizations are more important than anything else for compression.  libvpx, a much more powerful encoder than ffmpeg’s jpeg encoder, loses because it tries too hard to optimize for PSNR.

These results raise an obvious question — is Google nuts?  I could understand the push for “WebP” if it was better than JPEG.  And sure, technically as a file format it is, and an encoder could be made for it that’s better than JPEG.  But note the word “could”.  Why announce it now when libvpx is still such an awful encoder?  You’d have to be nuts to try to replace JPEG with this blurry mess as-is.  Now, I don’t expect libvpx to be able to compete with x264, the best encoder in the world — but surely it should be able to beat an image format released in 1992?

Earth to Google: make the encoder good first, then promote it as better than the alternatives.  The reverse doesn’t work quite as well.

Addendum (added Oct. 2, 03:51):

maikmerten gave me a Theora-encoded image to compare as well.  Here’s the PNG and the source (155KB).  And yes, that’s Theora 1.2 (Ptalarbvorm) beating VP8 handily.  Now that is embarassing.  Guess what the main new feature of Ptalarbvorm is?  Psy optimizations…

Addendum (added Apr. 20, 23:33):

There’s a new webp encoder out, written from scratch by skal (available in libwebp).  It’s significantly better than libvpx — not like that says much — but it should probably beat JPEG much more readily now.  The encoder design is rather unique — it basically uses K-means for a large part of the encoding process.  It still loses to x264, but that was expected.

We tested on two 1080p clips: Parkjoy, a live-action 1080p clip, and the Sintel trailer, a CGI 1080p clip.  Testing was done using “time ffmpeg -vcodec {libvpx or vp8} -i input -vsync 0 -an -f null -”.  We all used the latest SVN FFmpeg at the time of this posting; the last revision optimizing the VP8 decoder was r24471.

As these benchmarks show, ffvp8 is clearly much faster than libvpx, particularly on 64-bit.  It’s even faster by a large margin on Atom, despite the fact that we haven’t even begun optimizing for it.  In many cases, ffvp8′s extra speed can make the difference between a video that plays and one that doesn’t, especially in modern browsers with software compositing engines taking up a lot of CPU time.  Want to get faster playback of VP8 videos?  The next versions of FFmpeg-based players, like VLC, will include ffvp8.  Want to get faster playback of WebM in your browser?  Lobby your browser developers to use ffvp8 instead of libvpx.  I expect Chrome to switch first, as they already use libavcodec for most of their playback system.

Keep in mind ffvp8 is not “done” — we will continue to improve it and make it faster.  We still have a number of optimizations in the pipeline that aren’t committed yet.

Developing ffvp8

The initial challenge, primarily pioneered by David and Ronald, was constructing the core decoder and making it bit-exact to libvpx.  This was rather challenging, especially given the lack of a real spec.  Many parts of the spec were outright misleading and contradicted libvpx itself.  It didn’t help that the suite of official conformance tests didn’t even cover all the features used by the official encoder!  We’ve already started adding our own conformance tests to deal with this.  But I’ve complained enough in past posts about the lack of a spec; let’s get onto the gritty details.

The next step was adding SIMD assembly for all of the important DSP functions.  VP8′s motion compensation and deblocking filter are by far the most CPU-intensive parts, much the same as in H.264.  Unlike H.264, the deblocking filter relies on a lot of internal saturation steps, which are free in SIMD but costly in a normal C implementation, making the plain C code even slower.  Of course, none of this is a particularly large problem; any sane video decoder has all this stuff in SIMD.

I tutored Ronald in x86 SIMD and wrote most of the motion compensation, intra prediction, and some inverse transforms.  Ronald wrote the rest of the inverse transforms and a bit of the motion compensation.  He also did the most difficult part: the deblocking filter.  Deblocking filters are always a bit difficult because every one is different.  Motion compensation, by comparison, is usually very similar regardless of video format; a 6-tap filter is a 6-tap filter, and most of the variation going on is just the choice of numbers to multiply by.

The biggest challenge in an SIMD deblocking filter is to avoid unpacking, that is, going from 8-bit to 16-bit.  Many operations in deblocking filters would naively appear to require more than 8-bit precision.  A simple example in the case of x86 is abs(a-b), where a and b are 8-bit unsigned integers.  The result of “a-b” requires a 9-bit signed integer (it can be anywhere from -255 to 255), so it can’t fit in 8-bit.  But this is quite possible to do without unpacking: (satsub(a,b) | satsub(b,a)), where “satsub” performs a saturating subtract on the two values.  If the value is positive, it yields the result; if the value is negative, it yields zero.  Oring the two together yields the desired result.  This requires 4 ops on x86; unpacking would probably require at least 10, including the unpack and pack steps.

After the SIMD came optimizing the C code, which still took a significant portion of the total runtime.  One of my biggest optimizations was adding aggressive “smart” prefetching to reduce cache misses.  ffvp8 prefetches the reference frames (PREVIOUS, GOLDEN, and ALTREF)… but only the ones which have been used reasonably often this frame.  This lets us prefetch everything we need without prefetching things that we probably won’t use.  libvpx very often encodes frames that almost never (but not quite never) use GOLDEN or ALTREF, so this optimization greatly reduces time spent prefetching in a lot of real videos.  There are of course countless other optimizations we made that are too long to list here as well, such as David’s entropy decoder optimizations.  I’d also like to thank Eli Friedman for his invaluable help in benchmarking a lot of these changes.

What next?  Altivec (PPC) assembly is almost nonexistent, with the only functions being David’s motion compensation code.  NEON (ARM) is completely nonexistent: we’ll need that to be fast on mobile devices as well.  Of course, all this will come in due time — and as always — patches welcome!

Appendix: the raw numbers

Here’s the raw numbers (in fps) for the graphs at the start of this post, with standard error values:

Core i7 620QM (1.6Ghz), Windows 7, 32-bit:
Parkjoy ffvp8: 44.58 +/- 0.44
Parkjoy libvpx: 33.06 +/- 0.23
Sintel ffvp8: 74.26 +/- 1.18
Sintel libvpx: 56.11 +/- 0.96

Core i5 520M (2.4Ghz), Linux, 64-bit:
Parkjoy ffvp8: 68.29 +/- 0.06
Parkjoy libvpx: 41.06 +/- 0.04
Sintel ffvp8: 112.38 +/- 0.37
Sintel libvpx: 69.64 +/- 0.09

Core 2 T9300 (2.5Ghz), Mac OS X 10.6.4, 64-bit:
Parkjoy ffvp8: 54.09 +/- 0.02
Parkjoy libvpx: 33.68 +/- 0.01
Sintel ffvp8: 87.54 +/- 0.03
Sintel libvpx: 52.74 +/- 0.04

Core Duo (2Ghz), Mac OS X 10.6.4, 32-bit:
Parkjoy ffvp8: 21.31 +/- 0.02
Parkjoy libvpx: 17.96 +/- 0.00
Sintel ffvp8: 41.24 +/- 0.01
Sintel libvpx: 29.65 +/- 0.02

Atom N270 (1.6Ghz), Linux, 32-bit:
Parkjoy ffvp8: 15.29 +/- 0.01
Parkjoy libvpx: 12.46 +/- 0.01
Sintel ffvp8: 26.87 +/- 0.05
Sintel libvpx: 20.41 +/- 0.02

These are less-so issues related to compression — that issue has been beaten to death, particularly in MSU’s recent comparison, where x264 beat the crap out of VP8 and the VP8 developers pulled a Pinocchio in the developer comments.  But that was expected and isn’t particularly interesting, so I won’t go into that.  VP8 doesn’t have to be the best in the world in order to be useful.

When the ffmpeg VP8 decoder is complete (just a few more asm functions to go), we’ll hopefully be able to post some benchmarks comparing it to libvpx.

1.  The spec, er, I mean, bitstream guide.

Google has reneged on their claim that a spec existed at all and renamed it a “bitstream guide”.  This is probably after it was found that — not merely was it incomplete — but at least a dozen places in the spec differed wildly from what was actually in their own encoder and decoder software!  The deblocking filter, motion vector clamping, probability tables, and many more parts simply disagreed flat-out with the spec.  Fortunately, Ronald Bultje, one of the main authors of the ffmpeg VP8 decoder, is rather skilled at reverse-engineering, so we were able to put together a matching implementation regardless.

Most of the differences aren’t particularly important — they don’t have a huge effect on compression or anything — but make it vastly more difficult to implement a “working” VP8 decoder, or for that matter, decide what “working” really is.  For example, Google’s decoder will, if told to “swap the ALT and GOLDEN reference frames”, overwrite both with GOLDEN, because it first sets GOLDEN = ALT, and then sets ALT = GOLDEN.  Is this a bug?  Or is this how it’s supposed to work?  It’s hard to tell — there isn’t a spec to say so.  Google says that whatever libvpx does is right, but I doubt they intended this.

I expect a spec will eventually be written, but it was a bit obnoxious of Google — both to the community and to their own developers — to release so early that they didn’t even have their own documentation ready.

2.  The TM intra prediction mode.

One thing I glossed over in the original piece was that On2 had added an extra intra prediction mode to the standard batch that H.264 came with — they replaced Planar with “TM pred”.  For i4x4, which didn’t have a Planar mode, they just added it without replacing an old one, resulting in a total of 10 modes to H.264′s 9.  After understanding and writing assembly code for TM pred, I have to say that it is quite a cool idea.  Here’s how it works:

1.  Let us take a block of size 4×4, 8×8, or 16×16.

2.  Define the pixels bordering the top of this block (starting from the left) as T[0], T[1], T[2]…

3.  Define the pixels bordering the left of this block (starting from the top) as L[0], L[1], L[2]…

4.  Define the pixel above the top-left of the block as TL.

5.  Predict every pixel <X,Y> in the block to be equal to clip3( T[X] + L[Y] – TL, 0, 255).

It’s effectively a generalization of gradient prediction to the block level — predict each pixel based on the gradient between its top and left pixels, and the topleft.  According to the VP8 devs, it’s chosen by the encoder quite a lot of the time, which isn’t surprising; it seems like a pretty good idea.  As just one more intra pred mode, it’s not going to do magic for compression, but it’s a cool idea and elegantly simple.

3.  Performance and the deblocking filter.

On2 advertised for quite some that VP8′s goal was to be significantly faster to decode than H.264.  When I saw the spec, I waited for the punchline, but apparently they were serious.  There’s nothing wrong with being of similar speed or a bit slower — but I was rather confused as to the fact that their design didn’t match their stated goal at all.  What apparently happened is they had multiple profiles of VP8 — high and low complexity profiles.  They marketed the performance of the low complexity ones while touting the quality of the high complexity ones, a tad dishonest.  More importantly though, practically nobody is using the low complexity modes, so anyone writing a decoder has to be prepared to handle the high complexity ones, which are the default.

The primary time-eater here is the deblocking filter.  VP8, being an H.264 derivative, has much the same problem as H.264 does in terms of deblocking — it spends an absurd amount of time there.  As I write this post, we’re about to finish some of the deblocking filter asm code, but before it’s committed, up to 70% or more of total decoding time is spent in the deblocking filter!  Like H.264, it suffers from the 4×4 transform problem: a 4×4 transform requires a total of 8 length-16 and 8 length-8 loopfilter calls per macroblock, while Theora, with only an 8×8 transform, requires half that.

This problem is aggravated in VP8 by the fact that the deblocking filter isn’t strength-adaptive; if even one 4×4 block in a macroblock contains coefficients, every single edge has to be deblocked.  Furthermore, the deblocking filter itself is quite complicated; the “inner edge” filter is a bit more complex than H.264′s and the “macroblock edge” filter is vastly more complicated, having two entirely different codepaths chosen on a per-pixel basis.  Of course, in SIMD, this means you have to do both and mask them together at the end.

There’s nothing wrong with a good-but-slow deblocking filter.  But given the amount of deblocking one needs to do in a 4×4-transform-based format, it might have been a better choice to make the filter simpler.  It’s pretty difficult to beat H.264 on compression, but it’s certainly not hard to beat it on speed — and yet it seems VP8 missed a perfectly good chance to do so.  Another option would have been to pick an 8×8 transform instead of 4×4, reducing the amount of deblocking by a factor of 2.

And yes, there’s a simple filter available in the low complexity profile, but it doesn’t help if nobody uses it.

4.  Tree-based arithmetic coding.

Binary arithmetic coding has become the standard entropy coding method for a wide variety of compressed formats, ranging from LZMA to VP6, H.264 and VP8.  It’s simple, relatively fast compared to other arithmetic coding schemes, and easy to make adaptive.  The problem with this is that you have to come up with a method for converting non-binary symbols into a list of binary symbols, and then choosing what probabilities to use to code each one.  Here’s an example from H.264, the sub-partition mode symbol, which is either 8×8, 8×4, 4×8, or 4×4.  encode_decision( context, bit ) writes a binary decision (bit) into a numbered context (context).

8×8: encode_decision( 21, 0 );

8×4: encode_decision( 21, 1 ); encode_decision( 22, 0 );

4×8: encode_decision( 21, 1 ); encode_decision( 22, 1 ); encode_decision( 23, 1 );

4×4: encode_decision( 21, 1 ); encode_decision( 22, 1 ); encode_decision( 23, 0 );

As can be seen, this is clearly like a Huffman tree.  Wouldn’t it be nice if we could represent this in the form of an actual tree data structure instead of code?  On2 thought so — they designed a simple system in VP8 that allowed all binarization schemes in the entire format to be represented as simple tree data structures.  This greatly reduces the complexity — not speed-wise, but implementation-wise — of the entropy coder.  Personally, I quite like it.

5.  The inverse transform ordering.

I should at some point write a post about common mistakes made in video formats that everyone keeps making.  These are not issues that are patent worries or huge issues for compression — just stupid mistakes that are repeatedly made in new video formats, probably because someone just never asked the guy next to him “does this look stupid?” before sticking it in the spec.

One common mistake is the problem of transform ordering.  Every sane 2D transform is “separable” — that is, it can be done by doing a 1D transform vertically and doing the 1D transform again horizontally (or vice versa).  The original iDCT as used in JPEG, H.263, and MPEG-1/2/4 was an “idealized” iDCT — nobody had to use the exact same iDCT, theirs just had to give very close results to a reference implementation.  This ended up resulting in a lot of practical problems.  It was also slow; the only way to get an accurate enough iDCT was to do all the intermediate math in 32-bit.

Practically every modern format, accordingly, has specified an exact iDCT.  This includes H.264, VC-1, RV40, Theora, VP8, and many more.  Of course, with an exact iDCT comes an exact ordering — while the “real” iDCT can be done in any order, an exact iDCT usually requires an exact order.  That is, it specifies horizontal and then vertical, or vertical and then horizontal.

All of these transforms end up being implemented in SIMD.  In SIMD, a vertical transform is generally the only option, so a transpose is added to the process instead of doing a horizontal transform.  Accordingly, there are two ways to do it:

1.  Transpose, vertical transform, transpose, vertical transform.

2.  Vertical transform, transpose, vertical transform, transpose.

These may seem to be equally good, but there’s one catch — if the transpose is done first, it can be completely eliminated by merging it into the coefficient decoding process.  On many modern CPUs, particularly x86, transposes are very expensive, so eliminating one of the two gives a pretty significant speed benefit.

H.264 did it way 1).

VC-1 did it way 1).

Theora (inherited from VP3) did it way 1).

But no.  VP8 has to do it way 2), where you can’t eliminate the transpose.  Bah.  It’s not a huge deal; probably only ~1-2% overall at most speed-wise, but it’s just a needless waste.  What really bugs me is that VP3 got it right — why in the world did they screw it up this time around if they got it right beforehand?

RV40 is the other modern format I know that made this mistake.

(NB: You can do transforms without a transpose, but it’s generally not worth it unless the intermediate needs 32-bit math, as in the case of the “real” iDCT.)

6.  Not supporting interlacing.

THANK YOU THANK YOU THANK YOU THANK YOU THANK YOU THANK YOU THANK YOU.

Interlacing was the scourge of H.264.  It weaseled its way into every nook and cranny of the spec, making every decoder a thousand lines longer.  H.264 even included a highly complicated — and effective — dedicated interlaced coding scheme, MBAFF.  The mere existence of MBAFF, despite its usefulness for broadcasters and others still stuck in the analog age with their 1080i, 576i , and 480i content, was a blight upon the video format.

VP8 has once and for all avoided it.

And if anyone suggests adding interlaced support to the experimental VP8 branch, find a straightjacket and padded cell for them before they cause any real damage.