Taking submissions for encoder comparison
With VP8 supposedly going to come out in about 2 weeks, it’s time to get a rough idea as to the visual state of the art in terms of encoders. Accordingly, I’m doing a small visual codec comparison in which we will take a few dozen encoders, encode a single test clip, and perform score-based visual tests on real humans using a blind test. There will be no PSNR or SSIM results posted.
See the doom9 thread for more information and feel free to submit streams for your own encoders. I’m particularly interested in some newer proprietary encoders for which I wouldn’t be able to get the software for due to NDAs or similar (such as VP8, Sony Blu-code, etc) — but for which I would be able to get a dump of the decoded output.
May 8th, 2010 at 3:13 pm
You can use Flix Cloud to get your VP8 video.
http://www.flixcloud.com/
That is On2′s VP8 cloud encoder.
May 8th, 2010 at 4:34 pm
@Louise
I don’t see VP8 listed…
May 8th, 2010 at 4:37 pm
This this PDF On2 compares VP7 with x264
http://www.on2.com/file.php?229
May 8th, 2010 at 4:39 pm
@Dark Shikari
Very sorry about that. I was pretty sure I saw it on the format list…
May 8th, 2010 at 4:45 pm
@Louise
Please stop posting On2 marketing material on my blog. We all know it’s bullshit, including you, and it’s certainly not grounds for an encoder comparison. On2 have spent the last 10 years claiming absurd gains over industry leaders and never actually delivered on any of their promises.
FYI, in my tests so far, VP7 got beaten pretty badly by every single good H.264 encoder that I’ve tested. It was still pretty respectable though (if you can get past the disastrous level of blurring) and beat basically everything pre-H.264.
May 8th, 2010 at 5:21 pm
MSU Video Quality Measurement Tools
Provides objective (“Metrics”) and subjective (“Blind Test”) video quality evaluation. Might be useful (including AviSynth-Support).
May 8th, 2010 at 6:26 pm
Why not wait until VP8 is released officially? You can’t just accepted the “decoded output” without knowing what the guy actually used, what settings, and whether he’s competent to do it in the first place.
I think that’s unfair.
(btw, I don’t care about On2 or advertising their codecs)
May 8th, 2010 at 9:51 pm
@Shiretoko
True, there’s no guarantee I can trust said person, but how can you guarantee that I’ll know the “best settings” to use with it either?
And what if VP8 doesn’t come out as promised by the gossip tech media?
I could just delay it a week or so and hope that it does come out, in which case I can include it in the main visual comparison (it’s hard to go and add anything to the comparison afterwards, because everyone’s ratings are going to be made in the context of all the other images).
May 9th, 2010 at 3:25 am
Any reasons you didn’t add FFmpeg MPEG-2?
May 9th, 2010 at 10:58 am
@Carl
MPEG-2 is basically the same as MPEG-1 without interlacing support. There are minor differences but it’s not worth bothering.
May 9th, 2010 at 1:06 pm
XVD Video (priority=quality, 2 pass, key interval=250, qp=15)
http://www.mediafire.com/?mtmzzdzjdgt
I used trial version encoder of deceased BHA (Japanese company) released in 2003.
it’s too dark and blurring.
May 9th, 2010 at 6:38 pm
I don’t know if it’s possible, but what I’d really be interested in seeing are some hardware implementations of H264; specifically those from TI and Analog Devices, for example.
I can encode you video with a TI DM365, if you’re interested. But what really irks me about the embedded encoder space is these people get away with marketing shit products under the guise of “it’s H264!” when everybody well versed in the industry is acutely aware of the differences in encoder quality.
May 10th, 2010 at 10:42 am
Using Unreal Live Server v7.0, you can compare one-pass, real-time live encoding of VC1 and x264 codecs.
As I wrote in another post here, we find VC1 slightly better at low bitrates and x264 better (more detailed picture) at medium-high bitrates
May 10th, 2010 at 2:10 pm
Are you going to be posting all the results and decompressed videos somewhere? How is the blind testing being done?
It would be nice if the actual app and command line used for each case was posted so they can be independently verified and checked for correctness.
June 12th, 2010 at 8:18 pm
Any news on this? It’s been a month or two since this was announced — any ETA on the results?
June 13th, 2010 at 7:30 pm
@R.T.
Very busy at my current company because of upcoming E3. After E3, I’ll get back to this. Harass me if I don’t.
July 26th, 2010 at 9:26 pm
E3 is long over now, and a bit over a month has passed. I understand that you’re probably quite busy, but it’d be nice to know if this is dead.