x264 video encoding benchmark

[mroz]

Can anyone comment on the following? I've just run this benchmark on my Q6600 & noticed the mp4 files from the five runs are all slightly different file sizes (varying by up to about 2.5KiB); also on average they're 16312KiB compared to 16281 which I got from my old XP 3200+ box. What's going on?

I could accept different md5s for the output runs as the mp4s contain info such as the creation time (possibly), but varying file sizes?

Is x264 output in multithreaded runs non-deterministic by default?

I've read x264 encoding with more than one thread produces slightly poorer quality output (don't know if this applies to recent builds), which makes sense as slicing the work up between threads could mean some frames are encoded necessarily without reference to frames being processed by other threads at the same time, however even then surely the output would be the same for different runs?

There also seems to be a non-deterministic cmd line option which I don't fuly understand, however it isn't included in the command line used by these tests, so that doesn't seem to explain anything. Hm, some info here but still doesn't explain anything unless --non-deterministic is on by default.

Can anone who's run this on a quad check to see if they're getting output files from each of the five runs of different sizes? Even if it's just a byte or two.

Thanks.

[mroz]

Here are my results fwiw:

---------- RUN1PASS1.LOG
encoded 1749 frames, 127.06 fps, 1850.94 kb/s

---------- RUN2PASS1.LOG
encoded 1749 frames, 127.79 fps, 1850.94 kb/s

---------- RUN3PASS1.LOG
encoded 1749 frames, 128.22 fps, 1850.94 kb/s

---------- RUN4PASS1.LOG
encoded 1749 frames, 127.94 fps, 1850.94 kb/s

---------- RUN5PASS1.LOG
encoded 1749 frames, 128.37 fps, 1850.94 kb/s

---------- RUN1PASS2.LOG
encoded 1749 frames, 33.81 fps, 1829.47 kb/s

---------- RUN2PASS2.LOG
encoded 1749 frames, 33.83 fps, 1829.52 kb/s

---------- RUN3PASS2.LOG
encoded 1749 frames, 34.01 fps, 1829.27 kb/s

---------- RUN4PASS2.LOG
encoded 1749 frames, 33.94 fps, 1829.45 kb/s

---------- RUN5PASS2.LOG
encoded 1749 frames, 34.02 fps, 1829.19 kb/s

Q6600 g0 @ stock 9x266, P35-DS4 (intel p35/g33/g31), 2GiB PC6400 ddr2 @ 266MHz (I forgot to reset the divider from 2 to 3) 4 4 4 12

[mroz]

---------- RUN1PASS1.LOG
encoded 1749 frames, 176.27 fps, 1850.94 kb/s

---------- RUN2PASS1.LOG
encoded 1749 frames, 177.38 fps, 1850.94 kb/s

---------- RUN3PASS1.LOG
encoded 1749 frames, 177.38 fps, 1850.94 kb/s

---------- RUN4PASS1.LOG
encoded 1749 frames, 177.11 fps, 1850.94 kb/s

---------- RUN5PASS1.LOG
encoded 1749 frames, 177.11 fps, 1850.94 kb/s

---------- RUN1PASS2.LOG
encoded 1749 frames, 47.77 fps, 1829.28 kb/s

---------- RUN2PASS2.LOG
encoded 1749 frames, 47.71 fps, 1829.27 kb/s

---------- RUN3PASS2.LOG
encoded 1749 frames, 47.82 fps, 1829.12 kb/s

---------- RUN4PASS2.LOG
encoded 1749 frames, 47.78 fps, 1829.42 kb/s

---------- RUN5PASS2.LOG
encoded 1749 frames, 47.78 fps, 1829.16 kb/s

Q6600 g0 @ 9x378, P35-DS4 (intel p35/g33/g31), 2GiB PC6400 ddr2 @ 378MHz 4 4 4 12

[mroz]

Can anone who's run this on a quad check to see if they're getting output files from each of the five runs of different sizes? Even if it's just a byte or two.

I've changed the batch file to output raw h264 (via an output file extension of .264) & changed the threads option from auto to 1. This gives me five identical output files (md5 checked). With threads set to auto they all still have different sizes & are trivially different.

As I was worried the differences were due to system instability (not seen in the threads 1 case as load then never went much above 30&#37, I repeated that encode with p95 also running. It's not finished yet, but so far the output files are all identical & match the version I was getting prior to adding p95 to up my load. Edit: Yup, all the same md5.

So it seems as well as slightly decreasing psnr, using multiple threads also results in non deterministic output. Can someone else please confirm this?

Does anyone know if this is intended or known about? Also, does it reflect the --non-deterministic option being on by default?

Just for clarity, here's an example of the modified cmd line:

echo ............................................... Now Starting Run 1: Pass 1 of 2
x264 --pass 1 --progress --quiet --bitrate 1823 --stats "1.stats" --bframes 3 --b-pyramid --direct auto --subme 1 --analyse none --threads 1 --thread-input --me hex --no-dct-decimate --no-psnr --no-ssim --output NUL "C:\work2\test\test-480p.avs" 2>&1 | tee run1pass1.log
echo ............................................... Now Starting Run 1: Pass 2 of 2
x264 --pass 2 --progress --quiet --bitrate 1823 --stats "1.stats" --ref 5 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --subme 6 --trellis 1 --analyse all --8x8dct --threads 1 --thread-input --me hex --sar 427:360 --no-dct-decimate --no-psnr --no-ssim --output "C:\work2\run1-480p-t1.264" "C:\work2\test\test-480p.avs" 2>&1 | tee run1pass2.log
del 1.stats

The resulting output file has size 16,649,380 bytes & md5 B9D6397726D8B8F5D2EE59063FE63BF9.

[graysky]

@mroz - thanks for the data. One question: which o/s are you running.

To answer your q about more info, see page 2 (the FAQ) for the test as a start. If you want a better reply, you can ask the questions over in the GUI section @ the doom9 forums. I'm sure sharktooth will answer it

You're right about the file size and threads. As you know, the --threads auto will use 1.5x your cores for the # of threads (6 in the case of a quad since 1.5 times 4 = 6). I have seen this as well and it's normal as far as I know. Have you enabled the quality checks when you did your threads = 1 vs. threads = auto test (e.g. remove the --no-psnr --no-ssim --quiet lines as well as the tee.exe part and they get reported).

With threads auto

Pass1: x264 [info]: x264 [info]: SSIM Mean Y:0.9660379
x264 [info]: PSNR Mean Y:38.014 U:44.609 V:44.008 Avg:39.289 Global:39.242 kb/s:
1850.84
Pass2: x264 [info]: SSIM Mean Y:0.9714373
x264 [info]: PSNR Mean Y:38.989 U:44.755 V:44.160 Avg:40.169 Global:40.103 kb/s:
1829.22

With threads 1

Pass 1: x264 [info]: SSIM Mean Y:0.9659990
x264 [info]: PSNR Mean Y:38.013 U:44.603 V:44.014 Avg:39.288 Global:39.238 kb/s:
1854.00
Pass 2: x264 [info]: SSIM Mean Y:0.9714169
x264 [info]: PSNR Mean Y:38.984 U:44.750 V:44.155 Avg:40.164 Global:40.104 kb/s:
1825.77

So they seem more or less identical.

[Salazaar]

C2D E4300 Stock 9x200 (1.8 Ghz), Asus P5B Deluxe (i965), 2Gib PC2 6400 @ 800Mhz 5-5-5-15, XP 32

---------- RUN1PASS1.LOG
encoded 1749 frames, 52.80 fps, 1850.89 kb/s

---------- RUN2PASS1.LOG
encoded 1749 frames, 52.80 fps, 1850.89 kb/s

---------- RUN3PASS1.LOG
encoded 1749 frames, 52.21 fps, 1850.89 kb/s

---------- RUN4PASS1.LOG
encoded 1749 frames, 52.26 fps, 1850.89 kb/s

---------- RUN5PASS1.LOG
encoded 1749 frames, 52.16 fps, 1850.89 kb/s

---------- RUN1PASS2.LOG
encoded 1749 frames, 12.66 fps, 1826.37 kb/s

---------- RUN2PASS2.LOG
encoded 1749 frames, 12.67 fps, 1826.38 kb/s

---------- RUN3PASS2.LOG
encoded 1749 frames, 12.67 fps, 1826.21 kb/s

---------- RUN4PASS2.LOG
encoded 1749 frames, 12.68 fps, 1826.37 kb/s

---------- RUN5PASS2.LOG
encoded 1749 frames, 12.55 fps, 1826.37 kb/s


C2D E4300 OC 8x400 (3.2 Ghz), Asus P5B Deluxe (i965), 2Gib PC2 6400 @ 800Mhz 5-5-5-15, XP 32

---------- RUN1PASS1.LOG
encoded 1749 frames, 90.56 fps, 1850.89 kb/s

---------- RUN2PASS1.LOG
encoded 1749 frames, 91.08 fps, 1850.89 kb/s

---------- RUN3PASS1.LOG
encoded 1749 frames, 90.93 fps, 1850.89 kb/s

---------- RUN4PASS1.LOG
encoded 1749 frames, 90.93 fps, 1850.89 kb/s

---------- RUN5PASS1.LOG
encoded 1749 frames, 91.15 fps, 1850.89 kb/s

---------- RUN1PASS2.LOG
encoded 1749 frames, 22.12 fps, 1826.38 kb/s

---------- RUN2PASS2.LOG
encoded 1749 frames, 22.17 fps, 1826.37 kb/s

---------- RUN3PASS2.LOG
encoded 1749 frames, 22.13 fps, 1826.38 kb/s

---------- RUN4PASS2.LOG
encoded 1749 frames, 22.14 fps, 1826.37 kb/s

---------- RUN5PASS2.LOG
encoded 1749 frames, 22.13 fps, 1826.38 kb/s

[mroz]

@mroz - thanks for the data. One question: which o/s are you running.

Sorry, meant to add that: XP SP2 (32 bit).

To answer your q about more info, see page 2 (the FAQ) for the test as a start. If you want a better reply, you can ask the questions over in the GUI section @ the doom9 forums. I'm sure sharktooth will answer it

Cheers - it just took me aback as I'm only now finishing my first phase of stability testing. Re the FAQ, I actually read that a day or two ago, but don't recall the filesize question being covered. Has that been added more recently or did I just not pay enough attention? Anything's possible especially at 4am

You're right about the file size and threads. As you know, the --threads auto will use 1.5x your cores for the # of threads (6 in the case of a quad since 1.5 times 4 = 6). I have seen this as well and it's normal as far as I know. Have you enabled the quality checks when you did your threads = 1 vs. threads = auto test (e.g. remove the --no-psnr --no-ssim --quiet lines as well as the tee.exe part and they get reported).

Nope, not yet. Thanks for the suggestion. Agreed that the quality metric differences are tiny, thankfully; from what I read I gather this changed significantly for the better somewhere around build 610.

/Snip

I might ask on the Doom9 forum about the determinism aspect. I could understand if the output is non-deterministic as a side effect of theading & also the support for an experimental option which intentionally makes output non-deterministic as a result of an algorithm to boost quality. What I don't get is why that option is called --non-deterministic, since in my mind that implies without it, output will be deterministic, which it clearly isn't.

[graysky]

@salazaar - thanks for the results.. added.
@mroz - no, you're not going nuts, I added that question based on your discussion here (thanks by the way).

[mroz]

@mroz - no, you're not going nuts, I added that question based on your discussion here (thanks by the way).

I already am nuts, but thanks for the benefit of the doubt

Looks like this result is an unintended side effect of certain combinations of options. I asked about it on doom9. For a given number of threads, & without the --non-deterministic option, it is supposed to be deterministic. See also this thread.

[nme_uk]

Newbie Q here, great thread - very useful results! One Q though - this test is concerned with encoding from mpeg2 to x264. I'm trying to understand if I can use the results posted here, to help me determine how my existing setup will cope with x264 video. I have a P4 3.00Ghz, and according to the results in the chart, this is somewhere at the very bottom in terms of fps - but that result is for encoding - correct? How relevent is that for actual playback - Is the intention here that the encoding result can be depended upon to illustrate how well a given processor will handle HD content de/coding? If that is the intention, then how do you negate the impact upon the results of the presence of HD capable graphics cards - surely a lot of the de/coding work is offloaded to the GPU?

Sorry for the newbie Q's!

[directhex]

Newbie Q here, great thread - very useful results! One Q though - this test is concerned with encoding from mpeg2 to x264. I'm trying to understand if I can use the results posted here, to help me determine how my existing setup will cope with x264 video. I have a P4 3.00Ghz, and according to the results in the chart, this is somewhere at the very bottom in terms of fps - but that result is for encoding - correct?

yep.

How relevent is that for actual playback - Is the intention here that the encoding result can be depended upon to illustrate how well a given processor will handle HD content de/coding?

not so much, no - whilst a cpu that blitzes though encoding will likely blitz through decoding too, the numbers will not match.

if you want to try your hand at that, try http://x264.nl/h.264.samples or http://tribler.org/content/Elephants...ac.mov.torrent

If that is the intention, then how do you negate the impact upon the results of the presence of HD capable graphics cards - surely a lot of the de/coding work is offloaded to the GPU?

depends on your codec and graphics card - not all codecs can do offboard calculation, and not all players can use those codecs. for example, right now, the videolan media player won't do playable HD h264 playback on any consumer CPU on the market - the playback codec is built-in, single-threaded, and less than optimal.

[graysky]

@nme_uk - yeah, what he said

[nme_uk]

yep.

not so much, no - whilst a cpu that blitzes though encoding will likely blitz through decoding too, the numbers will not match.

if you want to try your hand at that, try http://x264.nl/h.264.samples or http://tribler.org/content/Elephants...ac.mov.torrent



depends on your codec and graphics card - not all codecs can do offboard calculation, and not all players can use those codecs. for example, right now, the videolan media player won't do playable HD h264 playback on any consumer CPU on the market - the playback codec is built-in, single-threaded, and less than optimal.

Thanks directhex. I have downloaded the samples and played them back on my pc. Seems OK, but occasionally get some stuttering and pausing. Perhaps needs a slight boost in performance - One Q, what codec would you reccomend for offloading most of the de/coding to a suitable GPU instead of the CPU? Or to put the question the other way, what graphics card and codec would you suggest would be a good /best solution offering offloading of the work to the GPU?

@nme_uk - yeah, what he said

[mroz]

Aiui most recent gen cards - eg Nvidia's 8000 series, along with a player such as PowerDVD. Until many third party decoders support this I'd personally prefer a software solution like CoreAVC, on a decent spec multicore platform that is. If gpu offloading is the only option you've not much choice when it comes to decoders - I think.

[graysky]

As of 20-Sep-2007, we have data on over 100 Intel-based systems and on over 40 AMD-based systems. There are a few trends I picked-up on while browsing through the database. I put them into a single table and color coded them to make them easier to see. If you see a trend I missed, lemme know and I'll add it to the table.

Request: we don't have a single example of a machine that has both WinXP and WinVista on it. If you have a dual-boot setup, it would be cool to see the difference the O/S makes. Another missing trend is a 32-bit O/S vs. the same O/S that's 64-bit.

On to the table:



Yellow: Nearly 1:1 increase by adding an additional processor to a dual-chip MB
Orange: Some operating systems seem to handle x264 more efficiently than others
Red: Insignificant gain by upping the DRAM speed by 50 %
Blue: For the most part, these chips scale in a pretty linear fashion
Green: Tighter/looser memory timings have a pretty insignificant effect
Purple: Keeping the same over-all clock speed using a different combo of multiplier and FSB can give pretty insignificant gains

Again, I only gave this a once-over look; please point out any trends you see that I missed and also don't forgot about the O/S request!

Thanks again to all who contributed!

[directhex]

Request: we don't have a single example of a machine that has both WinXP and WinVista on it. If you have a dual-boot setup, it would be cool to see the difference the O/S makes. Another missing trend is a 32-bit O/S vs. the same O/S that's 64-bit.

you've got linux and xp on the same box, that's interesting, surely?