Additionally to one-pass mode in Main profile of standard
H.264 we tested two-pass mode and compression using possibilities of High
profile. Only x264, Elecard, Ateme and Fraunhofer IIS codecs took part in this
test. This test was held on "foreman" and "battle" sequences and without speed
measuring.
As it is seen on picture below, it is difficult to analyze
results using PSNR/Bitrate graphs. That is why we use Delta Y-PSNR graphs
instead of absolute Y-PSNR graphs in this part. In all graphs reference is Main
Profile one pass results for current codec.
|
|
Picture 1.
Different presets of codec x264, Absolute Y-PSNR
|
|
|
|
Picture 2.
Different presets of codec x264, Delta Y-PSNR
|
|
|
|
Picture 3.
Different presets of codec Elecard H.264, Delta Y-PSNR
|
|
|
|
Picture 4.
Different presets of codec Ateme H.264, Delta Y-PSNR
|
|
|
|
Picture 5.
Different presets of codec Ateme H.264, Delta Y-VQM
|
|
|
|
Picture 6.
Different presets of codec Ateme H.264, Delta-Y-SSIM
|
|
|
|
Picture 7.
Different presets of codec Fraunhofer IIS H.264, Delta
Y-PSNR
|
|
|
|
Picture 8.
Different presets of codec x264, Delta Y-PSNR
|
|
|
|
Picture 9.
Different presets of codec Elecard H.264, Delta Y-PSNR
|
|
|
|
Picture 10.
Different presets of codec Ateme H.264, Delta Y-PSNR
|
|
|
|
Picture 11.
Different presets of codec Ateme H.264, Delta Y-VQM
|
|
|
|
Picture 12.
Different presets of codec Ateme H.264, Delta Y-SSIM
|
|
|
|
Picture 13.
Different presets of codec Fraunhofer IIS H.264, Delta
Y-PSNR
|
Note:
In addition to average metrics' values we measured
metrics' values for separate frames. Basing on this data, Rate Control of each
codec could be analyzed. As a rule, on such graphs it is clearly seen whether
codec uses B-frames or not ("striation" of graph), location of I-frames (abrupt
peaks or collapses on graph), scene changing reaction, quality of R-D models,
etc.
On the next graphs there are frames on abscissa axis and
bit rates on axis of ordinates (low bit rates are below). Color shows metric's
value: the more red, the lesser is value. Thus, for Y-PSNR metric red color
means bad quality, blue - good quality. All graphs of one sequence have the
same colors (every color corresponds to only one value of metric on all
graphs).
Analyzing these graphics we should take into account bit rate
handling of codecs since these graphs do not consider it.
|
|
|
Picture 14.
Codec DivX. Sequence "foreman".
|
|
|
|
Picture 15.
Codec x264. Sequence "foreman".
|
|
|
|
Picture 16.
Codec ArcSoft H.264. Sequence "foreman".
|
|
|
|
Picture 17.
Codec Elecard H.264. Sequence "foreman".
|
|
|
|
Picture 18.
Codec Ateme H.264. Sequence "foreman".
|
|
|
|
Picture 19.
Codec Fraunhofer IIS H.264. Sequence "foreman".
|
|
|
|
Picture 20.
Codec DivX. Sequence "bbc".
|
On this graph compression quality does not significantly
depend from bit rate, because DivX codec does not maintain low bit rates well,
and works badly on high ones.
|
|
|
Picture 21.
Codec x264. Sequence "bbc".
|
|
|
|
Picture 22.
Codec ArcSoft H.264. Sequence "bbc".
|
|
|
|
Picture 23.
Codec VSS H.264. Sequence "bbc".
|
|
|
|
Picture 24.
Codec Elecard H.264. Sequence "bbc".
|
|
|
|
Picture 25.
Codec Ateme H.264. Sequence "bbc".
|
|
|
|
Picture 26.
Codec Fraunhofer IIS H.264. Sequence "bbc".
|
Remarks:
|
Graph type
|
Total number of graph
|
Inserted in this document
|
|
Per frame metrics
|
1414
|
13 (0.9%)
|
Even in spite of appearance of new metrics, that more
adequately reflect human perception of video, subjective video tests still do
not lose their importance.
We set ourselves a problem to compare video codecs using
only existing objective metrics. But in addition to various graphs that
impartially evaluate quality of obtained results, below we give several frames
from "bbc" and "battle" sequences, compressed with different codecs.
|
|
|
|
|
Picture
27.
Original
|
Picture
28.
DivX
|
Picture
29.
x264
|
|
|
|
|
|
Picture
30.
Original. Image differences. Y-PSNR
|
Picture
31.
DivX. Image differences. Y-PSNR
|
Picture
32.
x264. Image differences. Y-PSNR
|
|
|
|
|
|
Picture
33.
Original
|
Picture
34.
DivX
|
Picture
35.
x264
|
|
|
|
|
|
Picture
36.
Original. Image differences. Y-PSNR
|
Picture
37.
DivX. Image differences. Y-PSNR
|
Picture
38.
x264. Image differences. Y-PSNR
|
|
|
|
Picture
39.
Original
|
|
|
|
Picture
40.
DivX
|
|
|
|
Picture
41.
x264
|
|
|
|
Picture
42.
ArcSoft
|
|
|
|
Picture
43.
VSS
|
|
|
|
Picture
44.
Elecard
|
|
|
|
Picture
45.
Ateme
|
|
|
|
Picture
46.
Fraunhofer IIS
|
|
|
|
|
Picture
47.
Y-PSNR ArcSoft
|
Picture
48.
Y-PSNR Ateme
|
|
|
|
|
Picture
49.
Y-PSNR DivX
|
Picture
50.
Y-PSNR Elecard
|
|
|
|
|
Picture
51.
Y-PSNR Fraunhofer IIS
|
Picture
52.
Y-PSNR VSS
|
|
|
|
|
Picture 53.
Y-PSNR Original
|
Picture 54.
Y-PSNR x264
|
As it is clearly seen on the graphs, codecs show
significantly different results on different sequences. Selected sequences have
different types in terms of motion and noise, and this allows checking which
codecs are tuned to all types of sequences, and which one - to only one. But
also it is very interesting to evaluate codec behavior on all test data.
We considered averaging of results between different
sequences as incorrect, and used a method of assigning points to codecs for
different measurements. Points are united in arbitrary informal rating in the
end.
We tried to reflect real situation on test data as much as
possible. We are still developing methodology of our informal comparison.
Therefore it is better to consider following results as subjective opinion of
comparison authors.
Separate comparisons were
carried out for "Best Quality" and "Best Speed" presets. Several graphs were
chosen for each preset, on which comparison was held. Obtained marks were
summed with some predefined weights.
Codecs received points in a
following way:
Following graphs were used for
"Best Quality" preset with corresponding coefficients:
Following graphs were used for
"Best Speed" preset with corresponding coefficients:
Average values were computed
as arithmetic average of all successfully encoded sequences (i.e. codecs'
errors were not taken into account).
Sequence "foreman"
|
|
DivX
|
x264
|
ArcSoft
|
VSS
|
Elecard
|
Ateme
|
Fraunhofer
IIS
|
|
Y-PSNR
|
1
|
2
|
1
|
0
|
2
|
3
|
2
|
|
U-PSNR
|
1
|
3
|
1
|
0
|
3
|
3
|
2
|
|
V-PSNR
|
1
|
3
|
2
|
0
|
3
|
3
|
2
|
|
SSIM
|
1
|
3
|
1
|
0
|
2
|
2
|
2
|
|
VQM
|
1
|
2
|
1
|
0
|
2
|
3
|
2
|
|
BH
|
1
|
2
|
2
|
0
|
2
|
2
|
2
|
|
Time
|
2
|
2
|
3
|
0
|
2
|
3
|
1
|
|
Total
|
13
|
27
|
18
|
0
|
26
|
32
|
23
|
Sequence "susi"
|
|
DivX
|
x264
|
ArcSoft
|
VSS
|
Elecard
|
Ateme
|
Fraunhofer IIS
|
|
Y-PSNR
|
2
|
2
|
1
|
2
|
2
|
3
|
2
|
|
U-PSNR
|
2
|
3
|
1
|
2
|
2
|
2
|
1
|
|
V-PSNR
|
2
|
3
|
1
|
3
|
2
|
2
|
1
|
|
SSIM
|
2
|
2
|
1
|
2
|
2
|
3
|
2
|
|
VQM
|
2
|
2
|
1
|
2
|
1
|
3
|
2
|
|
BH
|
1
|
2
|
3
|
2
|
2
|
2
|
1
|
|
Time
|
2
|
2
|
3
|
2
|
2
|
2
|
1
|
|
Total
|
21
|
26
|
20
|
25
|
23
|
30
|
18
|
Sequence "bbc"
|
|
DivX
|
x264
|
ArcSoft
|
VSS
|
Elecard
|
Ateme
|
Fraunhofer
IIS
|
|
Y-PSNR
|
1
|
2
|
1
|
2
|
2
|
3
|
3
|
|
U-PSNR
|
1
|
3
|
1
|
2
|
2
|
2
|
3
|
|
V-PSNR
|
2
|
3
|
1
|
2
|
2
|
3
|
3
|
|
SSIM
|
2
|
2
|
1
|
3
|
2
|
3
|
2
|
|
VQM
|
2
|
2
|
1
|
2
|
2
|
3
|
3
|
|
BH
|
1
|
3
|
1
|
2
|
2
|
2
|
3
|
|
Time
|
2
|
2
|
3
|
2
|
2
|
2
|
1
|
|
Total
|
16
|
29
|
14
|
25
|
24
|
31
|
33
|
Sequence "battle"
|
|
DivX
|
x264
|
ArcSoft
|
VSS
|
Elecard
|
Ateme
|
Fraunhofer
IIS
|
|
Y-PSNR
|
2
|
2
|
1
|
1
|
2
|
3
|
2
|
|
U-PSNR
|
1
|
3
|
1
|
2
|
3
|
2
|
3
|
|
V-PSNR
|
1
|
3
|
1
|
2
|
3
|
2
|
3
|
|
SSIM
|
2
|
2
|
1
|
1
|
2
|
3
|
2
|
|
VQM
|
2
|
2
|
1
|
2
|
2
|
3
|
3
|
|
BH
|
1
|
3
|
2
|
2
|
2
|
2
|
3
|
|
Time
|
2
|
2
|
3
|
2
|
2
|
2
|
1
|
|
Total
|
19
|
29
|
17
|
19
|
26
|
30
|
29
|
Sequence "simpsons"
|
|
DivX
|
x264
|
ArcSoft
|
VSS
|
Elecard
|
Ateme
|
Fraunhofer IIS
|
|
Y-PSNR
|
2
|
3
|
1
|
3
|
2
|
3
|
2
|
|
U-PSNR
|
1
|
3
|
1
|
3
|
2
|
2
|
2
|
|
V-PSNR
|
1
|
3
|
1
|
3
|
3
|
2
|
2
|
|
SSIM
|
2
|
3
|
1
|
3
|
2
|
2
|
2
|
|
VQM
|
2
|
3
|
1
|
3
|
2
|
2
|
2
|
|
BH
|
1
|
2
|
2
|
2
|
2
|
2
|
3
|
|
Time
|
2
|
2
|
3
|
2
|
2
|
2
|
1
|
|
Total
|
19
|
32
|
17
|
32
|
25
|
28
|
26
|
Sequence "matrix"
|
|
DivX
|
x264
|
ArcSoft
|
VSS
|
Elecard
|
Ateme
|
Fraunhofer IIS
|
|
Y-PSNR
|
2
|
2
|
1
|
2
|
2
|
3
|
1
|
|
U-PSNR
|
2
|
3
|
1
|
3
|
2
|
2
|
1
|
|
V-PSNR
|
2
|
3
|
1
|
2
|
3
|
2
|
1
|
|
SSIM
|
2
|
2
|
1
|
2
|
2
|
3
|
1
|
|
VQM
|
2
|
2
|
1
|
2
|
2
|
3
|
2
|
|
BH
|
1
|
1
|
2
|
3
|
2
|
2
|
2
|
|
Time
|
2
|
2
|
3
|
2
|
2
|
2
|
1
|
|
Total
|
21
|
23
|
17
|
28
|
25
|
30
|
16
|
Sequence "concert"
|
|
DivX
|
x264
|
ArcSoft
|
VSS
|
Elecard
|
Ateme
|
Fraunhofer IIS
|
|
Y-PSNR
|
0
|
2
|
1
|
3
|
1
|
3
|
3
|
|
U-PSNR
|
0
|
3
|
3
|
2
|
2
|
2
|
1
|
|
V-PSNR
|
0
|
3
|
3
|
2
|
2
|
2
|
1
|
|
SSIM
|
0
|
2
|
1
|
3
|
2
|
1
|
3
|
|
VQM
|
0
|
2
|
1
|
3
|
1
|
2
|
3
|
|
BH
|
0
|
2
|
2
|
2
|
1
|
2
|
3
|
|
Time
|
0
|
2
|
2
|
2
|
3
|
2
|
1
|
|
Total
|
0
|
26
|
20
|
30
|
17
|
27
|
30
|
Results
|
Codec
|
Average points
|
Place
|
|
Ateme
|
29.71
|
1
|
|
x264
|
27.43
|
2
|
|
VSS
|
26.5
|
3
|
|
Fraunhofer IIS
|
25
|
4
|
|
Elecard
|
23.71
|
5
|
|
DivX
|
18.17
|
6
|
|
ArcSoft
|
17.57
|
7
|
Sequence "foreman"
