Add --frame-packing option to emit the frame packing arrangement SEI

[danielcamposramos]

x265 defines the SEI payload type FRAME_PACKING but had no way to write it.
An HEVC encode of side-by-side or top-and-bottom 3D video could not carry the in-stream mark that players and 3D televisions read.
This adds the option, as discussed in #970.

What it adds

--frame-packing <integer> with the values H.265 defines in Table D.8.
3 is side by side, 4 is top and bottom, 5 is temporal interleaving.
It is off by default, and the bitstream does not change when it is not given.
Other values are rejected with an error, because H.265 does not define them.
Frame 0 is always signalled as the left view, as x264 does.

Where the SEI is written

It is a prefix SEI NAL unit written in FrameEncoder::compressFrame(), next to the alternative transfer characteristics SEI.
Types 3 and 4 write it with each IRAP picture, with the persistence flag set.
Type 5 writes it with every picture, because current_frame_is_frame0_flag alternates, with the persistence flag clear.
The syntax is H.265 D.2.16 and the semantics are D.3.16.
The new SEIFramePacking class is in sei.h.

API

The new member framePacking is at the end of x265_param, with X265_BUILD raised to 218.
Parsing, default, copy, the parameter string, the CLI help, cli.rst and the test lists are updated.

How it was tested

The SEI NAL units were compared byte for byte with an independent writer that follows the D.2.16 syntax table field by field.
They matched for types 3 and 4 on every IRAP picture, and for type 5 on every picture with the flag alternating in output order.
The same writer, in its H.264 form, reproduces the 16-byte message x264 writes.
Over 60-frame clips with several GOPs, with and without B-frames, open and closed GOP, there is one SEI with each IRAP picture and none elsewhere for types 3 and 4, and one on every picture for type 5.
Without the option the output is byte-identical to the unpatched build (--frame-threads 1 --pools 1).
The 8-bit and 10-bit builds have no new warnings.
FFmpeg 9.0.2 reads the message and reports Stereo 3D side data with the right type on the pictures (and the left and right view alternating for type 5).
A stock FFmpeg with libx264 carries the layout through a transcode to H.264.
The clips were also played on a Linux stereo desktop: the player reads the SEI from the stream and declares the video as stereo, and each eye gets its own view.
The captures are attached (hevc-sbs-capture.png for --frame-packing 3, hevc-tb-capture.png for --frame-packing 4): LEFT in the left eye, RIGHT in the right eye, the same frame number in both.

Reference SEI

The prefix SEI NAL units the option writes, without the start code, for the byte-for-byte comparison:

Option	NAL unit
--frame-packing 3	4E 01 2D 06 81 81 00 00 03 00 02 80
--frame-packing 4	4E 01 2D 06 82 01 00 00 03 00 02 80
--frame-packing 5, frame 0	4E 01 2D 04 82 81 10 00 80
--frame-packing 5, frame 1	4E 01 2D 04 82 81 00 00 80

Notes

I did not run TestBench, as no primitives are touched.
The three lines in regression-tests.txt and the one in smoke-tests.txt use big_buck_bunny_360p24.y4m, which I have not run through the CI harness.

[vunguyen1989]

Hi @danielcamposramos , I noticed something unexpected with frame-packing type 3 when using --single-sei.

I tested commit c1362f8 with an SBS Y4M input:

  x265 --y4m --preset ultrafast --frames 48 \
    --frame-threads 1 --pools none \
    --keyint 12 --min-keyint 12 --no-scenecut --bframes 0 \
    --no-info --no-aud --no-repeat-headers \
    --frame-packing 3 --single-sei \
    --input input.y4m --output single-sei.h265

  ffmpeg -hide_banner -nostdin -i single-sei.h265 \
    -map 0:v:0 -c:v copy -bsf:v trace_headers -f null -

With --no-single-sei, all four FPA messages before POC 0/12/24/36 are valid. With --single-sei, only the first is valid; FFmpeg reports Invalid SEI message: payload_size too large for the other three (log is attached)

For example, before POC 12, the prefix SEI bytes excluding the start code are:

Expected: 4E 01 2D 06 81 81 00 00 03 00 02 80
Actual: 4E 01 D0 59 FE 20 EC 2D 06 81 81 00 00 03 00 02 80

The extra five bytes match the beginning of the preceding picture’s slice-header payload. Could you take a look at whether the buffer is being reset correctly before collecting the prefix SEI messages?

single-sei_header.log

[danielcamposramos]

Thank you @vunguyen1989, good catch, and thank you for the exact command and bytes.

You were right: m_bs was not reset before the prefix SEI messages were collected.
In single SEI NAL mode it is reset only by the access unit delimiter or by the repeated stream headers, so with --no-aud, on a picture without repeated headers, it still held the previous picture's slice header.

This is older than --frame-packing.
On master, without our option, --hrd --single-sei --no-aud --no-repeat-headers gives the same FFmpeg error on the picture timing SEI of every picture after the first (47 of 48 SEI NAL units in a 48 picture encode).
The frame packing message made it visible in one more combination.

The fix is a separate commit, 4712634, which resets m_bs in that case and in no other.
With your command, the four messages before POC 0/12/24/36 are now all 4E 01 2D 06 81 81 00 00 03 00 02 80, and trace_headers reports no errors.

What I checked, 8 and 10 bit:

• types 3, 4 and 5 with --single-sei in all four --aud/--no-aud and --repeat-headers/--no-repeat-headers combinations: every prefix SEI NAL unit parses exactly, and every frame packing payload matches an independent reference writer;
• --single-sei with --hrd and --max-cll in the same NAL units: clean;
• streams that were already correct do not change: with --aud, without --single-sei, and on keyframes with --repeat-headers the output is byte-identical to before the fix, and without --frame-packing it is byte-identical to master.

Our earlier tests ran --single-sei only together with --repeat-headers, which hides the problem on keyframes, so your case is now part of our test matrix.

If you prefer the fix as its own PR, since it also affects master, I can split it out.

[vunguyen1989]

Thanks for the detailed follow-up and for verifying all those combinations.

Since the m_bs issue also affects master independently of frame packing, I think splitting the fix into its own PR would be cleaner and make the frame-packing PR easier to review.

Glad the test case helped uncover it.

[danielcamposramos]

Thank you, done: the fix is now its own pull request, #988, on current master, and this one is back to the single frame packing commit.
Until #988 is merged, --single-sei --no-aud --no-repeat-headers writes the same stale bytes with or without --frame-packing.

[Akilan-Sivakumar]

Hi @danielcamposramos. Thanks for the PR. The SEI matches the reference bytes in the description, and the output without the option is unchanged. Two combinations need a fix:

1. Type 5 with --interlace / --field: The view flag comes from m_poc & 1, which counts fields, so the two fields of one frame get different views:

x265 --input in.y4m -o out.hevc --frames 8 --bframes 0 --interlace tff --field --frame-packing 5
got:      L R L R L R ...
expected: L L R R L L ...

Types 3 and 4 work fine with fields.

2. --num-views 2: Multi-view writes each view as its own full picture, but the side-by-side SEI is still written on both layers, so a player would split a single view in half.

Minor

• Please rebase on current master. The branch is older than the --field fixes, but it applies cleanly.
• With --alpha, the SEI is also written on the alpha layer. Is that intended?
• Doc: mention that other values give an error, and that for type 5 the first encoded picture is the left view (so an odd --seek swaps the views).

[danielcamposramos]

Thank you for the careful review, and for the exact command that showed the field problem.

All points are fixed in the rebased branch (one commit, on top of current master).

Type 5 with --interlace and --field:
The view now comes from the frame, not the field (POC >> 1 when --field is on).
Your command gives L L R R L L R R now, and L R L R before.
Without --field the output is byte for byte what it was, and so are types 3 and 4 with and without --field.

--num-views 2:
You are right, each layer is a full view, so a packed arrangement would be wrong for it.
I chose a clear error ("Frame packing arrangement SEI cannot be used with multi-view encoding, each layer is a full view") instead of silently dropping the SEI.
A user who asked for the SEI should know that it is not written.
Without --frame-packing the multi-view output is unchanged.

--alpha:
It was not intended.
The message is now written on the base layer only (nuh_layer_id 0), because the alpha layer is an auxiliary picture and not a view, so it has nothing to describe.
Read back, layer 0 carries the SEI on the IRAP (types 3 and 4) or on every picture (type 5), and layer 1 carries none.

Rebase and docs:
The branch is rebased on current master.
The reST docs, the CLI help and x265.h now say that other values give an error, that for type 5 the first encoded picture is the left view (so an odd --seek swaps the views), that --field keeps both fields of a frame in one view, and the base layer and --num-views rules.
I added a regression test line for type 5 with --field.

Evidence:
I read the SEI back with a small Annex B parser that I wrote for this, and with the ffmpeg 7.1 HEVC decoder (Stereo 3D side data), and both agree.
I built with ENABLE_ALPHA and ENABLE_MULTIVIEW on, and ran types 3, 4 and 5 with and without --field, --interlace alone, --alpha, --num-views 2, --seek 1 and the invalid values 0, 1, 2, 6 and 7.
The tables are in the evidence folder of our workspace and I can paste any of them here.

One thing I noticed while testing, which is not part of this change:
the check "Adaptive B-frame decision method should be closed for field feature" fires when --b-adapt is 0, so --field needs a nonzero --b-adapt (the default works, the ultrafast preset does not).
It looks inverted, but I left it alone.

Daniel

[Akilan-Sivakumar]

Thanks for the quick update. Verified on bf495e5: --field (tff and bff) now gives L L R R, --num-views 2 gives the error, the alpha layer has no SEI, types 3/4 are unchanged, and the output without the option is identical to master.

Three small things left:

1. --interlace without --field. Here the input pictures are already fields, so two pictures make one frame, but the view still alternates per picture:

x265 --input in.y4m -o out.hevc --frames 8 --bframes 0 --interlace tff --frame-packing 5
got:      L R L R L R L R
expected: L L R R L L R R

Using m_param->interlaceMode ? m_poc >> 1 : m_poc (without the bField condition) should cover both cases.

2. The new regression line fails. --preset veryfast sets b-adapt 0, so --field stops with "Adaptive B-frame decision method should be closed for field feature". Please use a preset that keeps b-adapt (e.g. --preset medium), or add --b-adapt 2. We'll look at the inverted check separately.

3. Doc wording. "The first encoded picture of the coded video sequence is the left view" is not quite accurate: the view follows the picture count from the start of the stream, so with --keyint 25 --no-open-gop the second CVS starts at frame 25 with the right view. "The first encoded picture is the left view" would be correct.

[danielcamposramos]

Thanks, Akilan, all three points are fixed.
Type 5 now takes the view from the frame count whenever interlace is enabled, so tff and bff both give L L R R with or without --field.
Our round 1 note was wrong about interlace without --field, since those input pictures are already fields.
The field regression keeps --preset veryfast and adds --b-adapt 2, and all four frame packing regression option lines pass with generated test footage.
The docs and CLI help now say the first encoded picture is the left view and keep the odd --seek note.
I also checked --keyint 25 --no-open-gop, and picture 25 starts the second CVS with the right view.
The Annex B parser and the FFmpeg decoder agree on the corrected field sequences, while progressive type 5 still gives L R L R.
Types 3 and 4, alpha, and streams without the option remain byte identical to bf495e5 in the comparison cases, and --num-views 2 still gives the same error with frame packing enabled.
The alpha auxiliary layer still has no frame packing SEI because it carries alpha rather than a view.
The branch is rebased on current master, with the readback tables, commands and comparisons in evidence/round2/.
Thank you too, @kirithika7, for merging #988.

[vunguyen1989]

Minor nit: writeFramePacking is declared about 60 lines before it's used (see screenshot).

Could we move it closer to the if (writeFramePacking) block for better readability? It can still be reused by isSei below.

[danielcamposramos]

@vunguyen1989 thank you, moved it next to its first use, just above if (writeFramePacking); isSei still sees it there.