AV2 Bitstream & Decoding Process Specification

1. Scope

This document specifies the Alliance for Open Media Video 2 (AV2) bitstream format and decoding process.

2. Terms and definitions

For the purposes of this document, the following terms and definitions apply:

AC coefficient

Any transform coefficient whose frequency indices are non-zero in at least one dimension.

ADST

Asymmetric Discrete Sine Transform.

AOMedia

Alliance for Open Media.

Atlas

A virtual 2D image associated with the decoded layers of a bitstream. The atlas can provide information on how to interpret, render, and utilize all such layers, depending on the application.

Base layer

The layer with obu_mlayer_id and obu_tlayer_id values equal to 0.

BAWP

Block Adaptive Weighted Prediction modifies inter prediction samples with a linear equation based on a scaling factor and an offset. The model parameters are based on observations from surrounding samples in the decoded frame and reference frame or the OrderHints distance of the decoded frame and reference frame.

Bitstream

The sequence of bits generated by encoding a sequence of frames.

Bit string

An ordered string with limited number of bits. The left most bit is the most significant bit (MSB), the right most bit is the least significant bit (LSB).

Block

A square or rectangular region of samples.

BRU

Backwards reference update allows an existing reference frame to be partially updated.

Byte

An 8-bit bit string.

Byte alignment

One bit is byte aligned if the position of the bit is an integer multiple of eight from the position of the first bit in the bitstream.

CCSO

Cross Component Sample Offset filter designed to modify both luma and chroma samples based on luma brightness and brightness gradient.

CDEF

Constrained Directional Enhancement Filter designed to adaptively filter blocks based on identifying the direction.

CDF

Cumulative distribution function representing the probability times 32768 that a symbol has value less than or equal to a given level.

Chroma

A sample value matrix or a single sample value of one of the two color difference signals.

NOTE: Symbols of chroma are U and V.

Coded frame

The representation of one frame before the decoding process.

Component

One of the three sample value matrices (one luma matrix and two chroma matrices) or its single sample value.

Compound prediction

A type of inter prediction where sample values are computed by blending together predictions from two reference frames (the frames blended can be the same or different).

DC coefficient

A transform coefficient whose frequency indices are zero in all dimensions.

DCT

Discrete Cosine Transform.

Decoded frame

The frame reconstructed out of the bitstream by the decoder.

Decoder

One embodiment of the decoding process.

Decoding process

The process that derives decoded frames from syntax elements, including any processing steps used prior to and for the film grain synthesis process.

Dequantization

The process in which transform coefficients are obtained by scaling the quantized coefficients.

Encoder

One embodiment of the encoding process.

Encoding process

A process not specified in this Specification that generates the bitstream that conforms to the description provided in this document.

Enhancement layer

A layer with either obu_mlayer_id greater than 0 or obu_tlayer_id greater than 0.

Embedded layer

A layer denoted by obu_mlayer_id.

Extended layer

A layer denoted by obu_xlayer_id.

Frame

The representation of video signals in the spatial domain, composed of one luma sample matrix (Y) and zero or two chroma sample matrices (U and V).

Frame context

A set of probabilities used in the decoding process.

GDF

Guided detail filter designed to selectively enhance details.

Inter coding

Coding one block or frame using inter prediction.

Inter frame

A frame compressed by referencing previously decoded frames and which may use intra prediction or inter prediction.

Inter prediction

The process of deriving the prediction value for the current frame using previously decoded frames.

IBP

Intra bi-prediction blends two different intra predictions together for a single block.

Independent Bitstream

Bitstream that contains no more than one value for the extended layer identifier.

Intra coding

Coding one block or frame using intra prediction.

Intra frame

A frame compressed using only intra prediction which can be independently decoded.

Intra prediction

The process of deriving the prediction value for the current sample using previously decoded sample values in the same decoded frame.

Inverse transform

The process in which a transform coefficient matrix is transformed into a spatial sample value matrix.

Key frame

An Intra frame which resets the decoding process when it is shown.

Layer

A set of tile group OBUs with identical obu_mlayer_id and identical obu_tlayer_id values.

LCR

Layer Configuration Record.

Level

A defined set of constraints on the values for the syntax elements and variables.

Loop filter

A filtering process applied to the reconstruction intended to reduce the visibility of block edges.

LSB

Least Significant Bit.

Luma

A sample value matrix or a single sample value representing the monochrome signal related to the primary colors.

NOTE: The symbol representing luma is Y.

MHCCP

Multi-hypothesis cross component prediction.

Mode info

Syntax elements sent for a block containing an indication of how a block is to be predicted during the decoding process.

Mode info block

A luma sample value block of size 4x4 or larger and its two corresponding chroma sample value blocks (if present).

Motion vector

A two-dimensional vector used for inter prediction which refers the current frame to the reference frame, the value of which provides the coordinate offsets from a location in the current frame to a location in the reference frame.

MSB

Most Significant Bit.

Multistream

Bitstream that contains two or more values for the extended layer identifier and at least one Multi Stream Decoder Operation OBU.

OBU

All structures are packetized in "Open Bitstream Units" or OBUs. Each OBU has a header, which provides identifying information for the contained data (payload).

OPS

Operating Point Set.

ORIP

Offset based refinement for intra prediction.

Parse

The procedure of getting the syntax element from the bitstream.

Prediction

The implementation of the prediction process consisting of either inter or intra prediction.

Prediction process

The process of estimating the decoded sample value or data element using a predictor.

Prediction value

The value, which is the combination of the previously decoded sample values or data elements, used in the decoding process of the next sample value or data element.

Profile

A subset of syntax, semantics and algorithms defined in a part.

Quantization parameter

A variable used for scaling the quantized coefficients in the decoding process.

Quantized coefficient

A transform coefficient before dequantization.

Raster scan

Maps a two dimensional rectangular raster into a one dimensional raster, in which the entry of the one dimensional raster starts from the first row of the two dimensional raster, and the scanning then goes through the second row and the third row, and so on. Each raster row is scanned in left to right order.

Reconstruction

Obtaining the addition of the decoded residual and the corresponding prediction values.

Reference

One of a set of tags, each of which is mapped to a reference frame.

Reference frame

A storage area for a previously decoded frame and associated information.

Reserved

A special syntax element value which may be used to extend this part in the future.

Residual

The differences between the reconstructed samples and the corresponding prediction values.

Sample

The basic elements that compose the frame.

Sample value

The value of a sample. This is an integer from 0 to 255 (inclusive) for 8-bit frames, from 0 to 1023 (inclusive) for 10-bit frames, and from 0 to 4095 (inclusive) for 12-bit frames.

Segmentation map

One 4-bit number per 4x4 block in the frame specifying the segment affiliation of that block. A segmentation map is stored for each reference frame to allow new frames to use a previously coded map.

Sequence

The highest level syntax structure of coding bitstream, including one or several consecutive coded frames.

Superblock

The top level of the block tree within a tile. All superblocks within a frame are the same size and are square. The superblocks may be 256x256 luma samples, 128x128 luma samples, or 64x64 luma samples. A superblock may contain multiple blocks, which may themselves be further subpartitioned, forming the block tree.

Switch Frame

An inter frame that can be used as a point to switch between sequences. Switch frames overwrite all the reference frames without forcing the use of intra coding. The intention is to allow a streaming use case where videos can be encoded in small chunks (say of 1 second duration), each starting with a switch frame. If the available bandwidth drops, the server can start sending chunks from a lower bitrate encoding instead. When this happens the inter prediction uses the existing higher quality reference frames to decode the switch frame. This approach allows a bitrate switch without the cost of a full key frame.

Syntax element

An element of data represented in the bitstream.

Temporal delimiter OBU

An indication that the following OBUs will have a different presentation/decoding time stamp from the one of the last frame prior to the temporal delimiter.

Temporal unit

A Temporal unit consists of all the OBUs that are associated with a specific, distinct time instant.

Temporal group

A set of frames whose temporal prediction structure is used periodically in a video sequence.

Tier

A specified category of level constraints imposed on the values of the syntax elements in the bitstream.

Tile

A rectangular region of the frame that can be decoded and encoded independently, although loop filtering across tile edges is still applied in some cases.

TIP

Temporally interpolated prediction.

Transform block

A rectangular transform coefficient matrix, used as input to the inverse transform process.

Transform coefficient

A scalar value, considered to be in a frequency domain, contained in a transform block.

TCQ

Trellis coded quantization adjusts the quantizer levels based on the parity of the decoded coefficients.

Uncompressed header

High level description of the frame to be decoded that is encoded without the use of arithmetic encoding.

WAIP

Wide Angle Intra Prediction.

WHT

Walsh Hadamard Transform.

3. Symbols

The specification makes use of a number of constant integers. Constants that relate to the semantics of a particular syntax element are defined in § 6 Syntax structures semantics.

Additional constants are defined below:

4. Conventions

4.1. General

The mathematical operators and their precedence rules used to describe this specification are similar to those used in the C programming language. However, the operation of integer division with truncation is specifically defined.

In addition, a length 2 array used to hold a motion vector (indicated by the variable name ending with the letters Mv or Mvs) can be accessed using either array notation (e.g., Mv[ 0 ] and Mv[ 1 ]), or by just the name (e.g., Mv). The only operations defined when using the name are assignment and equality/inequality testing. Assignment of an array is represented using the notation A = B and is specified to mean the same as doing both the individual assignments A[ 0 ] = B[ 0 ] and A[ 1 ] = B[ 1 ]. Equality testing of 2 motion vectors is represented using the notation A == B and is specified to mean the same as (A[ 0 ] == B[ 0 ] && A[ 1 ] == B[ 1 ]). Inequality testing is defined as A != B and is specified to mean the same as (A[ 0 ] != B[ 0 ] || A[ 1 ] != B[ 1 ]).

If a process specifies something happens for x = L..H, where x is a variable name and L and H are expressions, it means that the variable takes all integer values starting at L and going up to (and including) H.

When a variable is said to be representable by a signed integer with x bits, it means that the variable is greater than or equal to -(1 << (x-1)), and that the variable is less than or equal to (1 << (x-1))-1.

The key words “must”, “must not”, “required”, “shall”, “shall not”, “should”, “should not”, “recommended”, “may”, and “optional” in this document are to be interpreted as described in RFC 2119.

4.2. Arithmetic operators

4.3. Logical operators

4.4. Relational operators

4.5. Bitwise operators

4.6. Assignment

4.7. Mathematical functions

The following mathematical functions (Abs, Clip3, Clip1, Min, Max, Round2 and Round2Signed) are defined as follows:

$$ \text{Abs}(x) = \begin{cases} x; & x \geq 0\\ -x; & x < 0 \end{cases} $$

$$ \text{Clip1}(x) = \text{Clip3}(0, 2^{BitDepth}-1, x) $$

$$ \text{Clip3}(x,y,z) = \begin{cases} x; & z < x \\ y; & z > y \\ z; & \text{otherwise} \end{cases} $$

$$ \text{Min}(x, y) = \begin{cases} x; & x \leq y \\ y; & x > y \end{cases} $$

$$ \text{Max}(x, y) = \begin{cases} x; & x \geq y \\ y; & x < y \end{cases} $$

$$ \text{Round2}(x,n) = \left\lfloor \frac{x+2^{n-1}}{2^n} \right\rfloor $$

$$ \text{Round2Signed}(x,n) = \begin{cases} \text{Round2}(x,n); & x \geq 0\\ -\text{Round2}(-x,n); & x < 0 \end{cases} $$

The definition of Round2 uses standard mathematical power and division operations, not integer operations. An equivalent definition using integer operations is:

Round2( x, n ) {  if ( n == 0 )    return x  return (x + (1 << (n - 1)) ) >> n}

The FloorLog2(x) function is defined to be the floor of the base 2 logarithm of the input x.

The input x will always be an integer, and will always be greater than or equal to 1.

This function extracts the location of the most significant bit (MSB) in x.

An equivalent definition (using the pseudo-code notation introduced in the following section) is:

FloorLog2( x ) {  s = 0  while ( x != 0 ) {    x = x >> 1    s++  }  return s - 1}

The GetMsb(x) function is the same as FloorLog2, except that an input of 0 is also allowed.

The function is defined as follows:

GetMsb( x ) {  if ( x==0 ) {    return 0  }  return FloorLog2( x )}

The CeilLog2(x) function is defined to be the ceiling of the base 2 logarithm of the input x (when x is 0, it is defined to return 0).

The input x will always be an integer, and will always be greater than or equal to 0.

This function extracts the number of bits needed to code a value in the range 0 to x-1.

An equivalent definition (using the pseudo-code notation introduced in the following section) is:

CeilLog2( x ) {  if ( x < 2 )    return 0  i = 1  p = 2  while ( p < x ) {    i++    p = p << 1  }  return i}

4.8. Method of describing bitstream syntax

The description style of the syntax is similar to the C programming language. Syntax elements in the bitstream are represented in bold type. Each syntax element is described by its name (using only lower case letters with underscore characters) and a descriptor for its method of coded representation. The decoding process behaves according to the value of the syntax element and to the values of previously decoded syntax elements. When a value of a syntax element is used in the syntax tables or the text, it appears in regular (i.e., not bold) type. If the value of a syntax element is being computed (e.g., being written with a default value instead of being coded in the bitstream), it also appears in regular type (e.g., tile_size_minus_1).

In some cases the syntax tables may use the values of other variables derived from syntax elements values. Such variables appear in the syntax tables, or text, named by a mixture of lower case and upper case letters and without any underscore characters. Variables starting with an upper case letter are derived for the decoding of the current syntax structure and all depending syntax structures. These variables may be used in the decoding process for later syntax structures. Variables starting with a lower case letter are only used within the process from which they are derived. (Single-character variables are allowed.)

Constant values appear in all upper case letters with underscore characters (e.g. MI_SIZE).

Constant lookup tables appear as words (with the first letter of each word in upper case, and remaining letters in lower case) separated with underscore characters (e.g., Block_Width[…]).

Hexadecimal notation, indicated by prefixing the hexadecimal number by 0x, may be used when the number of bits is an integer multiple of 4. For example, 0x1a represents a bit string 0001 1010.

Binary notation is indicated by prefixing the binary number by 0b. For example, 0b00011010 represents a bit string 0001 1010. Binary numbers may include underscore characters to enhance readability. If present, the underscore characters appear every 4 binary digits starting from the LSB. For example, 0b11010 may also be written as 0b1_1010.

A value equal to 0 represents a FALSE condition in a test statement. The TRUE condition is represented by any value not equal to 0.

The following table lists examples of the syntax specification format. When syntax_element appears (in bold font), it specifies that this syntax element is parsed from the bitstream.

4.9. Functions

Bitstream functions used for syntax description are specified in this section.

Other functions are included in the syntax tables. The convention is that a section is called _syntax_ if it causes syntax elements to be read from the bitstream, either directly or indirectly through subprocesses. The remaining sections are called _functions_.

The specification of these functions makes use of a bitstream position indicator. This bitstream position indicator locates the position of the bit that is going to be read next.

get_position( ): Return the value of the bitstream position indicator.

init_symbol( sz ): Initialize the arithmetic decode process for the symbol decoder with a size of sz bytes as specified in § 8.2.2 Initialization process for symbol decoder.

exit_symbol( ): Exit the arithmetic decode process as described in § 8.2.4 Exit process for symbol decoder (this includes reading trailing bits).

4.10. Descriptors

4.10.1. General

The following descriptors specify the parsing of syntax elements. Lower case descriptors specify syntax elements that are represented by an integer number of bits in the bitstream; upper case descriptors specify syntax elements that are represented by arithmetic coding.

4.10.2. f(n)

Unsigned n-bit number appearing directly in the bitstream. The bits are read from highest to lowest. The parsing process specified in § 8.1 Parsing process for f(n) is invoked, and the syntax element is set equal to the return value.

4.10.3. uvlc()

Variable-length unsigned number appearing directly in the bitstream. The parsing process for this descriptor is specified below:

It is a requirement of bitstream conformance that leadingZeros is less than 32 when this function returns.

Note: This means that the largest value that can be returned by a uvlc() descriptor is ( 1 << 32 ) - 2.

4.10.4. le(n)

Unsigned little-endian n-byte number appearing directly in the bitstream. The parsing process for this descriptor is specified below:

4.10.5. leb128()

Unsigned integer represented by a variable number of little-endian bytes.

Note: This syntax element will only be present when the bitstream position is byte aligned.

In this encoding, the most significant bit of each byte is equal to 1 to signal that more bytes should be read, or equal to 0 to signal the end of the encoding.

A variable Leb128Bytes is set equal to the number of bytes read during this process.

The parsing process for this descriptor is specified below:

It is a requirement of bitstream conformance that the value returned from the leb128 parsing process is less than or equal to (1 << 32) - 1.

leb128_byte contains 8 bits read from the bitstream. The bottom 7 bits are used to compute the variable value. The most significant bit is used to indicate that there are more bytes to be read.

It is a requirement of bitstream conformance that the most significant bit of leb128_byte is equal to 0 if i is equal to 7. (This ensures that this syntax descriptor never uses more than 8 bytes.)

Note: There are multiple ways of encoding the same value, depending on how many leading zero bits are encoded. There is no requirement that this syntax descriptor uses the most compressed representation. This can be useful for encoder implementations by allowing a fixed amount of space to be filled in later when the value becomes known.

Note: Only 5 bytes (providing 35 bits) are needed for this syntax descriptor because the bitstream conformance requirement limits the return value to 32 bits (7 bits in each of the first 4 bytes, and 4 bits in the 5th byte).

4.10.6. su(n)

Signed integer converted from an n-bits unsigned integer in the bitstream. (The unsigned integer corresponds to the bottom n bits of the signed integer.) The parsing process for this descriptor is specified below:

4.10.7. ns(n)

Unsigned encoded integer with maximum number of values n (i.e., output in range 0..n-1).

This descriptor is similar to f(CeilLog2(n)), but reduces wastage incurred when encoding non-power of two value ranges by encoding 1 fewer bits for the lower part of the value range. For example, when n is equal to 5, the encodings are as follows (full binary encodings are also presented for comparison):

The parsing process for this descriptor is specified as:

The abbreviation ns stands for _non-symmetric_. This encoding is non-symmetric because the values are not all coded with the same number of bits.

4.10.8. tu(mx)

Integer in the range 0 to mx using truncated unary encoding (a series of zero or more 1s followed by a single 0, except that the final 0 is omitted if the maximum is reached).

The parsing process for this descriptor is specified below:

4.10.9. rg(n)

Integer with Rice-Golomb coding with parameter n (a fixed length coding of the n least significant bits preceded by a unary encoding of the most significant bits).

The parsing process for this descriptor is specified below:

It is a requirement of bitstream conformance that this descriptor never returns a value less than 0.

4.10.10. L(n)

Unsigned arithmetic encoded n-bit number encoded as n flags (a _literal_). The flags are read from highest to lowest. The syntax element is set equal to the return value of read_literal( n ) (see § 8.2.5 Parsing process for read_literal for a specification of this process).

4.10.11. S()

An arithmetic encoded symbol coded from a small alphabet of at most 8 entries.

The symbol is decoded based on a context-sensitive CDF (see § 8.3 Parsing process for CDF encoded syntax elements for the specification of this process).

4.10.12. NS(n)

Unsigned arithmetic encoded integer with maximum number of values n (i.e., output in range 0..n-1).

This descriptor is the same as ns(n), except the underlying bits are coded arithmetically.

The parsing process for this descriptor is specified as:

5. Syntax structures

5.1. General

This section presents the syntax structures in a tabular form. The meaning of each of the syntax elements is presented in § 6 Syntax structures semantics.

5.2. OBU syntax

5.2.1. General OBU syntax

where some helper functions used to identify collections of OBU types are specified as:

is_tip_frame() {    return obu_type == OBU_LEADING_TIP || obu_type == OBU_REGULAR_TIP}

is_sef() {    return obu_type == OBU_LEADING_SEF || obu_type == OBU_REGULAR_SEF}

is_tile_group() {    return obu_type == OBU_LEADING_TILE_GROUP ||            obu_type == OBU_REGULAR_TILE_GROUP ||            obu_type == OBU_CLOSED_LOOP_KEY ||            obu_type == OBU_OPEN_LOOP_KEY ||           obu_type == OBU_SWITCH ||           obu_type == OBU_RAS_FRAME}

5.2.2. OBU header syntax

5.2.3. Trailing bits syntax

5.2.4. Byte alignment syntax

5.3. Reserved OBU syntax

Note: Reserved OBUs do not have a defined syntax. The obu_type reserved values are reserved for future use. Decoders should ignore the entire OBU if they do not understand the obu_type. The last byte of the valid content of the payload data for this OBU type is considered to be the last byte that is not equal to zero. This rule is to prevent the dropping of valid bytes by systems that interpret trailing zero bytes as a continuation of the trailing bits in an OBU. This implies that when any payload data is present for this OBU type, at least one byte of the payload data (including the trailing bit) shall not be equal to 0.

5.4. Sequence header OBU syntax

5.4.1. General sequence header OBU syntax

5.4.2. Sequence partition config syntax

5.4.3. Sequence segment config syntax

5.4.4. Sequence intra config syntax

5.4.5. Sequence inter config syntax

5.4.6. Sequence screen content config syntax

5.4.7. Sequence transform quant entropy config syntax

5.4.8. Segment information syntax

5.4.9. Sequence filter config syntax

5.4.10. User defined QM syntax

where Fundamental_Tx_Size (which gives the order of quantization matrices) is specified as:

Fundamental_Tx_Size[ 3 ] = { TX_8X8, TX_8X4, TX_4X8 }

5.4.11. Color config syntax

5.4.12. Timing info syntax

5.4.13. Decoder model info syntax

5.4.14. Operating parameters info syntax

5.5. Temporal delimiter OBU syntax

Note: The temporal delimiter has an empty payload.

5.6. Multi stream decoder operation OBU syntax

5.7. Multi frame header OBU syntax

5.8. Layer config record OBU syntax

5.8.1. LCR global info syntax

5.8.2. LCR local info syntax

5.8.3. LCR aggregate profile tier level information syntax

5.8.4. LCR sequence profile tier level information syntax

5.8.5. LCR global payload syntax

5.8.6. LCR xlayer info syntax

5.8.7. LCR rep info syntax

5.8.8. LCR embedded layer info syntax

5.8.9. LCR xlayer color info syntax

5.9. Atlas segment info OBU syntax

5.9.1. Atlas label segment info syntax

5.9.2. Atlas region info syntax

5.9.3. Atlas region to segment mapping syntax