Introduction Last updated: 2019-06-01

Overview

Closed captions were created to assist in comprehension. Caption Inspector extracts and decodes Closed-Captions (CC) from various Video or Caption File Formats to validate and ensure Closed Captioning is available.

The Caption Inspector project builds a C library, C executable, and Docker image that can be used to extract and decode CC.

Supported Formats

Caption Inspector Supports CEA-608 and CEA-708 in MPEG-2 and MPEG-4 (.mpg, .ts, and .mp4 containers), MCC (MacCaption Closed Captions), and SCC (Scenarist Closed Captions) files. For this, FFMPEG is required*.

OVERVIEW DIAGRAM 1.0

Caption Inspector Overview

Decoding and Examining the Closed Captioning

Use Caption Inspector to validate video content adheres to expected Specifications, Compliance Policies and/or laws.

Television Decoder Circuitry Act&

Communications and Video Accessibility Act

---

"Closed captioning has been provided on a voluntary basis by many movie producers, studios, and distributors for movie videos and DVDs produced for sale or rent."

"The Internet is the new frontier for captioning. The law is generally clear that government agencies must make their websites accessible. The 21st Century Communications and Video Accessibility Act (“CVAA”) expanded the scope of devices that must display captions under the Television Decoder Circuitry Act to all video devices that receive or display video programming transmitted simultaneously with sound, including those that can receive or display programming carried over the Internet. The NAD was a leading member of the Coalition of Organizations for Accessible Technology (“COAT”) which was instrumental in drafting and getting CVAA passed."
- National Association of the Deaf

REFERENCES: https://www.nad.org/resources/technology/captioning-for-access/when-is-captioning-required/ https://en.wikipedia.org/wiki/Closed_captioning#Legislative_development_in_the_U.S.

Caption Inspector decodes the Closed Captioning Data into two basic types of output files.

1. The first type (demultiplexed) has the Closed Captioning Data demultiplexed into output files for each service/channel with closed caption tags and text in the file. Simply put, demultiplexed Closed Captioning Data exists in external files.
2. The second type (multiplexed) has all of the Closed Captioning Data decoded in place, still multiplexed together.

Developer Reference

As a extra benefit the Caption Inspector Codebase can be utilized as a reference. Developers can leverage the implementation of the Closed Captioning code without knowing the exact Specifications by examining the output and logs.

Caption Inspector has a plugin pipeline architecture that can be configured in various ways and allows the user to add new plugins to perform various transformations. Currently, the following are the main use cases that the Caption Inspector Software covers:

• Pulling Captions from a Video Asset and writing them to an MCC Caption File, a CEA-608 Decode File, a CEA-708 Decode File, and a Closed Caption Descriptor Decode File.
• Decoding a MCC Caption file and writing the decoded captions into a CEA-608 Decode File, a CEA-708 Decode File, and a Closed Caption Descriptor Decode File.
• Decoding a SCC Caption file and writing the decoded captions into a MCC Caption File, a CEA-608 Decode File, and a Closed Caption Descriptor Decode File.

The Expected Quality of Captions

Today, there is a gap in the process of Closed Caption Quality production caused by errors within the CC content. Prior to this software, content delivered to Distributors could not be easily validated without going through (watching) all of the video content, frame by frame. This makes checking content impractical, and its often overlooked when distributed.

This has caused the content to be delivered with Closed Caption errors. As a result, this content is then provided to consumers with errors, putting the Distributors at fault for bad or faulty content. Regardless of where the responsibility lies, whether the Content Provider, or the Distributor, with Caption Inspector each video can now be validated by each party in a relatively short period of time. Thus, Content Providers can be certain of what they deliver, and Distributors do not have to accept faulty content, and be at risk of providing a low-quality product.

Industry Accepted Guidelines

The following are the accepted industry guidelines and best practices for quality: (https://dcmp.org)

1. Accuracy Errorless captions are the goal for each production
2. Consistancy Uniformity in style and presentation of all captioning features is crucial for viewer understanding.
3. Clarity A complete textual representation of the audio, including speaker identification and non-speech information, provides clarity
4. Readable Captions are displayed with enough time to be read completely, are in synchronization with the audio, and are not obscured by (nor do they obscure) the visual content.
5. Equality of Accessibility Equal access requires that the meaning and intention of the material is completely preserved.

How is it Used

Caption Inspector is a Command Line Interface (CLI) tool that takes a video or caption file as input, extracts the embedded data, and outputs this to a text file, formatted in either a standard `.mcc`, or `.scc` file format. Yet, this tool is not simply an extractor of embedded data, but a primary purpose of this tool is to validate the embedded transcript for errors.

Processing content will create several files, which includes:

• Content Descriptor Document (CDD) File. A single CDD `.ccd file` shows each frame's binary caption data as hex values followed by decoded characters and control-codes (much like how the output of the Linux `xxd` command shows the hex values followed by ascii).
• And, Individual .608 and .708 files are also generated for each channel/service, separating the binary caption data into frame-indexed, service-specific, human-readable text and control-codes for analysis.

Prerequisites

Basic Software and computer skills are required for installation and usage.

Prerequisites to Building the Caption Inspector Executable

• FFMPEG*
  Caption Inspector requires FFMPEG to be installed on your machine. Please download the appropriate version of FFMPEG from the website and install it on your machine. Caption Inspector is known to run with FFMPEG Version 4.0.2..
  
  Get   https://ffmpeg.org/download.html

• Drop Frame
  While not required, Caption Inspector leverages a tool called MediaInfo to determine whether or not an asset is, or is not, Drop Frame. You can download the command line version of MediaInfo. Caption Inspector is known to run with MediaInfo Version 18.12.
  
  Get   https://mediaarea.net/en/MediaInfo/Download

Developer Overview

The Caption Inspector project builds a C library, C executable, and Docker image that can be used to extract and decode Closed-Captions from various Video or Caption File Formats. Caption Inspector Supports CEA-608 and CEA-708 in MPEG-2 and MPEG-4 (.mpg, .ts, and .mp4 containers), MCC (MacCaption Closed Captions), and SCC (Scenarist Closed Captions) files.

The second type has all of the Closed Captioning Data decoded in place, still multiplexed together. Beyond the current use case of decoding and examining the Closed Captioning, the Caption Inspector Codebase can be leveraged as a reference implementation of the Closed Captioning Specifications which would allow a developer to not have to understand the specifics of the specs just to access, manipulate, and modify the encoded Closed Captioning.

Installation & Start Up

Caption Inspector Executable

This is a one-time installation. When the Executable has been installed you will only need to start up your server to be able to use the tools. If you shutdown, and restart your computer, you will need to restart your server from within Docker, unless you configure Docker to automatically restart at Login.

Future updates will require a reinstallation along with steps we will at that time provide and will likely be a similar process. This step is a one-time set up process. Building the Caption Inspector Executable can be done at the root level, or at the "src level". Once it has been built, Caption Inspector can be used to decode the captions from a test file.

Building and Running Locally

Caption Inspector requires FFMPEG to be installed on your machine. Please download the appropriate version of FFMPEG from here and install it in your machine.

While not required, Caption Inspector leverages a tool called MediaInfo to determine whether or not an asset is, or is not, Drop Frame. You can download the command line version of MediaInfo from https://mediaarea .net/en/MediaInfo/Download

Using the CLI

make caption-inspector
./caption-inspector -h
./caption-inspector -o .test/media/BigBuckBunny_256x144-24fps.ts
./caption-inspector -o .test/media/Plan9fromOuterSpace.scc -f 2400
./caption-inspector -o .test/media/NightOfTheLivingDead.mcc

Building and Running in a Docker Container

Obviously you need docker running on your local machine to build. Building inside of a docker image will remove the need to install any dependencies, but comes at the expense of a slightly more complicated command line execution.

Using the CLI with Docker

make docker
docker run -t caption-inspector -h
docker run -tv $(pwd):/files caption-inspector -o /files /files/test/media/BigBuckBunny_256x144-24fps.ts
docker run -tv $(pwd):/files caption-inspector -o /files /files/test/media/Plan9fromOuterSpace.scc -f 2400
docker run -tv $(pwd):/files caption-inspector -o /files /files/test/media/NightOfTheLivingDead.mcc

In the docker run command, your current working directory will be remapped to /files inside of the container, so you will need to prefix your input and output paths to that so that it can place the files in the correct spot. For this example, the output file is located in the current directory ./ and the input file is located in a directory underneath the current directory, specifically ./test/media/*.

Regression Testing

While the testing is not complete yet, there are several Unit Tests, Integration Tests, and System Tests that run on the Caption Inspector Codebase. These are intended to be run as integration tests and verify that nothing has broken as a result of a change. They are run as part of a pull request, but for debugging purposes they can also be run locally (even in an IDE) before a pull request is issued. The easiest way to run these is inside of docker, as the dependencies are handled for you. But if you need to run them with an IDE/debugger, they can also be run from the command line.

To run the regression tests from inside of docker, simply call make docker-test at the root directory. This will build the docker image and run the tests. The output generated by this activity can give information about any test case that does not pass, but even more useful is an HTML file that is the output of the test. It contains all of the tests, and their statuses. It is in the root directory and takes the form <date>__<time>_test_output.html. The build system will attempt to open it in a browser on your local machine. If there is a problem with that operation you can open the file manually from a browser.

To run the regression tests from the command line, you need to install Xunit Viewer, which can be found Xunit Viewer . Xunit Viewer is the application that takes the Xunit XML and converts it into a beautified HTML format. Once you have Xunit Viewer installed, or if you ignore it, you just need to build and run the regression tests from the test directory.

cd test
make test

The tests are a mix of C and Python, depending on which made more sense for the specific test. The build system will aggregate all of the results from the tests in both languages.

Leveraging the Caption Inspector Functionality from Python

C was chosen as the language for Caption Inspector because of interoperability with FFMPEG and ease of implementation of the specifications. While that choice made sense, there are lots of reasons to want to access this functionality as a library from a higher level language than C. To that end Python bindings, using CTypes, were added to the C functionality which can be compiled into a shared library and referenced from Python code. To use this functionality you just need to build the shared library using the command: make sharedlib. Then leverage the file python/cshim.py inside of your Python code, making sure that it knows where to find the shared library with the Caption Inspector Code.

The Caption Inspector code that makes this possible, and the library it generates, can likely be reused for other high level languages such as Java/JNI, Golang/Cgo, etc.

Caption Inspector Artifacts (FILES)

There are several different types of artifacts that can be generated by Caption Inspector. They get output when an artifact directory gets specified as a command line argument to Caption Inspector.

File Types

Understanding Closed Caption Descriptor

Closed Caption Descriptor File (.ccd)

Caption Inspector will output a single Closed Caption Descriptor File for each asset that it processes. The Closed Caption Descriptor Files are intended to be annotated hex dumps of the closed captioning descriptors located on every frame of video or every line of an MCC file.

Unlike the other artifacts, the CCD files are decoded inline, and all channels and services have been left "multiplexed" together and have traceability back to the original/unmodified hex that was extracted from the asset or caption file. The Closed Caption Descriptor File contains a decoded binary representation of Closed Captioning Data.

CCD Details

Each grouping of text immediately following a timestamp represents the Closed Captioning Data - `cc_data()` that would be associated with a frame of video, or a line from an MacCaption (MCC) file. Each "line" contains the same groups of elements: Timestamp, Hex Data, Decoded Data, Supplemental Decoded Data, and Text (if it exists in the line). These elements can describe both CEA-608 and CEA-708 data contained in the Closed Captioning Descriptors.

About Closed Caption Windows

Timestamp

On the leftmost portion of the document is the timestamp. Each Closed Caption Data (CCD) packet has a timestamp, which corresponds with the frame of the asset containing the CCD packet. For readability sake, the timestamp is only printed out when specific Global Control Codes are encountered. There are three types of timestamps: Frame Number, Drop Frame Number, and Millisecond. This refers to the last number, and the symbol that separates it from the second to last number.

There are three types of timestamps:

• Frame Number
• Drop Frame Number
• Millisecond

Drop Frame Number Timestamps format
`<Hour>:<Minute>:<Second>;<Frame>`

Frame Number Timestamps format
`<Hour>:<Minute>:<Second>:<Frame>`

Millisecond Timestamps format
`<Hour>:<Minute>:<Second>,<Millisecond>`
 

The Frame Number vs. Drop Frame Number is differentiated by using either a colon `:` for Frame or a semi-colon`;` for Drop Frame.

For a description of what Drop Frame is, and why we have it see: http://www.bodenzord.com/archives/79

Aggregated Text

If there is text detected in the line, it will be pulled together and gets labeled with the Channel or Service Number that it is associated with. This is done to easily be able to see what the text spells out.

The Basis for Reading Output - Hex Data

01. { cc_valid  }{cc_type}:{cc_data_1}{cc_data_2}
02. { F | P | X  }{  S  | D }:

The `cc_valid` field determines whether the subsequent fields are to be considered valid and thus be interpreted. The value will either be `F` for Valid Line 21, `P` for Valid DTVCC, or `X` for Invalid.

The `cc_type` field determines the type of data that will be in `cc_data_1` and `cc_data_2`. This field can have the following values:

'1': Line 21 Field 1 Data,
'2': Line 21 Field 2 Data,
    'S': DTVCC Packet Data Start,
    'D': DTVCC Packet Data.

The fields `cc_data_1` and `cc_data_2` are the actual two bytes of payload data, which are represented in hexadecimal.

Decoded Data and Supplemental Decoded Data

Decoded Data and Supplemental Decoded Data are intended to be the inline decode of the corresponding Hex Data using the 608 and 708 Specs. The Decoded Data is a high level decode of the code referenced by the hex value, and the supplemental decoded data describes the flags and values referenced by the hex value.

Supplemental decoded data plays a larger role in 708 because more information is crammed into bit fields in the magic numbers.

CEA-608

NULL Data  (CEA-608)

CEA-608 tends to have its fill data be NULL Data. NULL Data is just `0x00` with a parity bit in the msb position. Since this is placed in specific to 608, it gets decoded to NULL (as opposed to 708 NULL Data, which is shown as a blank line).

Global Control Codes  (CEA-608)

There is not much decoding to do around Global Control Codes in CEA-608. Basically, it is just the Channel Number that gets decoded and added to the Decoded Data. The Supplemental Decoded Data field is basically used to try to spell out the control code as much as space permits, rather than using the code that is found in the Decoded Data.

These commands apply to the Window, or the caption inside the Current Window.

Preamble Access Codes (PAC)  (CEA-608)

PACs come in two types, Address PACs which describe where the cursor should be located, and Style PACs which describe what style/color should be used when writing to the screen. The Decoded Data field denotes that the decoded code is a PAC, and which channel it is associated with. The Supplemental Decoded Data field spells out the Address and Style information of the PAC.

Mid Row Style Change Code  (CEA-608)

Midrow Style Changes come in two types, Foreground and Background Style Change. Additionally, the Foreground

MidRow Style Change can indicate that the text is Partially Transparent, and the Background Midrow Style Change can indicate that the text is underlined.

Tab Control Code  (CEA-608)

Tab Controls, in combination with an Address PAC can point the cursor to anywhere inside of the text window.

CEA-708

Packet Structure  

CEA-708 is able to support 64 services, rather than the 4 channels in 608, in part by defining a variable length packet structure and then associating each group of packets to a specific service.

To make the start of the packet structure stick out in the decoded data, it has an arrow on the Packet Start: `<-`.

Spurious Data  (CEA-708)

When a byte is unexpected, it gets printed out in the below format. For this example, two unexpected 0 bytes were encountered.

Set Current Window  (CEA-708)

Set Current Window specifies the window to which all subsequent window style and pen commands are directed.

Define Window  (CEA-708)

Define Window creates a window and initializes the window with the parameters listed in the command.

Display Windows  (CEA-708)

Display Windows causes the specified windows to be visible.

708 Display Window  

Clear Windows  (CEA-708)

Delete Windows  (CEA-708)

Delete Windows removes all specified windows.

Hide Windows  (CEA-708)

Hide Windows causes the specified windows to be removed from the screen.

Toggle Windows  (CEA-708)

Toggle Windows causes all specified windows to toggle whether they are displayed or hidden.

Set Window Attributes  (CEA-708)

Set Window Attributes assigns the specified style attributes to the current window.

Set Pen Attributes  (CEA-708)

Set Pen Attributes assigns pen style attributes for the currently defined window. Text written to the current window will have the attributes specified by the most recent Set Pen Attributes command written to the window.

Set Pen Color  

Set Pen Color assigns the pen color attributes for the current window. Text written to that window will have the color attributes specified by the most recent Set Pen Color command.

Set Pen Location  (CEA-708)

Set Pen Location repositions the pen cursor for the current window, as specified by the current window ID.

Justification Rules  (CEA-708)

When the window justification type is 'left' the next group of text written to the current window will start at the specified row and column, and justification will be ignored.

When the window justification type is not left and the print direction is left-to-right or right-to-left, the column parameter shall be ignored.

Windows - CEA-708

When the window justification type is not left and the print direction is top-to-bottom or bottom-to-top, the row parameter shall be ignored.

When the window justification type is not left, text shall be formatted based upon the current window justification type.

Delay  (CEA-708)

Delay suspends interpretation of the command input buffer. Once the delay expires captioning resumes.

Delay Cancel (CEA-708)

Delay Cancel terminates an active Delay command and captioning resumes.

Reset  (CEA-708)

Reset command reinitializes the service for which it is received.

Reserved (CEA-708)

Reserved Codes are just that, Reserved Codes. They are not believed to be used.

Reference Tables

Window Commands

CEA-608/CEA-708 Codes

The Hex Data specifies the hexadecimal equivalents for Channels 1 through 4 found in the SCC file, using the standard ISO 8859-0 Character Set. Decoded Data refers to Channel Number (1 - 4).

Glossary of Terms

Resources