Introduction to Netflix Quality Control (QC)

Purpose: 

This document provides an overview of the three different types of QC at Netflix. The aim is to provide our delivery partners with an understanding of our QC processes.  This document also outlines common asset failures and provides further context to Backlot redelivery requests, which is a reflection of our commitment to providing a great customer experience. 

Types of QC: 

  • Catalog QC
    • Roles & Responsibilities:  Protects the customer experience for all content. Examines transcoded material on our internal web player.
  • Master QC
    • Roles & Responsibilities: Ensures the best possible quality for Netflix Originals. Checks the technical quality of A/V and audio assets.  Examines source files on QC vendors’ systems.
  • Localization QC
    • Roles & Responsibilities: Ensures the best possible quality for Netflix Originals. Enforces style guides & translation consistency.  Examines source files in our internal tool, QC 2.0.

 

Catalog QC

Catalog QC identifies issues that would be disruptive to the streaming experience.  The Catalog QC team takes the approach of simulating the customer experience.  Focusing on: 

  • Issues that would prevent consumption (e.g. the incorrect content being delivered)
  • Issues that noticeably reduce the quality of the title (translation issues, typos, video artifacts)
  • Maintain consideration for licensed content (varying styles)

Delivery & Ingest:

The delivery/ingest process for Licensed Content going through Catalog QC consists of three steps: Asset Upload, Auto QC/Inspection as a Service (IaaS), and Manual QC.  The following flow chart highlights this process:

 1_CatalogQC-WorkflowChart.png

Types of Catalog QC: 

  • Spot QC:
    • Consists of a Spot Check at: 
      • First 2 minutes
      • 1min: Find the credits
      • 1min: 50%
      • 1min: 75%
      • Last 2 minutes

2_SpotQC.png

  • Full QC:
    • Entire asset is reviewed
    • Performed for Netflix Originals and high profile content (at Netflix’s discretion)

3_FullQC.png


Common Catalog QC Errors: 

Video Audio Timed Text

Burned In Subtitles
(Lifestyle Content)

Requires FN

Typos

Segment bumper/recap
(Anime)

Censored Content
(Lifestyle Content)

Translation Missing

Blended Frames
(Telecine/broadcast)

Incorrect Language

Positioning

Censored Content

Extra Content

Objective Translation

Extra Content

Sync

Wrong Frame Rate

 

Master QC

Master QC validates Source Asset quality against our Netflix Originals Delivery Specifications and checks for content consistency.  Master QC is performed exclusively on Netflix Original content.  Master QC operators perform multiple full linear reviews of source assets, and flag issues that are inconsistent in the context of the program itself.

Master QC is performed on the following asset types in a unity environment (e.g. 4K is watched in 4K; HD is watched in HD): 

  • Primary A/V IMF/ProRes
  • Secondary Dubbed Audio

Error Checks

Master QC Error Codes can be categorized as follows:

  • Technical Specifications: consisting of Framing Issues, Aspect Ratio Issues, Image Levels, Audio Loudness, Peak Levels, Formatting, and more.
  • Content Issues: consisting of Visible Production Equipment & Crew,  Dead Pixels, Dropped/Duplicate Frames, Audio Ticks/Pops, Burned In Subs, etc. 

Ultimately, Master QC checks and corrects inconsistencies introduced during the production and post-production processes.  Issues are checked in a full linear QC and in a frame-by-frame capacity if need be. 

Master QC NPVs

The Majority of Netflix Originals are QCed by a Netflix Preferred Vendor. All error codes and reports are verified internally, validating the actionability of a flag.  Actionable reports are rejected back to both Fulfillment and Content Partners.

Using a Master QC NPV prior to delivery increases the likelihood of QC approval at Netflix. 

Established rates are available on our NPV Website

Why Master QC? 

  • Flag and correct issues that could negatively impact the customer experience
  • Flag and correct issues that detract from the original creative intent.  It is unlikely that a boom mic, tape mark, or crew member were intended to be in the shot. 
  • Ensure adherence to Netflix Originals Delivery Specifications and technical standards. 

Common Master QC Errors: 

  1. Pixel Error/Stain
  2. Animation Error
  3. Frame Error
  4. Visible Production Equipment
  5. Compression Artifacts
  6. Compositing Errors
  7. Audio Tick
  8. Luminance Shift
  9. Quantization
  10. Ident - Peak Levels Low

*For a full list of Master QC Error Codes and their corresponding severity ratings, please refer to the Mezzanine QC Errors & Severity document located in the Master QC Overview document.

QC Reports - Error Code Severity Ratings & Action Items: 

  • FYI - Something that we would like to draw attention to that is not necessarily an issue -Will not trigger a redelivery request - No action necessary.
  • Needs Review. Subjective issues that may not be detected by the customer (subjective translation and potential pronunciation issues) - Will trigger a redelivery request - Studio is expected to review, but we trust the studio to exercise their best judgment in addressing. Netflix will not review or approve any studio responses.
  • Needs Fix - Objective issues with the translation or technical aspects of a file that customers would notice (incorrect Key Names & Places "KNP" terms, objective translation issues that would impact customer experience, and other issues that would negatively impact quality and/or do not preserve creative intent) - Will trigger a redelivery request - Studio/NPV must implement fixes and redeliver.
  • Blocker - Issues that render the file unable to be consumed or would reflect very poorly on Netflix (wrong language, missing dubs, timing not in sync, offensive/derogatory language) - Will trigger a redelivery request - Studio/NPV must fix and redeliver in priority, before launch.


Localization QC

Localization QC is a linguistic QC that qualifies translation quality, consistency and style guide conformance. Reserved for Netflix Original content, this process involves a QC operator suggesting changes and a verifier reviewing and implementing them accordingly.

Localization QC is performed on the following asset types:

  • Dub & AD Scripts
  • Subtitles
  • Forced Narratives

Localization QC Workflow - QC 2.0

Timed-Text Assets are QC'd in our internal tool, QC 2.0.   Dialogue events are lined up with waveforms to verify sync and a linguistic QC is performed.

4_QC_2.0_Workflow.png

Fixes to a timed-text file are made directly in the tool and changes are shared back to the Fulfillment Partner in the form of Localization QC Report in Backlot:

5_QC_2.0_sourcerequestdetail.png

Following this "Link to the Localization QC Report" will provide access to The Change Log.  The Change Log details all fixes that the Localization QC team made to this file, as shown below: 

6_ChangeLog_QC20.png

Localization QC - Metrics Evaluation: 

Error rates for each asset type are defined as follows.  

  • Subtitle error rate:
    • Events with errors / total # of events
  • Language error rate:
    • Events with translation errors / total # of events
  • Technical error rate:
    • Events with technical errors / total # of events
  • Audio error rate:
    • Number of audio flags / total run time

These metrics are accumulated and used to drive conversations with Fulfillment Partners who are delivering assets for Netflix Originals undergoing Localization QC. 


Common Localization QC Errors:

Categorized between Technical Errors (Dark Blue) and Linguistic Errors (Light Blue): 

7_Metrics-2.png

 



• PDF Download - English

Was this article helpful?
10 out of 12 found this helpful