Virtual Production Fundamentals
Designed as a starter guide to help with integrating CG and plate photography in a convincing manner, with emphasis on real-time pixel output.
There are three key areas where, as each are built upon and refined, increases the level of detail and improves the final output.
NOTE: This does not discuss methods of production with NDisplay technologies, made popular by shows such as “The Mandalorian”, and exclusively deals with traditional green/blue screen keying. With that said, there are transferable ideas between the two.
Camera
No matter the camera being used, there are two sets of data points that can be generated (per camera). This data can be used to recreate the exact perspective to be captured in the real or digital world. It is the first step in creating a convincing composite.
Intrinsic Data:
These data points describe how the camera captures a frame of information
Film back
Focal Length
Focus Distance
Aperture
ISO
White Balance
Colour space
Frame rate
Shutter speed/angle
Extrinsic Data:
These data points are in reference to the centre of the camera sensor/film back, and describe its location and rotation in space
Position
Rotation
By noting Camera ID (Intrinsic Data) and ED (Extrinsic Data) it is possible to directly translate and replicate physical to virtual, and the opposite. If encoded dynamically, these details can be used to move within both worlds simultaneously, whilst maintaining visual consistency.
Mattes
The limiting factor in real-time work flows is frame time, and therefore, single method solutions for generating mattes is used. In reality, a combination of mattes will typically result in a better final output.
Garbage
Used to remove from the plate, areas of the frame/scene that should never appear in the final output.
Keying
Achieved through various strategies, can consist of “core+edge” to retain data within opaque objects, as well as fine detail edges (such as hair).
Luma: Luminance-based key, based on apparent brightness
Colour/Chroma: Chrominance-based key, based on the apparent hue+saturation
Difference: Based on a reference, subtracts values that match
Despill
A necessary component to remove the colour cast on objects in the vicinity of “screens” being used for chroma-key.
Lighting
Something I discovered whilst doing my post-graduate capstone; a simple coloured key-light, or even just a hint as the fill, helps cement the plate into the CG if that light also exists in the CG.
Scene/Talent Lighting First
Get the people lit nicely, then worry about even screen lighting. We want the people to look correctly situated; the lighting on them should be motivated.
Shadows
Inevitable and important for grounding everything real. Avoid shadows on real surfaces that don’t exist in the CG world if possible.
Reflections
Can be achieved by sending the keyed plate into your rendering engine and either hiding in main pass or frame-syncing with the overlaid keyed plate.