The aim of my capstone is to provide real time dynamic lighting of backgrounds for virtual production.

The technique I am using is called "Reflectance Transformation Imaging" or "RTI" for short. This technique was created by Cultural Heritage Imaging and founded by Mark Mudge. It allows for relighting of a 2D image from any direction, hence the name 2.5D.

The process for RTI is as follows:
A reflectance sphere is placed in the scene to be used as a background. With the camera in a fixed position, multiple photographs are taken of the scene with a main light source at different locations and directions, in a half dome shape around the scene. On average around 50 - 60 photographs are taken. Once the images are collected, these are fed into software that interpolates lighting positions based on the reflectance spheres in the scene to create a surface normal map.This model is exported in the form of a .rti or .relight format, and is viewed through a renderer such as OPENLIME on the web.

During the past semester, I have focused on researching proper capture methods for RTI. Currently, I am researching and developing ways to put RTI into Unreal Engine as a plugin. I am also researching light tracking. For example, if a tracker can be placed onto a practical light (such as an actor holding a flashlight with a tracker) in front of the LED wall, and if this can be integrated into the plugin for light tracking in 2.5D.

The videos below showcase my timeline and research process working with RTI's, with the first video being my initial RTI test and the last two RTI's I have made of practical sets.