Summary: As you may know, Theia3D utilizes synchronized color video data in order to track human movement. Naturally, it works best with high-quality video data! As a new user of markerless motion capture software, you may be unfamiliar with video data and its various characteristics, but in this blog post we’ll give you a quick overview of the aspects of video data that are important to consider when recording data to analyze in Theia3D.
Image Resolution
Image resolution describes the number of pixels that are captured in each frame of video data; the more pixels, the clearer the image. This is usually expressed as the pixels in width by the pixels in height, for example, 1920x1080 (also known as “1080p”). Some other standard image resolutions include 3840x2160 (“4K”), 1280x720 (“720p”), and 960x540 (“540p”). Each of these resolutions has an aspect ratio of 16:9 (width:height). Other video image resolutions with different aspect ratios are also common, such as 1200x1000.
This image compares some of the more common display resolutions as well as their aspect ratios.
Image resolution is important for markerless motion capture as it determines the level of detail with which the subject is captured, with higher resolution offering greater detail. Capturing your subject with as much resolution as you can is always recommended, and can be achieved by positioning your cameras close to the subject and using the highest resolution your camera supports. For more detailed recommendations you can refer to the Theia3D documentation on camera system requirements or check out this blog post that outlines some practices to avoid. Below we’ve compiled supported video resolutions and corresponding frame rates for the various camera systems we support:
Camera System | Resolutions | Frame Rates/Hz |
Sony RX0 II | 4K | 24, 30 Hz |
1080p | 24, 30, 60, 120 Hz | |
Qualisys Miqus Video | 1080p | up to 120 Hz (new Miqus Video Plus cameras, several other modes available) |
720p | up to 180 Hz | |
540p | up to 330 Hz | |
480p | up to 550 Hz | |
OptiTrack Prime Color | 1080p | up to 250 Hz |
720p | up to 380 Hz | |
540p | up to 500 Hz | |
Teledyne FLIR Blackfly S (integrates with Vicon systems) | 1200p | up to 150 Hz |
Vicon Vue | 1080p | up to 60 Hz |
720p | up to 120 Hz |
Frame Rate
Frame rate describes the frequency or rate at which images are captured during the video recording, and is typically expressed in either frames per second (FPS) or Hertz (Hz).
The leftmost clip was recorded at 60FPS and appears much smoother than the same clip down-sampled to lower frame rates.
When it comes to motion capture of fast and complex movements, the higher the frame rate, the better. The slower the recording frame rate, the more information is lost, and vice versa. It’s especially important to perform pilot testing to ensure your movement can be captured sufficiently using the frame rates available to you; if you notice changes in distances in the position of your subject between frames, you probably need a higher frame rate! For a detailed discussion on the effects of frame rate, check out this blog post specifically on the topic.
Exposure and Shutter Speed
Exposure and shutter speed are terms that are used to describe the length of time for which the camera sensor is absorbing light, which is controlled by the time the camera’s shutter is spent open - hence, shutter speed. These two terms are closely related, but are expressed slightly differently; exposure refers to the length of time that the sensor is exposed and is typically expressed in microseconds (us), while shutter speed refers to the speed of the shutter’s close-open-close movement and is typically expressed in fractions of a second (e.g. 1/120, 1/125). These two types of values can be converted quite easily, for example a shutter speed of 1/125 = 0.008 seconds = 8000 microseconds of exposure.
Exposure and shutter speed have two main implications: image brightness and motion blur. Longer exposure time leads to more light absorption by the image sensor, while shorter exposures lead to less light absorption. However, since the shutter is open for longer during longer exposures, the subject will have moved more during that time than they would during a shorter exposure. So, longer exposure means brighter images, but more motion blur which reduces the image clarity. Since Theia3D uses machine learning techniques to estimate key point positions all over the body, it’s crucial that all segments of the body are crisp and clear within the videos’ constituent images, so properly selecting exposure and shutter speed settings is paramount. Here are some guidelines to get you started: maximum exposure = 1/(2x frame rate). For example, the longest exposure that should be used with a 120 fps frame rate is 1/(2x120) = 1/240 seconds = ~4160 microseconds. In general, shorter exposure times are always a better bet, until the image brightness starts to be affected. To see if your exposure is too long, review the videos you’ve recorded and look closely at the fastest-moving body parts - if they’re blurry, try reducing the exposure.
Aperture
The camera aperture adjusts size of the opening into the camera lens, and plays a critical role in capturing clear, high quality images. Adjusting the aperture controls the amount of light that is allowed to enter the camera lens and the depth of field of the resultant image. Opening the aperture allows more light into the camera which increases the brightness, and simultaneously shortens the depth of field in the image which can blur the background and foreground. In contrast, closing the aperture allows less light to enter, which decreases the brightness while extending the depth of field of the image.
The above figure shows how the aperture opens and closes to control how much light enters the camera lens.
When adjusting the aperture on your cameras, it is important to try to balance the level of incoming light with an appropriate depth of field. In general, the best results are often obtained with the aperture mostly open. Some cameras, such as the Qualisys Miqus Video and OptiTrack Prime Color have manually-adjusted aperture dials on their lenses, while others do not have an adjustable aperture, such as the Sony RX0 II cameras.
Focus
The camera’s focus determines how crisp or blurry the objects in an image appear. For the purpose of recording video data, you want to ensure the focus is set to the distance of where the subject will be located during the recording. When setting the focus, it is best to have the cameras in their desired recording position and to use an image or object with visible text or other details positioned within the capture volume. This object can then be viewed in the camera video streams to verify the point at which the focus adjustment provides the clearest images of the object. Some cameras do not have the option to adjust the focus manually, so it’s important to know how your cameras operate.
Gamma
Gamma correction is a characteristic of video data that can impact the contrast, color accuracy, and brightness of the images captured. As all of these factors that are affected by the gamma setting help determine the overall image quality, it is important to adjust the gamma setting so as to prevent any of these characteristics from becoming out of balance. Gamma is not available for all camera types, and should be adjusted as a secondary setting after the frame rate, exposure, and aperture settings.
Gain or ISO
Similar to exposure and shutter speed, gain and ISO are similar settings that can be used to adjust the overall brightness of the video images. However, instead of affecting the amount of light that hits the image sensor these settings impact the image brightness by amplifying the signals from the sensor. Too high of a gain or ISO correction can lead to degradation in the image quality, so it’s important to set it after an appropriate exposure is determined, and use these settings as a secondary measure. Of the Theia3D-supported cameras, only the Sony RX0 II has ISO while the remainder have gain. Again, it’s important to know your cameras and run pilot tests to make the necessary adjustments.
Conclusion
We hope this first post on the aspects of video data gives you a good starting point for the many characteristics to consider when performing your markerless motion capture data collections. In a follow-up post we’ll continue this discussion and illuminate some important aspects of lighting.
To learn more about Theia, click here to book a demo.