Summary: Thinking of running your own validation study using Theia3D? Let us share our experience and advice for getting the most out of your efforts.
Validation is an important topic to us. It gives confidence to our users and their stakeholders, provides feedback on the strengths and weaknesses of our markerless tracking solution, and keeps us striving for improvement. It’s one of the ways we can assess the work we do and determine where to focus our efforts in order to make Theia3D even better. And interestingly, it’s usually performed independently by third parties following their own protocol and analysis methods, offering us a glimpse into the data users acquire with our software. To date, ~20 validation studies have been performed using Theia3D. If you would like a copy of these research articles just reach out to us here.
Ultimately, we love to see these studies - it’s useful information for us, and provides unbiased results for the research community to interpret. But, despite their simple appearance, validation studies can be challenging to undertake. Having had the opportunity to perform some of the earliest validation work on Theia3D while at the Queen’s University Human Mobility Research Lab under the supervision of Dr. Kevin Deluzio, I hope to provide some insight for anyone interested in performing their own validation studies using Theia3D.
Recognize the inescapable challenges of concurrent comparisons.
The very first thing you should do, even before designing your study, is to set aside any notions that you will be comparing markerless to a gold standard technology. This perspective can lead to one-sided conclusions where differences between the systems are attributed to markerless measurement errors, whereas in reality, these differences are much more complex and nuanced. Even though marker-based systems offer incredible precision in measuring the position of markers in 3D space, there are several well-documented sources of error that reduce this precision once the markers are mounted to the skin of moving humans and used to model skeletal movements. A similar point can be made about markerless motion capture: it’s an incredible technology, but we’ll be the first to admit that we don’t know its sensitivities in every possible context. So, the first step is to acknowledge the imperfection that is inherent to any and all concurrent validation studies of motion capture technology, and prepare to present your findings accordingly.
Some comparisons are not possible.
There are some questions that we’re all curious about, that can never be answered in the way that we would like, due to the nature of the measurement methods. For example, there is no perfect way to examine the effect of clothing on markerless motion capture measurements (although that hasn’t stopped us from trying! [1, 2]) because the clothing conditions themselves will affect the participants’ movements, and it’s impossible to compare different clothing on the same participant simultaneously. This being the case, it’s important to recognize the limitations of any given approach used to try to answer these types of questions.
Figure 1: Same participant, same session, different clothing. The measured biomechanical signals were similar; but, does their slight difference indicate that markerless is sensitive to different attire or that the participant’s biomechanics changed due to the attire or is it a result of natural inter-trial variability?
Choose whether you’re comparing consistency or agreement.
When you begin designing a concurrent motion capture study, it is important to consider what kind of comparison you are aiming to make. Will you treat both systems fully independently, as if you only had access to one, and compare signals which will include differences in segment definitions and movement tracking? Or will you account for any pre-existing and expected differences, such as segment coordinate system differences, in an effort to compare only the movement tracking? There is no correct answer, but it is important to recognize the difference between these comparisons and the implications of their results. In some ways, this is analogous to selecting an intraclass correlation coefficient: are you trying to assess absolute agreement, or consistency?
Figure 2: Hip flexion/extension angles during running from the same dataset, using (left) a standard marker-based pelvis definition known to be anteriorly tilted, and (right) a neutral marker-based pelvis definition.The left comparison incorporates inherent differences in the segment definitions into the overall representation of differences between systems, while the right comparison accounts for static segment definition differences and is a more direct comparison of the movement tracking between systems.
Prepare to repeat the analysis again and again (and again).
One of the benefits of markerless motion capture is that its raw data (videos + calibration) can be re-analyzed with new and improved versions of the tracking algorithm, preventing your data from becoming obsolete with new releases. However, this also means that you will have the opportunity (and usually the curiosity that comes with it) to re-do your validation results with each new version of Theia3D. So, the best thing to do with this knowledge is to design your data structures, processing pipelines, scripts and analyses in such a way that they can be quickly and easily repeated. Thinking ahead when you undertake your first validation analysis can easily save hours or days when it comes time to re-process the validation data and produce an updated version of the results. This is also especially important if you intend to publish your results, as you can update the results more easily if a new version is released during the review process. This is something we continually do at Theia and we hope that other researchers will do the same.
Figure 3: Hip joint angles from concurrent marker-based (black) and two versions of Theia3D markerless motion capture (blue, teal) during walking gait.
Collect a dataset that can answer multiple questions.
The biggest barriers to performing validation studies are the time and effort that are associated with all marker-based studies. So, if you’re going to take on this task, you might as well collect the data that would be required to answer any and all questions you might want to ask. Collect multiple movement types; collect treadmill and overground (if possible); even build in repeatability measurements by performing multiple concurrent sessions or using multiple marker placement operators. These studies are a significant undertaking, so it’s best to collect thoroughly and carefully the first time to reduce the likelihood of wanting to perform additional data collections later on.
What I’d do differently.
1. Include a static comparison.
Include a comparison of measurements from static trials to assess coordinate system differences between marker-based and markerless modalities. This could help clarify the source of differences during dynamic trials by providing a ‘baseline’ for the coordinate system offsets. Use a short trial containing some movement of all limbs that ends in a static pose with bent limbs (slightly bent knees, elbows bent with palms facing down) to ensure good markerless tracking and model scaling from this trial. When building your marker-based model, use frames from the static region of the trial only. Although this doesn’t fully account for differences in coordinate system definitions, it's a good starting point.
Figure 4: Posture for evaluating static pose and offsets in coordinate system definition between marker-based and markerless data, with slightly bent knees and bent elbows.
2. Optimize your camera views.
Follow our recommendations for camera setups to collect high quality markerless data, and don’t place your video cameras next to your marker-based cameras for convenience (this is often too high and may not be a useful view). Maximize the view of the participant and minimize the proportion of each camera view that captures unneeded space (floor, walls, ceilings, etc.). Capture multiple excellent frontal and sagittal plane views, plus oblique and other views from all sides. Be sure that there aren’t occlusions within your volume (such as a treadmill bar) that limits visibility of certain limbs.
3. Don’t skimp on frame rate.
There’s no need to go overboard, but make sure to collect your movement trials with a frame rate that is commonly used to study your movements of interest and is guaranteed to capture the movements without blur. Depending on your camera hardware, this may mean reducing the resolution to unlock higher frame rates and then bringing your cameras closer to the participant to maintain high resolution. On the same note, make sure the shutter speed (a.k.a. exposure) for the video cameras is sufficiently fast that you do not see blur in the videos.
4. Include simultaneous reliability.
If I were to perform a new comparison of marker-based and markerless motion capture, the protocol would look something like this: 3 separate days, each consisting of 2 separate sessions, with marker placement performed fresh for each session, possibly by different experimenters. This would allow you to not only compare simultaneous measurements from both systems, but to do so across multiple sessions and days, including the effect of the individual placing the markers. This also makes it easy to compare the reliability of both systems within and across sessions and days. It would, of course, be a lot of work.
Figure 5: A protocol for assessing agreement between marker-based and markerless motion capture and the reliability of each system, using two sessions on three separate days.
Conclusion
We encourage all users who are thinking about performing validation studies using Theia3D markerless motion capture to do so, taking care to consider the details and execution of the study before starting. This undertaking can provide you with experience using new markerless technology, knowledge as to how markerless motion capture measures the movements of interest compared to the marker-based systems you may be used to, and can allow new findings to be shared with the research community for the benefit of all.
Human Mobility Research Lab validation studies:
Please note that these studies were performed independently under the supervision of Dr. Kevin Deluzio in the Human Mobility Research Lab, and were not funded by Theia Markerless. I had no role or affiliation with Theia Markerless at the time.
To learn more about Theia, click here to book a demo.
Thanks for this post, it is really an amazing (and very useful) read! Regards from BeMoveD, Berlin Matej