Important Considerations for Multi-Person Tracking and Analysis

Summary: In this blog post, we discuss how to optimize your markerless motion capture system for multi-person data collections. We will also quickly review the preferences menu within Theia3D and answer frequently asked questions related to multi-person tracking situations.

One of the unique advantages of Theia3D is its ability to perform automatic, multi-person tracking by identifying each unique individual within a recording. If a person is sufficiently visible in three or more camera views, they will be tracked. This can be useful in several different scenarios, including for data collections where person-to-person interactions are being studied, for in-practice or in-competition applications where multiple athletes will be simultaneously or sequentially visible, or for expediting data collections by recording multiple people performing a movement at the same time. This functionality enables a wide range of unique applications that may not have been previously possible, and an opportunity to reduce the data collection time for existing projects.

While multi-person tracking in Theia3D requires no specific subject preparation, there are some important considerations that should be made prior to data collection to ensure high quality tracking of all people. Each multi-person collection is different, however we hope to use this blog post to answer some basic questions to ensure your data collection and processing runs smoothly!

How many people can I track? 

One of the most frequently asked questions we receive related to multi-person capture is: How many people can be tracked? The short answer is that, in theory, there is no limit! However, when it comes to real-world data collection, there are some practical limitations that can impact the number of people you can track at once.

The main factor affecting the number of trackable people is person visibility. Generally speaking, each person needs to be clearly and fully visible in at least three camera views in order to be tracked by Theia3D. This means that all parts of their body, including hands and feet, are in direct sight of at least three cameras, and that the person is recorded with sufficient resolution to capture rich visible details (we recommend 500 pixels or more in height).

From a practical perspective, this means that the number of trackable people is actually determined by the number of cameras, size of the capture volume, number of people within the volume, and how those people move around or interact with each other. As you can see in the following video, we have had success in larger volumes collecting up to ten subjects at a time! But be wary of the details here - adequate system preparation and piloting is crucial to ensure success.

Do I need to calibrate differently? 

Another question you may have related to multi-person data collections is whether or not there are specific calibration requirements, to which the answer is no. There are no differences in how a camera system needs to be calibrated for multi-person vs single-person data collections. However, it is always important that the camera calibration is strong in order to obtain high quality markerless tracking, so stick with your usual calibration method and make sure that you pay close attention to this step.

When it comes to subject calibration, Theia3D does not require any static calibration trials and the subject models are built based on the tracked landmarks and the movement of the person throughout the trial. This is also the case for multi-person trials - each person will be fit with a unique model that is properly scaled to their individual body and there are no subject calibration steps or requirements.

Can subjects interact closely? 

Subject interaction in multi-person trials is a challenging topic, as it is highly dependent on the camera system setup, capture volume size, number of subjects, and the nature of the interaction. Of particular importance is the proximity between the subjects, which has a significant impact on occlusion of each person. When two people interact very closely, their mutual occlusion prevents many important landmarks from being visible in many camera views, and it can be challenging to associate the visible limbs with the correct person; together, these present a challenge when tracking.

This being an ambiguous challenge, we don’t have any specific guidelines, but the interactions listed below provide some insight into movements that will generally be possible, and those that we would expect to cause difficulty with tracking:

Generally Possible	Challenging
1-on-1 sports interactions (e.g. hockey, basketball, etc.) Synchronized group movements (e.g. yoga class, spin class, seated rehab exercises, etc.) Striking-based combat sports (e.g. boxing, kickboxing, etc.) Manual labor tasks (e.g. two-person carries, etc.) Assisted tasks (e.g. walking, sit-to-stand, etc.)	Partner dancing (e.g. ballroom, tango, etc.) Hugging Grappling-based combat sports (e.g. wrestling, Brazilian jiu-jitsu, etc.) Continuous close contact of multiple players during team sport scenarios (e.g. rugby scrums, football line of scrimmage battles, etc.) Closely crowded environments (e.g. concerts, protests, etc.)

System preparation and optimization 

Before jumping into pilot testing and certainly before real data collections, spend a bit of time thinking over the feasibility of your desired multi-person data collections. Consider the number of cameras available to you, the desired capture environment, and the possible multi-person interactions to be captured. Think about the limitations to multi-person tracking - primarily subject visibility - and if you’ll be able to achieve your multi-person goals with these considerations in mind.

If your multi-person data collection seems feasible based on the previously discussed points, you can start to think about the practical implications for system setup, preparation, and optimization. In most cases, it’s best to have a standardized approach to recording the movement, including where each individual will be positioned or their direction of travel through the volume. Based on these decisions, position your cameras so that they are out of the way of the subjects’ movement, but so that they can capture each person with sufficient resolution and visibility from enough camera views. And of course, try to avoid occlusions as much as possible. 

With the camera setup determined, another critical aspect of multi-person data collection to consider is your approach to data management, including file naming, file storage locations, and ensuring consistency in the processing of each involved individual. When saving .c3d files, Theia3D can save all tracked people to a single multi-person .c3d file, or each individual will be saved to a separate .c3d file, so think about your preferred approach and how it will impact your post-processing analysis. If you opt for separate .c3d files per person, remember to use care in checking that individuals or specific roles of interest (e.g. defense or offense in 1-on-1 sport interactions) are identified consistently, for example as person 0 or person 1 in Theia-output .c3d files.

And, as always, remember to follow our general guidelines on avoiding failed data collections.

Theia3D preferences and settings 

When it comes to processing multi-person trials in Theia3D, there are a few important settings and tools to be aware of that can impact who is identified and tracked, and how:

Max People	Allows you to specify the maximum number of people to track, or when set to No Max will track as many people as possible.
Remove stale IDs	When enabled, this will remove any person IDs that are missing for 100 frames or more. For example, one individual who enters and leaves the volume repeatedly, separated by 100 or more frames, will be identified as a new person each time they enter the volume.
Person Tracking Mode Most Visible Closest to Origin Order of Appearance	The person tracking mode allows you to select the method used to order individuals’ identification. Most Visible will number individuals based on the number of frames in which they are visible across all cameras and throughout the entire trial, followed by proximity to the origin if their visibility is similar. Closest to Origin will number individuals based on their proximity to the origin throughout the entire trial. Order of Appearance will number individuals based on the order in which they appear during the trial.
Analysis Bounding Box	When enabled and defined, either using specific values or using the camera locations, this will create a 3D bounding box outside of which individuals will not be tracked.

Together, these settings can be used very flexibly to control who will be tracked, the order in which they will be identified, and the volume within which they will be tracked. For an interesting strategy in controlling subject identification, check out our previous blog post on the bounding box, stale IDs, and person tracking mode options.

If, after processing your trial, you find that the people in your trial are not identified as you desired, the Modify People IDs tool can be used to swap person identification between tracked individuals, or remove individuals’ tracking entirely.

In summary, there is a lot of opportunity afforded by the automatic multi-person tracking capabilities of Theia3D - check out our video from last week featuring some interesting multi-person clips! Reach out to us if you’re interested in hearing more or to find out if Theia3D is right for your multi-person movement application.