A Review of the Evolution of Vision-Based Motion Analysis and the Integration of Advanced Computer Vision Methods Towards Developing a Markerless System

A Review of the Evolution of Vision-Based Motion Analysis and the Integration of Advanced... Background: The study of human movement within sports biomechanics and rehabilitation settings has made considerable progress over recent decades. However, developing a motion analysis system that collects accurate kinematic data in a timely, unobtrusive and externally valid manner remains an open challenge. Main body: This narrative review considers the evolution of methods for extracting kinematic information from images, observing how technology has progressed from laborious manual approaches to optoelectronic marker-based systems. The motion analysis systems which are currently most widely used in sports biomechanics and rehabilitation do not allow kinematic data to be collected automatically without the attachment of markers, controlled conditions and/or extensive processing times. These limitations can obstruct the routine use of motion capture in normal training or rehabilitation environments, and there is a clear desire for the development of automatic markerless systems. Such technology is emerging, often driven by the needs of the entertainment industry, and utilising many of the latest trends in computer vision and machine learning. However, the accuracy and practicality of these systems has yet to be fully scrutinised, meaning such markerless systems are not currently in widespread use within biomechanics. Conclusions: This review aims to introduce the key state-of-the-art in markerless motion capture research from computer vision that is likely to have a future impact in biomechanics, while considering the challenges with accuracy and robustness that are yet to be addressed. Keywords: Automatic analysis, Body model, Cameras, Discriminative approaches, Gait, Generative algorithms, Motion capture, Rehabilitation, Sports biomechanics, Technique Key Points walking gait. However, accuracy requirements vary across different scenarios and the validity of 1. Biomechanists aspire to have motion analysis markerless systems has yet to be fully established tools that allow movement to be accurately across different movements in varying environments measured automatically and unobtrusively in 3. Further collaborative work between computer applied (e.g. everyday training) situations vision experts and biomechanists is required to 2. Innovative markerless techniques developed primarily develop such techniques further to meet the unique for entertainment purposes provide a potentially practical and accuracy requirements of motion promising solution, with some systems capable of analysis for sports and rehabilitation applications. measuring sagittal plane angles to within 2°–3° during * Correspondence: A.Salo@bath.ac.uk Review CAMERA—Centre for the Analysis of Motion, Entertainment Research and Background Applications, University of Bath, Bath BA2 7AY, UK 2 Vision-based motion analysis involves extracting informa- Department for Health, University of Bath, Bath BA2 7AY, UK Full list of author information is available at the end of the article tion from sequential images in order to describe © The Author(s). 2018 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. Colyer et al. Sports Medicine - Open (2018) 4:24 Page 2 of 15 movement. It can be traced back to the late nineteenth possible to compute joint angles, and with the incorpor- century and the pioneering work of Eadweard Muybridge ation of body segment inertia parameters, the whole who first developed techniques to capture image se- body centre of mass location can be deduced, as in pre- quences of equine gait [1]. Motion analysis has since vious research in sprinting [11, 12], gymnastics [13] and evolved substantially in parallel with major technological rugby place-kicking [14]. Moreover, kinematic and kinetic advancements and the increasing demand for faster, more data can be combined to allow the calculation of joint mo- sophisticated techniques to capture movement in a wide ments and powers through inverse dynamics analysis [15]. range of settings ranging from clinical gait assessment [2] Such analyses have value across diverse areas from, for ex- to video game animation [3]. Within sports biomechanics ample, understanding lower-limb joint power production and rehabilitation applications, quantitative analysis of hu- across fatiguing cycling efforts [16] to characterising joint man body kinematics is a powerful tool that has been used torque profiles following anterior cruciate ligament recon- to understand the performance determining aspects of struction surgery [17]. However, obtaining accurate body technique [4], identify injury risk factors [5], and facilitate pose is an essential step before reliable joint moments (and recovery from injury [6]or trauma [7]. powers) can be robustly acquired, as inaccuracies in kine- Biomechanical tools have developed considerably from matic data will propagate to larger errors in joint kinetic pa- manual annotation of images to marker-based optical rameters [18]. trackers, inertial sensor-based systems and markerless In certain cases within biomechanics, two-dimensional systems using sophisticated human body models, com- (2D) analyses using relatively simple body models suffice. puter vision and machine learning algorithms. The aim Examples include when assessing movements which are of this review is to cover some of the history of the considered to occur primarily in the sagittal plane such as development and use of motion analysis methods within walking [19] and sprinting [4], or when experimental con- sports and biomechanics, highlighting the limitations of trol is limited, such as when ski jumpers’ body positions existing systems. The state-of-the-art technologies from were analysed during Olympic competition [20]. Con- computer vision and machine learning, which have versely, when the movement under analysis occurs in mul- started to emerge within the biomechanics community, tiple planes, a multi-camera system and a more complex are introduced. This review considers how these new 3D model are required; for instance, when investigating technologies could revolutionise the fields of sports shoulder injury risks associated with different volleyball biomechanics and rehabilitation by broadening the spiking techniques [21]. The more extensive experimental applications of motion analysis to include everyday set-ups for whole-body 3D analysis typically necessitate training or competition environments. controlled laboratory environments and a challenge is to then ensure ecological validity (that movements accurately General Principles and Requirements of Vision-Based represent reality). Motion Analysis in Sports Biomechanics and The main differences between 2D and 3D analysis relate Rehabilitation to the complexity of the calibration and coordinate recon- Optical motion analysis requires the estimation of the struction processes, and joint angle definitions. Planar (2D) position and orientation (pose) of an object across analysis can be conducted with only one camera, whereas image sequences. Through the identification of at least two different perspectives are required to triangu- common object features in successive images, dis- late 2D information into 3D real-space coordinates [22, 23]. placement data can be “tracked” over time. However, The number of DOF that need to be recovered (and conse- accurate quantification of whole-body pose can be a quently the number of markers required) in order to define difficult problem to solve since the human body is an segment (or any rigid body) pose differs between 2D and extremely complex, highly articulated, self-occluding 3D methods. In 2D analysis, it is only possible to recover and only partially rigid entity [8–10]. To make this three DOF and this requires a minimum of two known process more tractable, the structure of the human points on the segment. Conversely, for 3D reconstruction body is usually simplified as a series of rigid bodies of a rigid body, six DOF can be specified by identifying at connected by frictionless rotational joints. least three non-collinear points. Three-dimensional (3D) pose of rigid segments can be A wide range of motion analysis systems allow movement fully specified by six degrees of freedom (DOF): three re- to be captured in a variety of settings, which can broadly be lating to translation and three defining orientation. As categorised into direct (devices affixed to the body, e.g. such, even for a relatively simple 14-segment human accelerometry) and indirect (vision-based, e.g. video or op- body model, a large number of DOF (potentially as toelectronic) techniques. Direct methods allow kinematic many as 84 depending on the anatomical constraints information to be captured in diverse environments. For employed) need to be recovered to completely character- example, inertial sensors have been used as tools to provide ise 3D body configuration. From such a model, it is insight into the execution of various movements (walking Colyer et al. Sports Medicine - Open (2018) 4:24 Page 3 of 15 gait [24], discus [25], dressage [26] and swimming [27]). reliability analyses, for example within sprint hurdle [39] Sensor drift, which influences the accuracy of inertial sen- and cricket [33] research. sor data, can be reduced during processing; however, this is One of the primary advantages of manual digitising, yet to be fully resolved and capture periods remain limited which has allowed this method to persist as a means to col- [28]. Additionally, it has been recognised that motion ana- lect kinematic data, is that the attachment of markers is not lysis systems for biomechanical applications should fulfil necessarily required. As such, manual digitisation remains a the following criteria: they should be capable of collecting valuable tool particularly in sports biomechanics as it allows accurate kinematic information, ideally in a timely manner, analysis of movement in normal training [12, 40, 41]and without encumbering the performer or influencing their also competition [20, 30–33] environments without imped- natural movement [29]. As such, indirect techniques can be ing the athlete(s). Additionally, this methodology provides a distinguished as more appropriate in many settings com- practical and affordable way of studying gait in applied ther- pared with direct methods, as data are captured remotely apy settings [42]. from the participant imparting minimal interference to Unfortunately, a trade-off exists between accuracy and their movement. Indirect methods were also the only pos- ecological validity when adopting field-based compared to sible approach for biomechanical analyses previously con- laboratory-based optical motion analyses [43]. Specifically, ducted during sports competition [20, 30–33]. Over the the manual digitising approach can be implemented unob- past few decades, the indirect, vision-based methods avail- trusively in applied settings with relative ease. However, the able to biomechanists have dramatically progressed towards resultant 3D vector-based joint angles are difficult to relate more accurate, automated systems. However, there is yet to to anatomically relevant axes of rotation. Moreover, if angles be a tool developed which entirely satisfies the aforemen- are projected onto 2D planes (in an attempt to separate the tioned important attributes of motion analysis systems. angle into component parts) movement in one plane can be incorrectly measured as movement in another, as discussed Historical Progression of Vision-Based Motion Analysis in in relation to the assessment of elbow extension legality dur- Sports Biomechanics and Rehabilitation ing cricket bowling [44]. Improvements to this early model- Manual Digitisation ling approach are made by digitising external markers, such Manual digitisation was the most widespread motion as medial and lateral condyles, which provide more accurate measurement technique for many decades, and prior to representations of 3D joint angles. However, certain draw- digital technologies, cine film cameras were traditionally backs remain including the fact that manual digitising is a used [32, 34–36]. These were well-suited to the field of notoriously time-consuming and laborious task, and is liable movement analysis due to their high image quality and to subjective error. These limitations have provided motiv- high-speed frame rates (100 Hz in the aforementioned ation for the development of automatic solutions made avail- studies). However, the practicality of this method was able by the emergence of more sophisticated technologies. limited by long processing times. With the advent of video cameras (initially tape-based before the transition Automatic Marker-Based Systems to digital), cine cameras have become essentially redun- A large number of commercial automatic optoelectronic dant in the field of biomechanics. Digital video cameras systems now exist for the study of human movement. The are now relatively inexpensive, have increasingly high majority of these utilise multiple cameras that emit invisible resolutions and fast frame rates (consumer cameras are infrared light, and passive markers that reflect this infrared generally capable of high-definition video at greater than back to the cameras and allow their 3D position to be de- 120 Hz, whereas industrial cameras are significantly fas- duced. Although the specifications of these systems differ ter), and are associated with shorter processing times. markedly [45], thesameunderlyingprinciplesapply inthe Regardless of the technology used to capture motion, sense that several points of interest are located in sequential manual digitising requires the manual localisation of images, converted to real-space coordinates and used to several points of interest (typically representing the infer 3D pose of the underlying skeleton. However, the pri- underlying joint centres) in each sequential image from mary difference between methodologies is that optoelec- each camera perspective. Providing a calibration trial has tronic systems are capable of automatically locating large been performed (where several control points of known numbers of markers, substantially improving the time effi- relative location are digitised in each camera view), the ciency of this process. At least three non-collinear markers position of the image body points can be reconstructed must be affixed to each segment to specify six DOF. If only into real-space coordinates, most commonly via direct two joint markers are used to define a segment, the same linear transformation [23]. Several software packages challenges exist as those outlined above in relation to man- exist to aid this process and allow accurate localisation ual digitisation. Increasing the number of markers attached of points on rigid structures [37, 38]. Moreover, the re- to each segment increases the system’sredundancy. How- peatability of these methods has been supported by ever, extensive marker sets encumber the natural Colyer et al. Sports Medicine - Open (2018) 4:24 Page 4 of 15 movement pattern and tracking marker trajectories can be- indoor conditions (where light conditions could be strin- come challenging if markers are clustered close to one an- gently controlled). However, innovative active filtering fea- other or become occluded [45]. tures can alleviate these errors and have even allowed data The accuracies of several widely utilised commercial to be captured during outdoor snow sports [63]. marker-based systems have been evaluated using a rigid, ro- It is, therefore, clear that optoelectronic systems tating structure with markers attached at known locations have made significant advancements in recent times [45]. Root mean square errors were found to be typically within the field of biomechanics. However, even less than 2.0 mm for fully visible moving markers and though careful methodological considerations can im- 1.0 mm for a stationary marker (errors were scaled to a prove the accuracy of the data acquired, several limi- standard 3-m long volume) indicating excellent precision tations remain, including long participant preparation when markers are attached to a rigid body. However, the times, potential for erroneous marker placement or exact placement of markers on anatomical landmarks is dif- movement, and the unfeasibility of attaching markers ficult to realise and markers placed on the skin do not dir- in certain settings (e.g. sports competition). Perhaps ectly correspond to 3D joint positions. Various protocols one of the most fundamental problems is the physical exist to locate joint centres and/or define segment pose and/or psychological constraints that attached from markers placed on anatomical landmarks; however, markers impart on the participant, influencing move- these different conventions produce varying out-of-sagittal ment execution. These drawbacks can limit the utility plane results when compared over the same gait cycles of marker-based systems within certain areas of sports [46]. In fact, there is also inevitable day-to-day and biomechanics and rehabilitation, and have driven the inter-tester variability in marker placement, which reduces exploration of potential markerless solutions. the reliability of marker-based measurements, particularly for transverse plane movements [47, 48]. It is well acknowledged that the rigid body assumption Markerless Motion Analysis Systems (underlying marker-based motion analysis) can be violated An attractive future advancement in motion analysis by soft tissue movement, particularly during dynamic activ- is towards a fully automatic, non-invasive, markerless ities [49]. This phenomenon has been consistently demon- approach, which would ultimately provide a major strated by studies which have compared marker-derived breakthrough for research and practice within sports kinematics with those using “gold standard” methods such biomechanics and rehabilitation. For example, motion as fluoroscopy [50], Roentgen photogrammetric techniques could be analysed during normal training environments [51, 52] and intra-cortical bone pins [53–55]. As soft tissue more readily, without the long subject preparation times as- movement introduces both systematic and random errors sociated with marker-based systems or the laborious pro- of similar frequency to the actual bone movement, this is cessing required for manual methods. Moreover, it could difficult to attenuate through data smoothing [56]. Over the provide a potential solution for a common dilemma faced last two decades, careful design of marker sets [57]and the by biomechanists, which stems from the trade-off between use of anatomical calibration procedures [58]havesome- accuracy (laboratory-based analyses) and external validity what alleviated this measurement artefact. For example, ini- (field-based analyses). tial static calibration trials can be captured whereby joint Markerless methods are not yet in widespread use centres and segment coordinate systems are defined relative within biomechanics, with only a small number of to markers. It is then possible to remove certain markers companies providing commercial systems (details for that are liable to movement during dynamic movement, a selection of which are provided in Table 1). How- without compromising the deduction of segment pose ever, it remains unclear exactly what precision these [59–61]. Moreover, the placement of marker clusters (typ- systems can achieve in comparison to the other, more ically three or four non-collinear markers rigidly affixed to established motion analysis systems available on the a plate) not only provides a practical method to define the market. Certainly, the technology is under rapid de- segment’s six DOF, but can also be strategically positioned velopment with modern computer vision algorithms to reduce soft tissue artefact [49]. The development of improving the robustness, flexibility and accuracy of more sophisticated pose-estimation algorithms [62]and markerless systems. A number of reviews [10, 64–67] joint angle definitions [43] has further advanced the accur- have been previously published detailing these devel- acy of marker-based analyses. opments, targeting specific application areas such as Optoelectronic systems are also relatively sensitive to security, forensics and entertainment. The aim of this the capture environment. In particular, sunlight, which in- section is to introduce the biomechanics community cludes a strong infrared component, can introduce un- to the current state-of-the-art markerless technology desirable noise into the measurements. In the past, from the field of computer vision, and to discuss marker-based analysis has therefore been restricted to where the technology stands in terms of accuracy. Colyer et al. Sports Medicine - Open (2018) 4:24 Page 5 of 15 Table 1 A selection of commercially available full-body markerless systems Company Cameras Capture environments Integration with other Real-time biomechanical tools capacity Captury Studio Ultimate Unlimited number with No specific background necessary. None. Applications primarily Yes The Captury combination of resolutions Can handle dynamic scenes and illumination within entertainment www.thecaptury.com changes, as long as sufficient contrast. BioStage 8–18 (120 fps in real-time) Laboratory-based Force plates and Yes Organic Motion electromyography www.organicmotion.com Shape 3D Up to 8 high-speed colour Can operate outdoors but stable background Force plates, No Simi cameras with good contrast is required electromyography and www.simi.com pressure sensors Information obtained from company web-pages (accessed July 2017) Recent Computer Vision Approaches to Markerless Motion parameters). In general, a markerless motion capture sys- Capture tem will have the form shown in Fig. 1. This consists of an In marker-based motion capture, cameras and lighting are offline stage where prior data inform model design or train- specially configured to make observation and tracking of ing of a machine learning-based discriminative algorithm, markers simple. Where multiple markers are used, indi- and then image data are captured, processed and input into vidual markers need to be identified, and measurements the algorithms that will estimate body pose and shape. then taken either directly from the positions of the markers, or from inferring the configuration of a skeleton Camera Systems for Markerless Motion Capture model that best fits the marker positions. Markerless sys- Two major families of camera systems are used for mar- tems have some similarity to this, with differences mostly kerless motion capture differing by whether or not a induced by the significantly more difficult process of gath- “depth map” is produced. A depth map is an image where ering information from the images. each pixel, instead of describing colour or brightness, The four major components of a markerless motion cap- describes the distance of a point in space from the camera ture system are (1) the camera systems that are used, (2) (Fig. 2). Depth-sensing camera systems range from the representation of the human body (the body model), (3) narrow-baseline binocular-stereo camera systems (such as the image features used and (4) the algorithms used to de- the PointGrey Bumblebee or the Stereolabs Zed camera) termine the parameters (shape, pose, location) of the body to “active” cameras which sense depth through the projec- model. The algorithms used to infer body pose given image tion of light into the observed scene such as Microsoft’s data are usually categorised as either “generative” (in which Kinect. Depth information can help alleviate problems model parameters can be used to “generate” ahypothesis that affect traditional camera systems such as shadows, that is evaluated against image data and then iteratively imperfect lighting conditions, reflections and cluttered refined to determine a best possible fit) or “discriminative” backgrounds. Active, depth-sensing camera systems (often (where image data is used to directly infer model termed RGB-D cameras where they capture both colour Fig. 1 General structure of a markerless motion capture whether using generative (green) or discriminative (orange) algorithms Colyer et al. Sports Medicine - Open (2018) 4:24 Page 6 of 15 accuracy required for precision biomechanics (it could be speculated, however, that a tracking system not dedicated to interactive systems may achieve greater accuracy using these devices). Active cameras have been applied to sports biomechanics [72, 73] using bespoke software, but current hardware limitations (effective only over short range, max- imum 30 Hz framerates, inoperability in bright sun light, and interference between multiple sensors) are likely to limit their application in sports biomechanics for the fore- seeable future. Body Models The body models used by markerless motion capture are generally similar to those used by traditional marker-based approaches. A skeleton is defined as a set of joints and the bones between these joints (Fig. 3). The skeleton is parame- Fig. 2 Example of a depth map. Brighter pixels are further away terised on the lengths of the bones and the rotation of each from the camera. Black pixels are either too far away or on objects joint with pose being described by the joint angles. For dis- that do not reflect near infrared light criminative approaches, this skeleton model can be enough, but generative approaches will also require a representation and depth) have proven effective for real-time full-body of the person’svolume. pose estimation in interactive systems and games [68, 69]. In earlier works, the volume of the model is represented The devices most commonly use one of two technologies: by simple geometric shapes [74]such ascylinders. Such structured light or time-of-flight (ToF). Structured light models remain the state-of-the-art within computer vision devices sense depth through the deformations of a known in the form of a set of “spatial 3D Gaussians” [75]attached pattern projected onto the scene, while ToF devices meas- to the bones of a kinematic skeleton (Fig. 4). The advan- ure the time for a pulse of light to return to the camera. tage of this representation has been to enable fast, almost The two technologies have different noise characteristics real-time fitting in a generative framework using passive and trade-offs between depth accuracy and spatial reso- cameras and a very simple set of image features. lution [70]. The most well-known active cameras are In general though, the trend has been to use 3D triangle Microsoft’s original structured light “Kinect”,and there- meshes common in graphics and computer games, either placement ToF based “Kinect For Xbox One” (often re- created by artists, as the product of high-definition 3D ferred to as the Kinect 2), which are provided with body scans [76], or more recently, by specialising a generic stat- tracking software designed for interactive systems. The per- istical 3D shape model [77–79](Fig. 5). Statistical body formance of this tracking system has been analysed for both shape models allow a wide range of human body shapes versions of the cameras [71], but clearly falls well below the to be represented in a relatively small number of Fig. 3 Example of a poseable skeleton model. “Bones” of a pre-specified length are connected at joints, and rotation of the bones around these joints allows the skeleton to be posed. The skeleton model is commonly fit to both marker-based motion capture data and computer vision-based markerless systems Colyer et al. Sports Medicine - Open (2018) 4:24 Page 7 of 15 Fig. 4 Sum of Gaussian body model from Stoll [75]. A skeleton (left) forms the foundation of the model, providing limb-lengths and body pose. The body is given volume and appearance information through the use of 3D Spatial Gaussians arranged along the skeleton (represented by spheres). The resulting information allows the model to be fit to image data parameters and improve how the body surface deforms region, or pixel. The process of determining how pixels re- under joint rotations. However, because these models focus late to objects is a fundamental task in computer vision, on the external surface appearance of the model, the under- and there have been many proposed approaches to extract- lying skeleton is questionably representative of an actual ing “features” from an image that are meaningful. It is the human skeleton and as such, care must be taken should great difficulty of this task that marker-based systems have these models be used for biomechanical measurements. been developed to avoid. The parameterisation of the human body used by motion For motion capture, the primary aim is to determine the capture models is always a simplification, and although cap- location and extents in the image of the person being cap- able of a realistic appearance, can also represent physically tured. The earliest and one of the most robust approaches unrealistic shapes and poses. If the algorithms are not care- to this task is termed chroma keying. This is where the fully constrained, these solutions can appear optimal for the background of the scene is painted a single specific colour, available data. To ensure only physically realistic solutions allowing the silhouette of the person (who is dressed in a are produced, the algorithms must be supported by con- suitable contrasting colour) to be easily segmented (Fig. 6). straints on the body model such as explicit joint limits [80] For environments where chroma keying is not possible, or probabilistic spaces of human pose and motion deduced there are a large number [82] of background subtraction al- by machine learning [81]. In either case, there exists a gorithms. However, these can all suffer from problems with balance between enforcing the constraints and trust- shadows, lighting changes, reflections and non-salient mo- ing the observed data to achieve a solution that is tions of the background (such as a crowd or other athletes). both plausible and precise. Image silhouettes are also inherently ambiguous and pro- vide no information on whether the observed subject is fa- Image Features for Markerless Motion Capture cing towards or away from the camera (Fig. 6). This A digital image, fundamentally, is a 2D grid of numbers ambiguity can only be reduced by the use of extra cameras each representing the brightness and colour of a small or more sophisticated image features. Where large numbers Fig. 5 Skinned Multi-Person Linear Model (SMPL) [79] body model. This model does not have an explicit skeleton. Instead, the surface of a person is represented by a mesh of triangles. A set of parameters (learnt through regression) allows the shape of the model to be changed from a neutral mean (left) to a fatter (middle) or thinner, taller, or other body shape. Once shaped, the centres of joints are inferred from the neutrally posed mesh, and then the mesh can be rotated around these joints to produce a posed body (right) Colyer et al. Sports Medicine - Open (2018) 4:24 Page 8 of 15 Fig. 6 Silhouette on the right from chroma keying the image on the left. When seen as only a silhouette, it is not possible to infer if the mannequin is facing towards or away from the camera of cameras are available, silhouettes can be combined into a given set of model parameters (body shape, bone lengths, 3D representation known as a visual hull [83], which is an joint angles), a representation of the model is generated. approximation of the space occupied by the observed per- This representation can then be compared against the fea- son (Fig. 7). More sophisticated 3D reconstructions can also tures extracted from the image and a single “error value” be carried out [84]; however, any added accuracy must be calculated, which represents how much the hypothesis dif- traded off against increased computational complexity. Im- fers from the observed data. In one possibility, the 3D tri- proving the reconstruction does not fully resolve all fitting angle mesh resulting from the predicted parameters can difficulties, however, and extra information that identifies be projected into the 2D image, and the overlap of the which regions of the silhouettes correspond to which re- mesh and the silhouette of the person can be maximised gions of the body are often needed to completely resolve all [92]. Alternatively, the 3D body model can be compared possible confusion [85]. Nevertheless, silhouettes have against a 3D reconstruction such as a visual hull by mini- formed a key aspect of many markerless motion capture mising the distances between the 3D vertices of the model, works including the work of Corazza et al. [86], which has and the 3D points of the visual hull [86, 93]through a reported some of the most accurate results for automatic standard algorithm known as iterative closest point. markerless body motion capture, and Liu et al. [87], which A key factor of generative approaches is the appropriate enables the kinematic motion analysis of multiple persons. definition of the function that compares a specific hypoth- The trend, however, has been to move away from the use esis with the information available in the images. If this is of image silhouettes to improve robustness, reduce ambigu- not carefully considered, then the search for the optimal ities, reduce the number of cameras and simplify capture set of model parameters can easily fail, resulting in poor procedures. In this regard, the work of Stoll et al. [75]is sig- estimates or nonsense configurations where joints bend at nificant for enabling the body model to fit to the image unrealistic angles and limbs penetrate inside the body. using only a simple colour model, while the advent of deep Constructing a cost function that is robust to image noise learning [88] and its provision of robust and fast body-part and to unrealistic model configurations is difficult mean- detectors has made dramatic improvements on what can ing generative models often need a reliable initial guess of be done outside of laboratory conditions [89, 90], including the model parameters. In extremes this would mean for- recognisingbodyposeofmanypeoplefroma singleuncali- cing the person being captured to assume a specific pose brated and moving camera [91]. at the start of tracking. If the fitting then becomes con- fused by occlusions, image noise or other failure, tracking Generative Algorithms will not be able to correct itself without manual interven- In generative motion capture approaches, the pose and tion. Researchers have attempted to address this situation shape of the person is determined by fitting the body using improved searching algorithms [92], extra informa- model to information extracted from the image. For a tion derived from robust body part detectors [90]and Colyer et al. Sports Medicine - Open (2018) 4:24 Page 9 of 15 Fig. 7 The generation of a visual hull, which is a type of 3D reconstruction of an object viewed from multiple cameras. Top row: images of an object are captured as 2D images from multiple directions. Middle row: these images are processed to produce silhouette images for each camera. Bottom left: the silhouettes are back-projected from each camera, resulting in cone-like regions of space. Bottom right: the intersection of these cones results in the visual hull recent pose-recognition algorithms [94–97], or by coup- possible to “teach” the computer how to determine the ling generative methods with discriminative methods [98]. pose of a simple skeleton model using only the image data. The most recent approaches in this family use deep learn- Discriminative Approaches ing to train a system that can identify the body parts of Discriminative algorithms avoid the process of iteratively multiple people, the probable ownership of joints, and then tuning the parameters of a body model to fit the image and quickly parse this to determine skeletons [91]. Alternatively, as such they are also often referred to as model-free algo- a database of pose examples can be created and then rithms. Compared with generative approaches, they will searched to discover the most similar known pose given the often have a much faster processing time, improved robust- current image, as used in previous studies [101–103]. ness and reduced dependence on an initial guess. However, The main difficulty in the use of discriminative algo- they can have reduced precision, and they require a very rithms is the creation of the exemplar data. If the avail- largedatabase ofexemplardata(farmorethanisrequired able data is insufficient, then poses, physiques and even even for constructing the statistical body shape models used camera positions that were not suitably represented will by generative algorithms) from which they can learn how lead to false results as the system will not be able to gen- to infer a result. eralise from what it “knows” to what it “sees”. This will Discriminative approaches have two major families. One also affect the precision of the result because the algo- approach is to discover a mapping directly from image fea- rithm is restricted to giving solutions close to what it tures to a description of pose, such as by using machine “knows” about, so small variations may not be fully rep- learning-based regression [99, 100]. In this way, it is resented in the results. As a result, discriminative Colyer et al. Sports Medicine - Open (2018) 4:24 Page 10 of 15 approaches are used as initial guesses for generative ap- pose-estimation algorithms such as global optimisation (in- proaches [98]. verse kinematics) with joint constraints [62]. As the “gold standard” methods are inappropriate in many contexts, and Summary of Markerless Approaches marker-based systems are the most frequently utilised mo- The current state-of-the-art shows the computer vi- tion analysis technique in the field, agreement between sion community aiming to develop solutions to mar- markerless and optoelectronicsystems wouldbe considered kerless motion capture that are applicable and to provideevidencefor the validityof markerlessmotion reliable outside of laboratory conditions. Although analysis techniques. carefully calibrated silhouette-based algorithms using There are already studies which have attempted to evalu- sophisticated subject-specific body models have ate the accuracy of markerless systems (summarised in shown the most accurate results to date, they have Table 2)bycomparing thekinematic output variables been limited to laboratory conditions using a large against those obtained using marker-based optoelectronic number of cameras [86]. By taking advantage of modern systems [10, 86, 93, 107–109] or manual digitisation [110]. technologies such as improved solvers [92], advanced image These validations mostly study relatively slow movements features and modern machine learning [100], recent works (typically walking gait), whereas to verify the utility of these areproviding solutionsthatreducethe required number of approaches in sports applications, much quicker move- cameras [104], allow moving cameras [105], increase the ments need to be thoroughly assessed. One clear observa- number of people that can be tracked and provide robust tion from these results is that transverse plane rotations are detection and fitting in varied environments [91]. The abil- currently difficult to extract accurately and reliably by mar- itytodothiswithout knowledgeofcameracalibration fur- kerlesstechnologies[107, 110]. ther improves the potential ease of use of future systems; In the computer vision community, it is common prac- however, calibration is likely to remain a necessity where tice to advancetechnologybyestablishingbenchmark data- precise measurement is needed such as in biomechanics. sets against which many authors can rank their algorithm’s performance. Two such benchmarks are the widely used Accuracy of Current Markerless Motion Capture Systems HumanEva dataset [111]and themore recent Human There are distinct differences in the accuracy requirements 3.6M dataset [112]. These datasets provide video of people betweenmotionanalysistechniques inthe fields of com- performing actions (walking, jogging, boxing etc.) while also puter vision and biomechanics, which must be taken into being tracked with marker-based tracking systems. Table 3 account when attempting to apply computer vision shows a sample of published comparison results for the methods more broadly across other disciplines. For in- HumanEva dataset. These results show that precision of stance, accuracy within computer vision (primarily enter- markerless techniques remains too low to be applicable for tainment applications) is typically assessed qualitatively and biomechanics analyses. However, the video data and mo- is primarily evaluated based on appearance. Conversely, in tioncapturedatainthe HumanEva datasetare themselves biomechanical settings, it is fundamental that any motion of limited quality. For example, videos are low resolution analysis system is capable of robustly quantifying subtle dif- and camera placement is sub-optimal, while markers are ferences in motion, which could be meaningful from a limited and sub-optimally located (often on relatively loose musculoskeletal performance or pathology perspective. clothing) with no marker clusters to aid tracking (Fig. 8). Nonetheless, there is no general consensus regarding the For comparison, the results of Corazza et al. [86]onHuma- minimum accuracy requirement of motion analysis systems nEva have a mean joint centre position error of 79 ± and the magnitudes of the inevitable measurement errors 12 mm, while on the authors’ own higher resolution data will vary depending on the context (laboratory vs. field), the with better camera and marker placement, a much smaller characteristics of the movement and the participant, the ex- 15 ± 10 mm error was achieved. perimental setup and how the human body is modelled. The discrepancies observed by Corazza et al. [86]be- As previously described, marker-based approaches are tween the validation results against the HumanEva bench- currently the most widely used systems in biomechanical mark and the more rigorously captured marker-based data laboratories. However, a prominent source of measurement show the difficulties of treating marker-based motion cap- error in marker-based systems is skin movement artefact ture as the criterion method. In fact, although these bench- [56], which violates the rigid body assumption underlying marks are useful for showing the general performance of these methods. Reports suggestthaterrors due to soft tis- different algorithms, neither the marker-based nor manual sue movement can exceed 10 mm for some anatomical digitising methods used to validate markerless technologies landmarks and 10° for some joint angles when compared to can provide exact “true” body pose due to experimental ar- more precise, yet invasive, methods (e.g. intra-cortical bone tefacts that are inevitably introduced. Additionally, ensuring pins) [106]. However, these errors in joint angle measure- a close match between the body model applied to both sys- ments may be reduced to 2°–4° by using more sophisticated tems is a challenge, which may necessitate “off-line” stages Colyer et al. Sports Medicine - Open (2018) 4:24 Page 11 of 15 Table 2 Overview of studies comparing markerless with conventional motion analysis systems Publication Movement(s) Markerless system description Procedure/ Number of Outcome analysed system for cameras comparison Trewartha et Starjump, Gen-locked video cameras (50 Hz), Manual 3 RMS differences for three movements ranged al. [110] somersaults subject-specific model digitising from 10 mm and 30 mm for pelvis location (TARGET and between 2° and 8° for body configuration system) angles. Corazza et al. Walking Visual hull construction and a priori Virtual 16 RMS errors of hip, knee and ankle angles [93] subject-specific model environment ranged from 2.0° (hip abduction/adduction) to (Poser 9.0 (ankle dorsi/plantar flexion) software) Mündermann Walking Video cameras (75 Hz), visual hull Qualisys 8 Average knee joint angle deviation: 2.3° et al. [10] construction and a priori subject- (120 Hz) (sagittal plane) and 1.6° (frontal plane). specific model Corazza et al. Walking Video cameras (120 Hz), visual hull Qualisys 8 Average deviations between joint (hip, knee, [86] construction and a priori subject- (120 Hz) ankle, shoulder, elbow and wrist) centres: specific model 15 mm mean absolute error (ranged from 9 to 19 mm) Choppin and Reaching, Microsoft Kinect (30 Hz) Motion 1 Kinect, 12 Flexion/extension and abduction/adduction of Wheat [72] throwing, Analysis optoelectronic hip, knee, elbow and shoulder; shoulder plane jumping Corporation and elevation studied. Maximum abduction (60 Hz) error: 44.1° and 13.9°. Maximum flexion error: 36.2° and 19.5° (NITE and IPIsoft tracking algorithms, respectively) Ceseracciu et Walking BTS BTS 8 Maximum RMS differences range: 11.0° (ankle al. [107] SMART-D SMART-D dorsi/ plantar flexion) to 34.7° (hip internal/ (100 Hz) (200 Hz) external rotation) Sandau et al. Walking Monochrome cameras (75 Hz), Ariel 8 RMS differences in lower limb 3D angles [108] unconstrained articulated model fit Performance ranged between 1.8° (hip abduction/ to 3D point clouds (aided by full Analysis adduction) and 4.9° (hip internal/external body patterned suit) System rotation) Ong et al. Walking and Point Grey cameras (25 Hz) Motion 2 markerless, RMS differences ranged from 0.2° (knee [109] jogging Analysis 8 marker- abduction/adduction of jogging) to 1.0° (ankle Corporation based dorsi/plantar flexion of walking). Significant (100 Hz) differences between markerless and marker- based for the ankle joint angles. RMS root mean square perhaps involving imaging, as in previous work [76]. Add- performance of markerless systems have also been consid- ing markers to the validation images might also unduly bias ered, such as utilising force plate data to analyse centre of the performance of a markerless system under test, if the mass movement [113] and creating virtual environments algorithm detects the markers and uses them for its benefit (synthesised images) in which a predefined model moves or if the markers adversely affect the performance of the with known kinematics [93]. Although synthesised images markerless algorithm (altering silhouette shapes, for ex- can be invaluable for developing an algorithm (synthetic ample) [107]. As such, alternative methods of validating the images were used for generating training images for Table 3 Selection of published validation results against the HumanEva datasets Publication 3D joint position error (mm) Standard deviation of error (mm) Corazza et al. [86] 79.0 11.5 Amin et al. [94] 54.5 Belagiannis et al. [95] 68.3 Saini et al. [92] 45.7 5.3 Guo et al. [103] 46.8 Elhayek et al. [90] 66.5 Rhodin et al. [98] 54.6 24.2 Bogo et al. [89] 79.9 Colyer et al. Sports Medicine - Open (2018) 4:24 Page 12 of 15 discriminative approach (to get good initialisation and ro- bustness) with a robust silhouette-free kinematic model fitting approach for precision. A fast, approximate pose estimation system has previ- ously been combined with a slow, more-accurate tech- nique in order to provide basic parameters to athletics coaches and inform training in real-time [76]. This type of system may have utility in the applied field by allowing some of the primary, “top-level” biomechanical determi- nants (for example, step frequency and step length in gait) to be fed-back during normal training or rehabilitation sit- uations. Importantly, the more complex and computation- ally expensive kinematic variables (such as 3D joint angles, which require modelling of the body) may still be acquired. However, the likelihood is that more time-consuming, offline processing will be necessary. This two-part approach could help address the apparent dis- connect between sports science research and practice [114] as short participant preparation times and timely feedback from the system may increase the perceived (and actual) value of such studies to those operating in the ap- plied field. Importantly, the more complex kinematic in- Fig. 8 An example image from the HumanEva dataset used to formation could still be computed and communicated to validate markerless systems within computer vision. White dots applied practitioners across longer time frames, but indicate the location of tracked reflective markers and the cyan lines equally these data can be used in research studies to con- represent the defined skeleton model fit to the marker data. Although useful as an early benchmark for markerless tracking tinually progress our scientific understanding of human systems, the dataset has clear limitations for assessing the quality of movement. any markerless tracking results, especially in the context of It should be noted that resolution (both spatial and tem- biomechanics. Notice that the markers are attached to clothing, poral) will affect the accuracy of markerless systems in the marker clusters are not utilised, and the joint centres inferred from same way as it does for marker-based systems. However, the fitted skeleton are not closely aligned with how the person appears in the image (e.g. right elbow and hip joints). See further video-based automatic systems must also consider the fact information in the text that the size of the data captured will be considerably larger and thus, markerless systems may need to compromise ac- Microsoft’s Kinect pose tracker [68]), the idealised image curacy to make a deployable, fast system feasible. Such a data is unlikely to capture the noise and error sources of system requires large amounts of video data to be handled real imagery. efficiently and effectively, which is likely to necessitate the purchase of an expensive (perhaps specially engineered) Future of Markerless Approaches to Analyse Motion in video-based system (e.g. machine vision). Sports Biomechanics and Rehabilitation It is clear that a broad range of markerless technologies Conclusions have emerged from computer vision research over recent Vision-based motion analysis methods within sports and times, which have the potential to be applied across di- rehabilitation applications have evolved substantially over verse disciplines and settings. The priorities and require- recent times and have allowed biomechanical research to ments for a markerless motion capture system will contribute a vast amount of meaningful information to depend on the research area and the unique capture envir- thesefields. However, themost widespread kinematic data onment, and are thus non-uniform across disciplines. In capture techniques (marker-based technologies and manual sports biomechanics and rehabilitation applications, mo- digitisation) are not without their drawbacks. Considerable tion analysis systems must be highly accurate in order to developments in computer vision have sparked interest in detect subtle changes in motion, as well as being adapt- markerless motion analysis and its possible wider applica- able, non-invasive and unencumbering. With these system tions. Although this potential is promising, it is not yet clear requirements in mind, the current progression of tech- exactly what accuracy can be achieved and whether such nologies suggests that the future of practical markerless systems can be effectively and routinely utilised in motion capture will lie with techniques such as those pre- field-based (more externally valid) settings. Over the com- sented by Elhayek et al. [90], which fuse together a ing years, collaborative research between computer vision Colyer et al. Sports Medicine - Open (2018) 4:24 Page 13 of 15 experts and biomechanists is required to further develop 7. Sjödahl C, Jarnlo G-B, Söderberg B, Persson BM. Kinematic and kinetic gait analysis in the sagittal plane of trans-femoral amputees before and after markerless techniques so they are able to meet the unique special gait re-education. Prosthetics Orthot Int. 2002;26(2):101–12. practical and accuracy requirements of motion analysis 8. O'Rourke J, Badler NI. Model-based image analysis of human motion using within sports and rehabilitation contexts. constraint propagation. IEEE Trans Pattern Anal Mach Intell. 1980;2(6):522–36. 9. Leung MK, Yang Y-H. First sight: a human body outline labeling system. IEEE Abbreviations Trans Pattern Anal Mach Intell. 1995;17(4):359–77. 2D: Two-dimensional; 3D: Three-dimensional; DOF: Degrees of freedom; 10. Mündermann L, Corazza S, Andriacchi TP. The evolution of methods for the capture RMS: Root mean square; SMPL: Skinned Multi-Person Linear Model of human movement leading to markerless motion capture for biomechanical applications. J Neuroeng Rehabil. 2006; https://doi.org/10.1186/1743-0003-3-6 Funding 11. Bezodis IN, Kerwin DG, Salo AIT. Lower-limb mechanics during the support phase This review was funded by CAMERA, the RCUK Centre for the Analysis of of maximum-velocity sprint running. Med Sci Sports Exerc. 2008;40(4):707–15. Motion, Entertainment Research and Applications, EP/M023281/1. 12. Churchill SM, Salo AIT, Trewartha G. The effect of the bend on technique and performance during maximal effort sprinting. Sports Authors’ Contributions Biomechanics. 2015;14(1):106–21. SC, ME, DC and AS all participated in planning the conception and design of 13. Hiley MJ, Yeadon MR. Achieving consistent performance in a complex whole this review article. SC and ME completed the majority of draft writing. DC body movement: the Tkatchev on high bar. Hum Mov Sci. 2012;31(4):834–43. and AS provided critical revisions for the manuscript. All authors provided 14. Bezodis NE, Trewartha G, Wilson C, Irwin G. Contributions of the non-kicking-side the final approval for the final submitted version and agreed to be arm to rugby place-kicking technique. Sports Biomechanics. 2007;6:171–86. accountable for all aspects of the work. 15. Winter DA. Biomechanics and motor control of human movement. 2nd ed. New York: Wiley; 1990. Authors’ Information 16. Martin JC, Brown NAT. Joint-specific power production and fatigue during SC has a PhD and is a Post-Doctoral Research Associate in the CAMERA project maximal cycling. J Biomech. 2009;42(4):474–9. (see funding above) with expertise in analysing athletes’ technique. ME has a 17. Devita P, Hortobagyi T, Barrier J, Torry M, Glover KL, Speroni DL, et al. Gait PhD and is a Post-Doctoral Research Associate in the CAMERA project with adaptation before and after anterior cruciate ligament reconstruction expertise in computer vision with time spent in the computer industry before surgery. Med Sci Sports Exerc. 1997;29(7):853–9. joining back to academia. DC has a PhD and is a Professor in Computer Science 18. Camomilla V, Cereatti A, Cutti AG, Fantozzi S, Stagni R, Vannozzi G. specialising in computer vision. He is the principal investigator and the director Methodological factors affecting joint moments estimation in clinical gait of the CAMERA project. AS has a PhD and is a Reader (Associate Professor) in analysis: a systematic review. Biomed Eng Online. 2017;16(106); https://doi. Sports Biomechanics with expertise in analysing athletes’ technique. He org/10.1186/s12938-017-0396-x. is a co-investigator in the CAMERA project. 19. Matsas A, Taylor N, McBurney H. Knee joint kinematics from familiarised treadmill walking can be generalised to overground walking in young Ethics Approval and Consent to Participate unimpaired subjects. Gait Posture. 2000;11(1):46–53. Not applicable 20. Schmölzer B, Müller W. Individual flight styles in ski jumping: results obtained during Olympic Games competitions. J Biomech. 2005;38(5):1055–65. Competing Interests 21. Seminati E, Marzari A, Vacondio O, Minetti AE. Shoulder 3D range of motion The authors Steffi L. Colyer, Murray Evans, Darren P. Cosker and Aki I. T. Salo and humerus rotation in two volleyball spike techniques: injury prevention declare that they have no competing interests. and performance. Sports Biomechanics. 2015;14(2):216–31. 22. Zhang Z. Camera calibration. In: Ikeuchi K, editor. Computer vision: a reference guide. Boston: Springer US; 2014. p. 76–7. Publisher’sNote 23. Abdel-Aziz YI, Karara HM, Hauck M. Direct linear transformation from Springer Nature remains neutral with regard to jurisdictional claims in comparator coordinates into object space coordinates in close-range published maps and institutional affiliations. photogrammetry. Photogramm Eng Remote Sens. 2015;81(2):103–7. 24. Mayagoitia RE, Nene AV, Veltink PH. Accelerometer and rate gyroscope Author details measurement of kinematics: an inexpensive alternative to optical motion CAMERA—Centre for the Analysis of Motion, Entertainment Research and analysis systems. J Biomech. 2002;35(4):537–42. Applications, University of Bath, Bath BA2 7AY, UK. Department for Health, University of Bath, Bath BA2 7AY, UK. Department of Computer Science, 25. Ganter N, Krüger A, Gohla M, Witte K, Edelmann-Nusser J. Applicability of a University of Bath, Bath BA2 7AY, UK. full body inertial measurement system for kinematic analysis of the discus throw. Michigan: Proceedings of the 28th conference of the International Received: 23 January 2018 Accepted: 22 May 2018 Society of Biomechanics in Sports; 2010. 26. Eckardt F, Münz A, Witte K. Application of a full body inertial measurement system in dressage riding. J Equine Vet Sci. 2014;34:1294–9. References 27. de Magalhaes FA, Vannozzi G, Gatta G, Fantozzi S. Wearable inertial sensors in 1. Muybridge E. Complete human and animal locomotion (all 781 plates from swimming motion analysis: a systematic review. J Sports Sci. 2015;33(7):732–45. the 1887 animal locomotion). In: Cappozzo A, Marchetti M, Tosi V, editors. 28. Dadashi F, Crettenand F, Millet GP, Aminian K. Front-crawl instantaneous Biolocomotion: a century of research using moving pictures. Rome: velocity estimation using a wearable inertial measurement unit. Sensors. Promograph; 1979. p. 69. 2012;12(10):12927–39. 2. Cimolin V, Galli M. Summary measures for clinical gait analysis: a literature 29. Atha J. Current techniques for measuring motion. Appl Ergon. 1984;15(4):245–57. review. Gait Posture. 2014;39(4):1005–10. 30. Salo AIT, Grimshaw PN, Marar L. 3-D biomechanical analysis of sprint hurdles at 3. Cosker D, Eisert P, Helzle V. Facial capture and animation in visual effects. In: different competitive levels. Med Sci Sports Exerc. 1997;29(2):231–7. Magnor MA, Grau O, Sorkine-Hornung O, Theobalt C, editors. Digital 31. Manning ML, Irwin G, Gittoes MJR, Kerwin DG. Influence of longswing representations of the real world: how to capture, model, and render visual technique on the kinematics and key release parameters of the straddle reality. Boca Raton, FL: CRC Press; 2015. p. 311–21. Tkachev on uneven bars. Sports Biomechanics. 2011;10(3):161–73. 4. Bezodis NE, Salo AIT, Trewartha G. Relationships between lower-limb 32. Lees A, Graham-Smith P, Fowler N. A biomechanical analysis of the last kinematics and block phase performance in a cross section of sprinters. Eur stride, touchdown, and takeoff characteristics of the men’s long jump. J J Sport Sci. 2015;15(2):118–24. Appl Biomech. 1994;10:61–78. 5. Ford KR, Myer GD, Toms HE, Hewett TE. Gender differences in the kinematics of 33. Portus MR, Rosemond CD, Rath DA. Cricket: fast bowling arm actions and unanticipated cutting in young athletes. Med Sci Sports Exerc. 2005;37(1):124–9. the illegal delivery law in me’s high performance cricket matches. Sports 6. Devita P, Hortobagyi T, Barrier J. Gait biomechanics are not normal after Biomechanics. 2006;5(2):215–30. anterior cruciate ligament reconstruction and accelerated rehabilitation. Med Sci Sports Exerc. 1998;30(10):1481–8. 34. Procter P, Paul JP. Ankle joint biomechanics. J Biomech. 1982;15(9):627–34. Colyer et al. Sports Medicine - Open (2018) 4:24 Page 14 of 15 35. Ericson M. On the biomechanics of cycling. A study of joint and muscle load 61. Snyder KR, Earl JE, O'Connor KM, Ebersole KT. Resistance training is during exercise on the bicycle ergometer. Scand J Rehabil Med. 1986;16:1–43. accompanied by increases in hip strength and changes in lower extremity 36. Bobbert MF, Huijing PA, van Ingen Schenau GJ. A model of the human triceps biomechanics during running. Clin Biomech. 2009;24(1):26–34. surae muscle-tendon complex applied to jumping. J Biomech. 1986;19(11):887–98. 62. Lu TW, O'Connor JJ. Bone position estimation for skin marker co-ordinates 37. Wilson DJ,Smith BK,Gibson JK, Choe BK,GabaBC, Voelz JT. Accuracy of using global optimisation with joint constraints. J Biomech. 1999;32(2):129–34. digitization using automated and manual methods. Phys Ther. 1999;79(6):558–66. 63. Nedergaard NJ, Heinen F, Sloth S, Hébert-Losier K, Holmberg H-C, Kersting UG. The effect of light reflections from the snow on kinematic data collected using 38. Scholz JP, Millford JP. Accuracy and precision of the PEAK performance stereo-photogrammetry with passive markers. Sports Eng. 2014;17:97–102. technologies motion measurement system. J Mot Behav. 1993;25(1):2–7. 39. Salo AIT, Grimshaw PN. An examination of kinematic variability of motion 64. Moeslund TB, Hilton A, Krüger V. A survey of advances in vision-based human analysis in sprint hurdles. J Appl Biomech. 1998;14:211–22. motion capture and analysis. Comput Vis Image Underst. 2006;104:90–126. 40. Bezodis I, Salo A, Kerwin D. A longitudinal case study of step characteristics 65. Poppe R. Vision-based human motion analysis: an overview. Comput Vis in a world class sprint athlete. In: Kwon Y-H, Shim J, Shim JK, Shin I-S, Image Underst. 2007;108:4–18. editors. . Seoul: Proceedings of 26th international conference on 66. Holte MB, Tran C, Trivedi MM, Moeslund TB. Human pose estimation and biomechanics in Sports; 2008. activity recognition from multi-view videos: comparative explorations of 41. Bezodis NE, Salo AIT, Trewartha G. Choice of sprint start performance measure recent developments. Select Topics Signal Process. 2012;6:538–52. affects the performance-based ranking within a group of sprinters: which is 67. Yang SXM, Christiansen MS, Larsen PK, Alkjær T, Moeslund TB, Simonsen EB, the most appropriate measure? Sports Biomechanics. 2010;9(4):258–69. et al. Markerless motion capture systems for tracking of persons in forensic biomechanics: an overview. Comput Methods Biomechanics Biomed Eng: 42. Churchill AJG, Halligan PW, Wade DT. RIVCAM: a simple video-based Imaging Visual. 2014;2(1):46–65. kinematic analysis for clinical disorders of gait. Comput Methods Prog Biomed. 2002;69(3):197–209. 68. Shotton J, Fitzgibbon A, Blake A, Kipman A, Finocchio M, Moore R, et al. 43. Elliott B, Alderson J. Laboratory versus field testing in cricket bowling: a Real-time human pose recognition in parts from a single depth image. review of current and past practice in modelling techniques. Sports Colorado: Proceedings of the IEEE Conference on Computer Vision and Biomechanics. 2007;6(1):99–108. Pattern Recognition; 2011. 44. Elliott BC, Alderson JA, Denver ER. System and modelling errors in motion 69. Ye M, Zhang Q, Wang L, Zhu J, Yang R, Gall JA. Survey on human motion analysis: implications for the measurement of the elbow angle in cricket analysis from depth data. In: Grzegorzek M, Theobalt C, Koch R, Kolb A, bowling. J Biomech. 2007;40(12):2679–85. editors. Time-of-flight and depth imaging sensors, algorithms, and 45. Richards JG. The measurement of human motion: a comparison of applications. Berlin, Heidelberg: Springer; 2013. p. 149–87. commercially available systems. Hum Mov Sci. 1999;18(5):589–602. 70. Sarbolandi H, Lefloch D, Kolb A. Kinect range sensing: structured-light versus time-of-flight kinect. J Comput Vision Image Understand. 2015;139:1–20. 46. Ferrari A, Benedetti MG, Pavan E, Frigo C, Bettinelli D, Rabuffetti M, et al. 71. Wang Q, Kurillo G, Ofli F, Bajcsy R. Evaluation of pose tracking accuracy in Quantitative comparison of five current protocols in gait analysis. Gait the first and second generations of Microsoft Kinect. IEEE Int Confer Healthc Posture. 2008;28:207–16. Inform (ICHI). 2015:380–9. 47. Growney E, Meglan D, Johnson M, Cahalan T, An K-N. Repeated measures of adult normal walking using a video tracking system. Gait 72. Choppin S, Wheat J. The potential of the Microsoft Kinect in sports analysis Posture. 1997;6(2):147–62. and biomechanics. Sports Technol. 2013;6(2):78–85. 48. Tsushima H, Morris ME, McGinley J. Test-retest reliability and inter-tester 73. Schmitz A, Ye M, Boggess G, Shapiro R, Yang R, Noehren B. The reliability of kinematic data from a three-dimensional gait analysis system. J measurement of in vivo joint angles during a squat using a single camera Japan Phys Ther Assoc. 2003;6(1):9–17. markerless motion capture system as compared to a marker based system. 49. Cappozzo A, Catani F, Leardini A, Benedetti MG, Della Croce U. Position and Gait Posture. 2015;41:694–8. orientation in space of bones during movement: experimental artefacts. 74. Gavrila DM, Davis LS. 3-D model-based tracking of humans in action: a Clin Biomech. 1996;11(2):90–100. multi-view approach. San Juan, Puerto Rico: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 1996. 50. Stagni R, Fantozzi S, Cappello A, Leardini A. Quantification of soft tissue artefact 75. Stoll C, Hasler N, Gall J, Seidel HP, Theobalt C. Fast articulated motion in motion analysis by combining 3D fluoroscopy and stereophotogrammetry: tracking using a sums of Gaussians body model. Barcelona: Proceedings of a study on two subjects. Clin Biomech. 2005;20(3):320–9. the international conference on computer vision; 2011. 51. Maslen BA, Ackland TR. Radiographic study of skin displacement errors in the foot and ankle during standing. Clin Biomech. 1994;9:291–6. 76. El-Sallam A, Bennamoun M, Sohel F, Alderson J, Lyttle A, Rossi R. A low cost 52. Tranberg R, Karlsson D. The relative skin movement of the foot: a 2D 3D markerless system for the reconstruction of athletic techniques. Tampa: roentgen photogrammetry study. Clin Biomech. 1998;13(1):71–6. IEEE Workshop on Applications of Computer Vision; 2013. 53. Fuller J, Liu L-J, Mann RW. A comparison of lower-extremity skeletal 77. Allen B, Curless B, Popović Z. The space of human body shapes: reconstruction kinematics measured using skin- and pin-mounted markers. Hum Mov Sci. and parameterization from range scans. ACM Trans Graph. 2003;22(3):587–94. 1997;16(2–3):219–42. 78. Anguelov D, Srinivasan P, Koller D, Thrun S, Rodgers J, Davis J. SCAPE: shape 54. Benoit DL, Ramsey DM, Lamontagne M, Xu L, Wretenberg P, Renström P. completion and animation of people. ACM Trans Graph. 2005;24(3):408–16. 79. Loper M, Mahmood N, Romero J, Pons-Moll G, Black MJ. SMPL: a skinned Effect of skin movement artifact on knee kinematics during gait and cutting multi-person linear model. Proc of SIGGRAPH Asia. 2015;34(6):1–16. motions measured in vivo. Gait Posture. 2006;24(2):152–64. 55. Reinschmidt C, van den Bogert AJ, Nigg BM, Lundberg A, Murphy N. Effect 80. Akhter IB, Black MJ. Pose-conditioned joint angle limits for 3D human pose reconstruction. Boston: Proceedings of the IEEE Conference on Computer of skin movement on the analysis of skeletal knee joint motion during Vision and Pattern Recognition; 2015. running. J Biomech. 1997;30(7):729–32. 56. Leardini A, Chiari L, Della Croce U, Cappozzo A. Human movement analysis 81. Holden D, Saito J, Komura T, Joyce T. Learning motion manifolds with using stereophotogrammetry: part 3. Soft tissue artifact assessment and convolutional autoencoders. SIGGRAPH Asia Tech Briefs. 2015;18; https://doi. compensation. Gait Posture. 2005;21(2):212–25. org/10.1145/2820903.2820918. 57. Manal K, McClay I, Stanhope S, Richards J, Galinat B. Comparison of surface 82. Bouwmans T. Traditional and recent approaches in background modeling mounted markers and attachment methods in estimating tibial rotations for foreground detection: an overview. Comput Sci Rev. 2014;11-12:31–66. during walking: an in vivo study. Gait Posture. 2000;11(1):38–45. 83. Laurentini A. The visual hull concept for silhouette-based image 58. Cappozzo A, Catani F, Della Croce U, Leardini A. Position and orientation in understanding. IEEE Trans Pattern Anal Mach Intell. 1994;16(2):150–62. space of bones during movement: anatomical frame definition and 84. Furukawa Y, Hernández C. Multi-view stereo: a tutorial. Found Trends determination. Clin Biomech. 1995;10(4):171–8. Comput Graphics Vision. 2015;9(1–2):1–148. 59. Milner CE, Ferber R, Pollard CD, Hamill J, Davis IS. Biomechanical factors 85. Kanaujia A, Kittens N, Ramanathan N. Part segmentation of visual hull for 3D associated with tibial stress fracture in female runners. Med Sci Sports Exerc. human pose estimation. Oregon: IEEE Conference on Computer Vision and 2006;38(2):323–8. Pattern Recognition Workshops; 2013. 60. Chmielewski TL, Rudolph KS, Fitzgerald GK, Axe MJ, Snyder-Mackler L. 86. Corazza S, Mündermann L, Gambaretto E, Ferrigno G, Andriacchi TP. Biomechanical evidence supporting a differential response to acute ACL Markerless motion capture through visual hull, articulated ICP and subject injury. Clin Biomech. 2001;16(7):586–91. specific model generation. Int J Comput Vis. 2010;87:156–69. Colyer et al. Sports Medicine - Open (2018) 4:24 Page 15 of 15 87. Liu Y, Gall J, Stoll C, Dai Q, Seidel HP, Theobalt C. Markerless motion capture 111. Sigal L, Balan AO, Black MJ. HumanEva: synchronized video and motion of multiple characters using multiview image segmentation. IEEE Trans capture dataset and baseline algorithm for evaluation of articulated human Pattern Anal Mach Intell. 2013;35(11):2720–35. motion. Int J Comput Vis. 2010;87:4–27. 88. LeCun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015;521(7553):436–44. 112. Ionescu C, Papava D, Olaru V, Sminchisescu C. Human3.6M: large scale datasets and predictive methods for 3D human sensing in natural 89. Bogo F, Kanazawa A, Lassner C, Gehler P, Romero J, Black MJ. Keep it SMPL: environments. IEEE Trans Pattern Anal Mach Intell. 2014;36(7):1325–39. automatic estimation of 3D human pose and shape from a single image. Amsterdam: Proceedings of the European Conference on Computer Vision; 2016. 113. Corazza S, Andriacchi TP. Posturographic analysis through markerless motion capture without ground reaction forces measurement. J Biomech. 90. Elhayek A, de Aguiar E, Jain A, Tompson J, Pishchulin L, Andriluka M, et al. 2009;42(3):370–4. Efficient ConvNet-based marker-less motion capture in general scenes with 114. Bishop D. An applied research model for the sport sciences. Sports Med. a low number of cameras. Boston: Proceedings of the IEEE Conference on 2008;38(3):253–63. Computer Vision and Pattern Recognition; 2015. 91. Cao Z, Simon T, Wei SE, Sheikh Y. Realtime multi-person 2D pose estimation using part affinity fields. Comput Vision Pattern Recogn. 2016; https://doi. org/10.1109/CVPR.2017.143. 92. Saini S, Zakaria N, Rambli DRA, Sulaiman S. Markerless human motion tracking using hierarchical multi-swarm cooperative particle swarm optimization. PLoS One. 2015;10(5):1–22. 93. Corazza S, Mündermann L, Chaudhari AM, Demattio T, Cobelli C, Andriacchi TP. A markerless motion capture system to study musculoskeletal biomechanics: visual hull and simulated annealing approach. Ann Biomed Eng. 2006;34(6):1019–29. 94. Amin S, Andriluka M, Rohrback M, Schiele B. Multi-view pictorial structures for 3D human pose estimation. British Machine Vision Confer. 2013; https:// doi.org/10.5244/C.27.45. 95. Belagiannis V, Amin S, Andriluka M, Schiele B, Navab N, Ilic S. 3D pictorial structures for multiple human pose estimation. Ohio: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2014. 96. Burenius M, Sullivan J, Carlsson S. 3D pictorial structures for multiple view articulated pose estimation. Oregon: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2013. 97. Kazemi V, Burenius M, Azizipour H, Sullivan J. Multi-view body part recognition with random forests. Br Machine Vision Confer. 2013; https:// doi.org/10.5244/C.27.48. 98. Rhodin H, Robertini N, Casas D, Richardt C, Seidel H-P, Theobalt C. General automatic human shape and motion capture using volumetric contour cues. In: Leibe B, Matas J, Sebe N, Welling M, editors. . Amsterdam: Proceedings of the European Conference on Computer Vision; 2016. 99. Agarwal A, Triggs B. Recovering 3D human pose from monocular images. Pattern Anal Machine Intelligence. 2006;28:44–58. 100. Hong C, Yu K, Xie Y, Chen X. Multi-view deep learning for image-based pose recovery. Hangzhou: Proceedings of the IEEE 16th International Conference on Communication Technology; 2015. 101. Chen C, Yang Y, Nie F, Odobez J-M. 3D human pose recovery from image by efficient visual feature selection. Comput Vis Image Underst. 2011;115(3):290–9. 102. Babagholami-Mohamadabadi B, Jourabloo A, Zarghami A, Kasaei S. A Bayesian framework for sparse representation-based 3-D human pose estimation. IEEE Signal Proc Lett. 2014;21(3):297–300. 103. Guo Y, Chen Z, Yu J. Supervised spectral embedding for human pose estimation. Suzhou: Proceedings for the 5th International Conference of Intelligence Science and Big Data Engineering Image and Video Data Engineering; 2015. 104. Mehta D, Sridhar S, Sotnychenko O, Rhodin H, Shafiei M, Seidel H-P, et al. VNect: real-time 3D human pose estimation with a single RGB camera. ACM Trans Graph. 2017;36(4): https://doi.org/10.1145/3072959.3073596. 105. Elhayek A, Stoll C, Kim KI, Theobalt C. Outdoor human motion capture by simultaneous optimization of pose and camera parameters. Computer Graphics Forum. 2015;34(6):86–98. 106. Peters A, Galna B, Sangeux M, Morris M, Baker R. Quantification of soft tissue artifact in lower limb human motion analysis: a systematic review. Gait Posture. 2010;31:1–8. 107. Ceseracciu E, Sawacha Z, Cobelli C. Comparison of markerless and marker- based motion capture technologies through simultaneous data collection during gait: proof of concept. PLoS One. 2014;9(3):e87640. https://doi.org/ 10.1371/journal.pone.0087640 108. Sandau M, Koblauch H, Moeslund TB, Aanæs H, Alkjær T, Simonsen EB. Markerless motion capture can provide reliable 3D gait kinematics in the sagittal and frontal plane. Med Eng Phys. 2014;36(9):1168–75. 109. Ong A, Harris IS, Hamill J. The efficacy of a video-based marker-less tracking system for gait analysis. Comput Methods Biomechanics Biomed Eng. 2017; 20(10):1089–95. 110. Trewartha G, Yeadon MR, Knight JP. Marker-free tracking of aerial movements. In: Müller R, Gerber H, Stacoff A, editors. . Zürich: Proceedings of the 18th congress of the international society of Biomechanics; 2001. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Sports Medicine - Open Springer Journals

A Review of the Evolution of Vision-Based Motion Analysis and the Integration of Advanced Computer Vision Methods Towards Developing a Markerless System

Free
15 pages

Loading next page...
 
/lp/springer_journal/a-review-of-the-evolution-of-vision-based-motion-analysis-and-the-Z0n7lVIUq2
Publisher
Springer Journals
Copyright
Copyright © 2018 by The Author(s).
Subject
Medicine & Public Health; Sports Medicine
ISSN
2199-1170
eISSN
2198-9761
D.O.I.
10.1186/s40798-018-0139-y
Publisher site
See Article on Publisher Site

Abstract

Background: The study of human movement within sports biomechanics and rehabilitation settings has made considerable progress over recent decades. However, developing a motion analysis system that collects accurate kinematic data in a timely, unobtrusive and externally valid manner remains an open challenge. Main body: This narrative review considers the evolution of methods for extracting kinematic information from images, observing how technology has progressed from laborious manual approaches to optoelectronic marker-based systems. The motion analysis systems which are currently most widely used in sports biomechanics and rehabilitation do not allow kinematic data to be collected automatically without the attachment of markers, controlled conditions and/or extensive processing times. These limitations can obstruct the routine use of motion capture in normal training or rehabilitation environments, and there is a clear desire for the development of automatic markerless systems. Such technology is emerging, often driven by the needs of the entertainment industry, and utilising many of the latest trends in computer vision and machine learning. However, the accuracy and practicality of these systems has yet to be fully scrutinised, meaning such markerless systems are not currently in widespread use within biomechanics. Conclusions: This review aims to introduce the key state-of-the-art in markerless motion capture research from computer vision that is likely to have a future impact in biomechanics, while considering the challenges with accuracy and robustness that are yet to be addressed. Keywords: Automatic analysis, Body model, Cameras, Discriminative approaches, Gait, Generative algorithms, Motion capture, Rehabilitation, Sports biomechanics, Technique Key Points walking gait. However, accuracy requirements vary across different scenarios and the validity of 1. Biomechanists aspire to have motion analysis markerless systems has yet to be fully established tools that allow movement to be accurately across different movements in varying environments measured automatically and unobtrusively in 3. Further collaborative work between computer applied (e.g. everyday training) situations vision experts and biomechanists is required to 2. Innovative markerless techniques developed primarily develop such techniques further to meet the unique for entertainment purposes provide a potentially practical and accuracy requirements of motion promising solution, with some systems capable of analysis for sports and rehabilitation applications. measuring sagittal plane angles to within 2°–3° during * Correspondence: A.Salo@bath.ac.uk Review CAMERA—Centre for the Analysis of Motion, Entertainment Research and Background Applications, University of Bath, Bath BA2 7AY, UK 2 Vision-based motion analysis involves extracting informa- Department for Health, University of Bath, Bath BA2 7AY, UK Full list of author information is available at the end of the article tion from sequential images in order to describe © The Author(s). 2018 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. Colyer et al. Sports Medicine - Open (2018) 4:24 Page 2 of 15 movement. It can be traced back to the late nineteenth possible to compute joint angles, and with the incorpor- century and the pioneering work of Eadweard Muybridge ation of body segment inertia parameters, the whole who first developed techniques to capture image se- body centre of mass location can be deduced, as in pre- quences of equine gait [1]. Motion analysis has since vious research in sprinting [11, 12], gymnastics [13] and evolved substantially in parallel with major technological rugby place-kicking [14]. Moreover, kinematic and kinetic advancements and the increasing demand for faster, more data can be combined to allow the calculation of joint mo- sophisticated techniques to capture movement in a wide ments and powers through inverse dynamics analysis [15]. range of settings ranging from clinical gait assessment [2] Such analyses have value across diverse areas from, for ex- to video game animation [3]. Within sports biomechanics ample, understanding lower-limb joint power production and rehabilitation applications, quantitative analysis of hu- across fatiguing cycling efforts [16] to characterising joint man body kinematics is a powerful tool that has been used torque profiles following anterior cruciate ligament recon- to understand the performance determining aspects of struction surgery [17]. However, obtaining accurate body technique [4], identify injury risk factors [5], and facilitate pose is an essential step before reliable joint moments (and recovery from injury [6]or trauma [7]. powers) can be robustly acquired, as inaccuracies in kine- Biomechanical tools have developed considerably from matic data will propagate to larger errors in joint kinetic pa- manual annotation of images to marker-based optical rameters [18]. trackers, inertial sensor-based systems and markerless In certain cases within biomechanics, two-dimensional systems using sophisticated human body models, com- (2D) analyses using relatively simple body models suffice. puter vision and machine learning algorithms. The aim Examples include when assessing movements which are of this review is to cover some of the history of the considered to occur primarily in the sagittal plane such as development and use of motion analysis methods within walking [19] and sprinting [4], or when experimental con- sports and biomechanics, highlighting the limitations of trol is limited, such as when ski jumpers’ body positions existing systems. The state-of-the-art technologies from were analysed during Olympic competition [20]. Con- computer vision and machine learning, which have versely, when the movement under analysis occurs in mul- started to emerge within the biomechanics community, tiple planes, a multi-camera system and a more complex are introduced. This review considers how these new 3D model are required; for instance, when investigating technologies could revolutionise the fields of sports shoulder injury risks associated with different volleyball biomechanics and rehabilitation by broadening the spiking techniques [21]. The more extensive experimental applications of motion analysis to include everyday set-ups for whole-body 3D analysis typically necessitate training or competition environments. controlled laboratory environments and a challenge is to then ensure ecological validity (that movements accurately General Principles and Requirements of Vision-Based represent reality). Motion Analysis in Sports Biomechanics and The main differences between 2D and 3D analysis relate Rehabilitation to the complexity of the calibration and coordinate recon- Optical motion analysis requires the estimation of the struction processes, and joint angle definitions. Planar (2D) position and orientation (pose) of an object across analysis can be conducted with only one camera, whereas image sequences. Through the identification of at least two different perspectives are required to triangu- common object features in successive images, dis- late 2D information into 3D real-space coordinates [22, 23]. placement data can be “tracked” over time. However, The number of DOF that need to be recovered (and conse- accurate quantification of whole-body pose can be a quently the number of markers required) in order to define difficult problem to solve since the human body is an segment (or any rigid body) pose differs between 2D and extremely complex, highly articulated, self-occluding 3D methods. In 2D analysis, it is only possible to recover and only partially rigid entity [8–10]. To make this three DOF and this requires a minimum of two known process more tractable, the structure of the human points on the segment. Conversely, for 3D reconstruction body is usually simplified as a series of rigid bodies of a rigid body, six DOF can be specified by identifying at connected by frictionless rotational joints. least three non-collinear points. Three-dimensional (3D) pose of rigid segments can be A wide range of motion analysis systems allow movement fully specified by six degrees of freedom (DOF): three re- to be captured in a variety of settings, which can broadly be lating to translation and three defining orientation. As categorised into direct (devices affixed to the body, e.g. such, even for a relatively simple 14-segment human accelerometry) and indirect (vision-based, e.g. video or op- body model, a large number of DOF (potentially as toelectronic) techniques. Direct methods allow kinematic many as 84 depending on the anatomical constraints information to be captured in diverse environments. For employed) need to be recovered to completely character- example, inertial sensors have been used as tools to provide ise 3D body configuration. From such a model, it is insight into the execution of various movements (walking Colyer et al. Sports Medicine - Open (2018) 4:24 Page 3 of 15 gait [24], discus [25], dressage [26] and swimming [27]). reliability analyses, for example within sprint hurdle [39] Sensor drift, which influences the accuracy of inertial sen- and cricket [33] research. sor data, can be reduced during processing; however, this is One of the primary advantages of manual digitising, yet to be fully resolved and capture periods remain limited which has allowed this method to persist as a means to col- [28]. Additionally, it has been recognised that motion ana- lect kinematic data, is that the attachment of markers is not lysis systems for biomechanical applications should fulfil necessarily required. As such, manual digitisation remains a the following criteria: they should be capable of collecting valuable tool particularly in sports biomechanics as it allows accurate kinematic information, ideally in a timely manner, analysis of movement in normal training [12, 40, 41]and without encumbering the performer or influencing their also competition [20, 30–33] environments without imped- natural movement [29]. As such, indirect techniques can be ing the athlete(s). Additionally, this methodology provides a distinguished as more appropriate in many settings com- practical and affordable way of studying gait in applied ther- pared with direct methods, as data are captured remotely apy settings [42]. from the participant imparting minimal interference to Unfortunately, a trade-off exists between accuracy and their movement. Indirect methods were also the only pos- ecological validity when adopting field-based compared to sible approach for biomechanical analyses previously con- laboratory-based optical motion analyses [43]. Specifically, ducted during sports competition [20, 30–33]. Over the the manual digitising approach can be implemented unob- past few decades, the indirect, vision-based methods avail- trusively in applied settings with relative ease. However, the able to biomechanists have dramatically progressed towards resultant 3D vector-based joint angles are difficult to relate more accurate, automated systems. However, there is yet to to anatomically relevant axes of rotation. Moreover, if angles be a tool developed which entirely satisfies the aforemen- are projected onto 2D planes (in an attempt to separate the tioned important attributes of motion analysis systems. angle into component parts) movement in one plane can be incorrectly measured as movement in another, as discussed Historical Progression of Vision-Based Motion Analysis in in relation to the assessment of elbow extension legality dur- Sports Biomechanics and Rehabilitation ing cricket bowling [44]. Improvements to this early model- Manual Digitisation ling approach are made by digitising external markers, such Manual digitisation was the most widespread motion as medial and lateral condyles, which provide more accurate measurement technique for many decades, and prior to representations of 3D joint angles. However, certain draw- digital technologies, cine film cameras were traditionally backs remain including the fact that manual digitising is a used [32, 34–36]. These were well-suited to the field of notoriously time-consuming and laborious task, and is liable movement analysis due to their high image quality and to subjective error. These limitations have provided motiv- high-speed frame rates (100 Hz in the aforementioned ation for the development of automatic solutions made avail- studies). However, the practicality of this method was able by the emergence of more sophisticated technologies. limited by long processing times. With the advent of video cameras (initially tape-based before the transition Automatic Marker-Based Systems to digital), cine cameras have become essentially redun- A large number of commercial automatic optoelectronic dant in the field of biomechanics. Digital video cameras systems now exist for the study of human movement. The are now relatively inexpensive, have increasingly high majority of these utilise multiple cameras that emit invisible resolutions and fast frame rates (consumer cameras are infrared light, and passive markers that reflect this infrared generally capable of high-definition video at greater than back to the cameras and allow their 3D position to be de- 120 Hz, whereas industrial cameras are significantly fas- duced. Although the specifications of these systems differ ter), and are associated with shorter processing times. markedly [45], thesameunderlyingprinciplesapply inthe Regardless of the technology used to capture motion, sense that several points of interest are located in sequential manual digitising requires the manual localisation of images, converted to real-space coordinates and used to several points of interest (typically representing the infer 3D pose of the underlying skeleton. However, the pri- underlying joint centres) in each sequential image from mary difference between methodologies is that optoelec- each camera perspective. Providing a calibration trial has tronic systems are capable of automatically locating large been performed (where several control points of known numbers of markers, substantially improving the time effi- relative location are digitised in each camera view), the ciency of this process. At least three non-collinear markers position of the image body points can be reconstructed must be affixed to each segment to specify six DOF. If only into real-space coordinates, most commonly via direct two joint markers are used to define a segment, the same linear transformation [23]. Several software packages challenges exist as those outlined above in relation to man- exist to aid this process and allow accurate localisation ual digitisation. Increasing the number of markers attached of points on rigid structures [37, 38]. Moreover, the re- to each segment increases the system’sredundancy. How- peatability of these methods has been supported by ever, extensive marker sets encumber the natural Colyer et al. Sports Medicine - Open (2018) 4:24 Page 4 of 15 movement pattern and tracking marker trajectories can be- indoor conditions (where light conditions could be strin- come challenging if markers are clustered close to one an- gently controlled). However, innovative active filtering fea- other or become occluded [45]. tures can alleviate these errors and have even allowed data The accuracies of several widely utilised commercial to be captured during outdoor snow sports [63]. marker-based systems have been evaluated using a rigid, ro- It is, therefore, clear that optoelectronic systems tating structure with markers attached at known locations have made significant advancements in recent times [45]. Root mean square errors were found to be typically within the field of biomechanics. However, even less than 2.0 mm for fully visible moving markers and though careful methodological considerations can im- 1.0 mm for a stationary marker (errors were scaled to a prove the accuracy of the data acquired, several limi- standard 3-m long volume) indicating excellent precision tations remain, including long participant preparation when markers are attached to a rigid body. However, the times, potential for erroneous marker placement or exact placement of markers on anatomical landmarks is dif- movement, and the unfeasibility of attaching markers ficult to realise and markers placed on the skin do not dir- in certain settings (e.g. sports competition). Perhaps ectly correspond to 3D joint positions. Various protocols one of the most fundamental problems is the physical exist to locate joint centres and/or define segment pose and/or psychological constraints that attached from markers placed on anatomical landmarks; however, markers impart on the participant, influencing move- these different conventions produce varying out-of-sagittal ment execution. These drawbacks can limit the utility plane results when compared over the same gait cycles of marker-based systems within certain areas of sports [46]. In fact, there is also inevitable day-to-day and biomechanics and rehabilitation, and have driven the inter-tester variability in marker placement, which reduces exploration of potential markerless solutions. the reliability of marker-based measurements, particularly for transverse plane movements [47, 48]. It is well acknowledged that the rigid body assumption Markerless Motion Analysis Systems (underlying marker-based motion analysis) can be violated An attractive future advancement in motion analysis by soft tissue movement, particularly during dynamic activ- is towards a fully automatic, non-invasive, markerless ities [49]. This phenomenon has been consistently demon- approach, which would ultimately provide a major strated by studies which have compared marker-derived breakthrough for research and practice within sports kinematics with those using “gold standard” methods such biomechanics and rehabilitation. For example, motion as fluoroscopy [50], Roentgen photogrammetric techniques could be analysed during normal training environments [51, 52] and intra-cortical bone pins [53–55]. As soft tissue more readily, without the long subject preparation times as- movement introduces both systematic and random errors sociated with marker-based systems or the laborious pro- of similar frequency to the actual bone movement, this is cessing required for manual methods. Moreover, it could difficult to attenuate through data smoothing [56]. Over the provide a potential solution for a common dilemma faced last two decades, careful design of marker sets [57]and the by biomechanists, which stems from the trade-off between use of anatomical calibration procedures [58]havesome- accuracy (laboratory-based analyses) and external validity what alleviated this measurement artefact. For example, ini- (field-based analyses). tial static calibration trials can be captured whereby joint Markerless methods are not yet in widespread use centres and segment coordinate systems are defined relative within biomechanics, with only a small number of to markers. It is then possible to remove certain markers companies providing commercial systems (details for that are liable to movement during dynamic movement, a selection of which are provided in Table 1). How- without compromising the deduction of segment pose ever, it remains unclear exactly what precision these [59–61]. Moreover, the placement of marker clusters (typ- systems can achieve in comparison to the other, more ically three or four non-collinear markers rigidly affixed to established motion analysis systems available on the a plate) not only provides a practical method to define the market. Certainly, the technology is under rapid de- segment’s six DOF, but can also be strategically positioned velopment with modern computer vision algorithms to reduce soft tissue artefact [49]. The development of improving the robustness, flexibility and accuracy of more sophisticated pose-estimation algorithms [62]and markerless systems. A number of reviews [10, 64–67] joint angle definitions [43] has further advanced the accur- have been previously published detailing these devel- acy of marker-based analyses. opments, targeting specific application areas such as Optoelectronic systems are also relatively sensitive to security, forensics and entertainment. The aim of this the capture environment. In particular, sunlight, which in- section is to introduce the biomechanics community cludes a strong infrared component, can introduce un- to the current state-of-the-art markerless technology desirable noise into the measurements. In the past, from the field of computer vision, and to discuss marker-based analysis has therefore been restricted to where the technology stands in terms of accuracy. Colyer et al. Sports Medicine - Open (2018) 4:24 Page 5 of 15 Table 1 A selection of commercially available full-body markerless systems Company Cameras Capture environments Integration with other Real-time biomechanical tools capacity Captury Studio Ultimate Unlimited number with No specific background necessary. None. Applications primarily Yes The Captury combination of resolutions Can handle dynamic scenes and illumination within entertainment www.thecaptury.com changes, as long as sufficient contrast. BioStage 8–18 (120 fps in real-time) Laboratory-based Force plates and Yes Organic Motion electromyography www.organicmotion.com Shape 3D Up to 8 high-speed colour Can operate outdoors but stable background Force plates, No Simi cameras with good contrast is required electromyography and www.simi.com pressure sensors Information obtained from company web-pages (accessed July 2017) Recent Computer Vision Approaches to Markerless Motion parameters). In general, a markerless motion capture sys- Capture tem will have the form shown in Fig. 1. This consists of an In marker-based motion capture, cameras and lighting are offline stage where prior data inform model design or train- specially configured to make observation and tracking of ing of a machine learning-based discriminative algorithm, markers simple. Where multiple markers are used, indi- and then image data are captured, processed and input into vidual markers need to be identified, and measurements the algorithms that will estimate body pose and shape. then taken either directly from the positions of the markers, or from inferring the configuration of a skeleton Camera Systems for Markerless Motion Capture model that best fits the marker positions. Markerless sys- Two major families of camera systems are used for mar- tems have some similarity to this, with differences mostly kerless motion capture differing by whether or not a induced by the significantly more difficult process of gath- “depth map” is produced. A depth map is an image where ering information from the images. each pixel, instead of describing colour or brightness, The four major components of a markerless motion cap- describes the distance of a point in space from the camera ture system are (1) the camera systems that are used, (2) (Fig. 2). Depth-sensing camera systems range from the representation of the human body (the body model), (3) narrow-baseline binocular-stereo camera systems (such as the image features used and (4) the algorithms used to de- the PointGrey Bumblebee or the Stereolabs Zed camera) termine the parameters (shape, pose, location) of the body to “active” cameras which sense depth through the projec- model. The algorithms used to infer body pose given image tion of light into the observed scene such as Microsoft’s data are usually categorised as either “generative” (in which Kinect. Depth information can help alleviate problems model parameters can be used to “generate” ahypothesis that affect traditional camera systems such as shadows, that is evaluated against image data and then iteratively imperfect lighting conditions, reflections and cluttered refined to determine a best possible fit) or “discriminative” backgrounds. Active, depth-sensing camera systems (often (where image data is used to directly infer model termed RGB-D cameras where they capture both colour Fig. 1 General structure of a markerless motion capture whether using generative (green) or discriminative (orange) algorithms Colyer et al. Sports Medicine - Open (2018) 4:24 Page 6 of 15 accuracy required for precision biomechanics (it could be speculated, however, that a tracking system not dedicated to interactive systems may achieve greater accuracy using these devices). Active cameras have been applied to sports biomechanics [72, 73] using bespoke software, but current hardware limitations (effective only over short range, max- imum 30 Hz framerates, inoperability in bright sun light, and interference between multiple sensors) are likely to limit their application in sports biomechanics for the fore- seeable future. Body Models The body models used by markerless motion capture are generally similar to those used by traditional marker-based approaches. A skeleton is defined as a set of joints and the bones between these joints (Fig. 3). The skeleton is parame- Fig. 2 Example of a depth map. Brighter pixels are further away terised on the lengths of the bones and the rotation of each from the camera. Black pixels are either too far away or on objects joint with pose being described by the joint angles. For dis- that do not reflect near infrared light criminative approaches, this skeleton model can be enough, but generative approaches will also require a representation and depth) have proven effective for real-time full-body of the person’svolume. pose estimation in interactive systems and games [68, 69]. In earlier works, the volume of the model is represented The devices most commonly use one of two technologies: by simple geometric shapes [74]such ascylinders. Such structured light or time-of-flight (ToF). Structured light models remain the state-of-the-art within computer vision devices sense depth through the deformations of a known in the form of a set of “spatial 3D Gaussians” [75]attached pattern projected onto the scene, while ToF devices meas- to the bones of a kinematic skeleton (Fig. 4). The advan- ure the time for a pulse of light to return to the camera. tage of this representation has been to enable fast, almost The two technologies have different noise characteristics real-time fitting in a generative framework using passive and trade-offs between depth accuracy and spatial reso- cameras and a very simple set of image features. lution [70]. The most well-known active cameras are In general though, the trend has been to use 3D triangle Microsoft’s original structured light “Kinect”,and there- meshes common in graphics and computer games, either placement ToF based “Kinect For Xbox One” (often re- created by artists, as the product of high-definition 3D ferred to as the Kinect 2), which are provided with body scans [76], or more recently, by specialising a generic stat- tracking software designed for interactive systems. The per- istical 3D shape model [77–79](Fig. 5). Statistical body formance of this tracking system has been analysed for both shape models allow a wide range of human body shapes versions of the cameras [71], but clearly falls well below the to be represented in a relatively small number of Fig. 3 Example of a poseable skeleton model. “Bones” of a pre-specified length are connected at joints, and rotation of the bones around these joints allows the skeleton to be posed. The skeleton model is commonly fit to both marker-based motion capture data and computer vision-based markerless systems Colyer et al. Sports Medicine - Open (2018) 4:24 Page 7 of 15 Fig. 4 Sum of Gaussian body model from Stoll [75]. A skeleton (left) forms the foundation of the model, providing limb-lengths and body pose. The body is given volume and appearance information through the use of 3D Spatial Gaussians arranged along the skeleton (represented by spheres). The resulting information allows the model to be fit to image data parameters and improve how the body surface deforms region, or pixel. The process of determining how pixels re- under joint rotations. However, because these models focus late to objects is a fundamental task in computer vision, on the external surface appearance of the model, the under- and there have been many proposed approaches to extract- lying skeleton is questionably representative of an actual ing “features” from an image that are meaningful. It is the human skeleton and as such, care must be taken should great difficulty of this task that marker-based systems have these models be used for biomechanical measurements. been developed to avoid. The parameterisation of the human body used by motion For motion capture, the primary aim is to determine the capture models is always a simplification, and although cap- location and extents in the image of the person being cap- able of a realistic appearance, can also represent physically tured. The earliest and one of the most robust approaches unrealistic shapes and poses. If the algorithms are not care- to this task is termed chroma keying. This is where the fully constrained, these solutions can appear optimal for the background of the scene is painted a single specific colour, available data. To ensure only physically realistic solutions allowing the silhouette of the person (who is dressed in a are produced, the algorithms must be supported by con- suitable contrasting colour) to be easily segmented (Fig. 6). straints on the body model such as explicit joint limits [80] For environments where chroma keying is not possible, or probabilistic spaces of human pose and motion deduced there are a large number [82] of background subtraction al- by machine learning [81]. In either case, there exists a gorithms. However, these can all suffer from problems with balance between enforcing the constraints and trust- shadows, lighting changes, reflections and non-salient mo- ing the observed data to achieve a solution that is tions of the background (such as a crowd or other athletes). both plausible and precise. Image silhouettes are also inherently ambiguous and pro- vide no information on whether the observed subject is fa- Image Features for Markerless Motion Capture cing towards or away from the camera (Fig. 6). This A digital image, fundamentally, is a 2D grid of numbers ambiguity can only be reduced by the use of extra cameras each representing the brightness and colour of a small or more sophisticated image features. Where large numbers Fig. 5 Skinned Multi-Person Linear Model (SMPL) [79] body model. This model does not have an explicit skeleton. Instead, the surface of a person is represented by a mesh of triangles. A set of parameters (learnt through regression) allows the shape of the model to be changed from a neutral mean (left) to a fatter (middle) or thinner, taller, or other body shape. Once shaped, the centres of joints are inferred from the neutrally posed mesh, and then the mesh can be rotated around these joints to produce a posed body (right) Colyer et al. Sports Medicine - Open (2018) 4:24 Page 8 of 15 Fig. 6 Silhouette on the right from chroma keying the image on the left. When seen as only a silhouette, it is not possible to infer if the mannequin is facing towards or away from the camera of cameras are available, silhouettes can be combined into a given set of model parameters (body shape, bone lengths, 3D representation known as a visual hull [83], which is an joint angles), a representation of the model is generated. approximation of the space occupied by the observed per- This representation can then be compared against the fea- son (Fig. 7). More sophisticated 3D reconstructions can also tures extracted from the image and a single “error value” be carried out [84]; however, any added accuracy must be calculated, which represents how much the hypothesis dif- traded off against increased computational complexity. Im- fers from the observed data. In one possibility, the 3D tri- proving the reconstruction does not fully resolve all fitting angle mesh resulting from the predicted parameters can difficulties, however, and extra information that identifies be projected into the 2D image, and the overlap of the which regions of the silhouettes correspond to which re- mesh and the silhouette of the person can be maximised gions of the body are often needed to completely resolve all [92]. Alternatively, the 3D body model can be compared possible confusion [85]. Nevertheless, silhouettes have against a 3D reconstruction such as a visual hull by mini- formed a key aspect of many markerless motion capture mising the distances between the 3D vertices of the model, works including the work of Corazza et al. [86], which has and the 3D points of the visual hull [86, 93]through a reported some of the most accurate results for automatic standard algorithm known as iterative closest point. markerless body motion capture, and Liu et al. [87], which A key factor of generative approaches is the appropriate enables the kinematic motion analysis of multiple persons. definition of the function that compares a specific hypoth- The trend, however, has been to move away from the use esis with the information available in the images. If this is of image silhouettes to improve robustness, reduce ambigu- not carefully considered, then the search for the optimal ities, reduce the number of cameras and simplify capture set of model parameters can easily fail, resulting in poor procedures. In this regard, the work of Stoll et al. [75]is sig- estimates or nonsense configurations where joints bend at nificant for enabling the body model to fit to the image unrealistic angles and limbs penetrate inside the body. using only a simple colour model, while the advent of deep Constructing a cost function that is robust to image noise learning [88] and its provision of robust and fast body-part and to unrealistic model configurations is difficult mean- detectors has made dramatic improvements on what can ing generative models often need a reliable initial guess of be done outside of laboratory conditions [89, 90], including the model parameters. In extremes this would mean for- recognisingbodyposeofmanypeoplefroma singleuncali- cing the person being captured to assume a specific pose brated and moving camera [91]. at the start of tracking. If the fitting then becomes con- fused by occlusions, image noise or other failure, tracking Generative Algorithms will not be able to correct itself without manual interven- In generative motion capture approaches, the pose and tion. Researchers have attempted to address this situation shape of the person is determined by fitting the body using improved searching algorithms [92], extra informa- model to information extracted from the image. For a tion derived from robust body part detectors [90]and Colyer et al. Sports Medicine - Open (2018) 4:24 Page 9 of 15 Fig. 7 The generation of a visual hull, which is a type of 3D reconstruction of an object viewed from multiple cameras. Top row: images of an object are captured as 2D images from multiple directions. Middle row: these images are processed to produce silhouette images for each camera. Bottom left: the silhouettes are back-projected from each camera, resulting in cone-like regions of space. Bottom right: the intersection of these cones results in the visual hull recent pose-recognition algorithms [94–97], or by coup- possible to “teach” the computer how to determine the ling generative methods with discriminative methods [98]. pose of a simple skeleton model using only the image data. The most recent approaches in this family use deep learn- Discriminative Approaches ing to train a system that can identify the body parts of Discriminative algorithms avoid the process of iteratively multiple people, the probable ownership of joints, and then tuning the parameters of a body model to fit the image and quickly parse this to determine skeletons [91]. Alternatively, as such they are also often referred to as model-free algo- a database of pose examples can be created and then rithms. Compared with generative approaches, they will searched to discover the most similar known pose given the often have a much faster processing time, improved robust- current image, as used in previous studies [101–103]. ness and reduced dependence on an initial guess. However, The main difficulty in the use of discriminative algo- they can have reduced precision, and they require a very rithms is the creation of the exemplar data. If the avail- largedatabase ofexemplardata(farmorethanisrequired able data is insufficient, then poses, physiques and even even for constructing the statistical body shape models used camera positions that were not suitably represented will by generative algorithms) from which they can learn how lead to false results as the system will not be able to gen- to infer a result. eralise from what it “knows” to what it “sees”. This will Discriminative approaches have two major families. One also affect the precision of the result because the algo- approach is to discover a mapping directly from image fea- rithm is restricted to giving solutions close to what it tures to a description of pose, such as by using machine “knows” about, so small variations may not be fully rep- learning-based regression [99, 100]. In this way, it is resented in the results. As a result, discriminative Colyer et al. Sports Medicine - Open (2018) 4:24 Page 10 of 15 approaches are used as initial guesses for generative ap- pose-estimation algorithms such as global optimisation (in- proaches [98]. verse kinematics) with joint constraints [62]. As the “gold standard” methods are inappropriate in many contexts, and Summary of Markerless Approaches marker-based systems are the most frequently utilised mo- The current state-of-the-art shows the computer vi- tion analysis technique in the field, agreement between sion community aiming to develop solutions to mar- markerless and optoelectronicsystems wouldbe considered kerless motion capture that are applicable and to provideevidencefor the validityof markerlessmotion reliable outside of laboratory conditions. Although analysis techniques. carefully calibrated silhouette-based algorithms using There are already studies which have attempted to evalu- sophisticated subject-specific body models have ate the accuracy of markerless systems (summarised in shown the most accurate results to date, they have Table 2)bycomparing thekinematic output variables been limited to laboratory conditions using a large against those obtained using marker-based optoelectronic number of cameras [86]. By taking advantage of modern systems [10, 86, 93, 107–109] or manual digitisation [110]. technologies such as improved solvers [92], advanced image These validations mostly study relatively slow movements features and modern machine learning [100], recent works (typically walking gait), whereas to verify the utility of these areproviding solutionsthatreducethe required number of approaches in sports applications, much quicker move- cameras [104], allow moving cameras [105], increase the ments need to be thoroughly assessed. One clear observa- number of people that can be tracked and provide robust tion from these results is that transverse plane rotations are detection and fitting in varied environments [91]. The abil- currently difficult to extract accurately and reliably by mar- itytodothiswithout knowledgeofcameracalibration fur- kerlesstechnologies[107, 110]. ther improves the potential ease of use of future systems; In the computer vision community, it is common prac- however, calibration is likely to remain a necessity where tice to advancetechnologybyestablishingbenchmark data- precise measurement is needed such as in biomechanics. sets against which many authors can rank their algorithm’s performance. Two such benchmarks are the widely used Accuracy of Current Markerless Motion Capture Systems HumanEva dataset [111]and themore recent Human There are distinct differences in the accuracy requirements 3.6M dataset [112]. These datasets provide video of people betweenmotionanalysistechniques inthe fields of com- performing actions (walking, jogging, boxing etc.) while also puter vision and biomechanics, which must be taken into being tracked with marker-based tracking systems. Table 3 account when attempting to apply computer vision shows a sample of published comparison results for the methods more broadly across other disciplines. For in- HumanEva dataset. These results show that precision of stance, accuracy within computer vision (primarily enter- markerless techniques remains too low to be applicable for tainment applications) is typically assessed qualitatively and biomechanics analyses. However, the video data and mo- is primarily evaluated based on appearance. Conversely, in tioncapturedatainthe HumanEva datasetare themselves biomechanical settings, it is fundamental that any motion of limited quality. For example, videos are low resolution analysis system is capable of robustly quantifying subtle dif- and camera placement is sub-optimal, while markers are ferences in motion, which could be meaningful from a limited and sub-optimally located (often on relatively loose musculoskeletal performance or pathology perspective. clothing) with no marker clusters to aid tracking (Fig. 8). Nonetheless, there is no general consensus regarding the For comparison, the results of Corazza et al. [86]onHuma- minimum accuracy requirement of motion analysis systems nEva have a mean joint centre position error of 79 ± and the magnitudes of the inevitable measurement errors 12 mm, while on the authors’ own higher resolution data will vary depending on the context (laboratory vs. field), the with better camera and marker placement, a much smaller characteristics of the movement and the participant, the ex- 15 ± 10 mm error was achieved. perimental setup and how the human body is modelled. The discrepancies observed by Corazza et al. [86]be- As previously described, marker-based approaches are tween the validation results against the HumanEva bench- currently the most widely used systems in biomechanical mark and the more rigorously captured marker-based data laboratories. However, a prominent source of measurement show the difficulties of treating marker-based motion cap- error in marker-based systems is skin movement artefact ture as the criterion method. In fact, although these bench- [56], which violates the rigid body assumption underlying marks are useful for showing the general performance of these methods. Reports suggestthaterrors due to soft tis- different algorithms, neither the marker-based nor manual sue movement can exceed 10 mm for some anatomical digitising methods used to validate markerless technologies landmarks and 10° for some joint angles when compared to can provide exact “true” body pose due to experimental ar- more precise, yet invasive, methods (e.g. intra-cortical bone tefacts that are inevitably introduced. Additionally, ensuring pins) [106]. However, these errors in joint angle measure- a close match between the body model applied to both sys- ments may be reduced to 2°–4° by using more sophisticated tems is a challenge, which may necessitate “off-line” stages Colyer et al. Sports Medicine - Open (2018) 4:24 Page 11 of 15 Table 2 Overview of studies comparing markerless with conventional motion analysis systems Publication Movement(s) Markerless system description Procedure/ Number of Outcome analysed system for cameras comparison Trewartha et Starjump, Gen-locked video cameras (50 Hz), Manual 3 RMS differences for three movements ranged al. [110] somersaults subject-specific model digitising from 10 mm and 30 mm for pelvis location (TARGET and between 2° and 8° for body configuration system) angles. Corazza et al. Walking Visual hull construction and a priori Virtual 16 RMS errors of hip, knee and ankle angles [93] subject-specific model environment ranged from 2.0° (hip abduction/adduction) to (Poser 9.0 (ankle dorsi/plantar flexion) software) Mündermann Walking Video cameras (75 Hz), visual hull Qualisys 8 Average knee joint angle deviation: 2.3° et al. [10] construction and a priori subject- (120 Hz) (sagittal plane) and 1.6° (frontal plane). specific model Corazza et al. Walking Video cameras (120 Hz), visual hull Qualisys 8 Average deviations between joint (hip, knee, [86] construction and a priori subject- (120 Hz) ankle, shoulder, elbow and wrist) centres: specific model 15 mm mean absolute error (ranged from 9 to 19 mm) Choppin and Reaching, Microsoft Kinect (30 Hz) Motion 1 Kinect, 12 Flexion/extension and abduction/adduction of Wheat [72] throwing, Analysis optoelectronic hip, knee, elbow and shoulder; shoulder plane jumping Corporation and elevation studied. Maximum abduction (60 Hz) error: 44.1° and 13.9°. Maximum flexion error: 36.2° and 19.5° (NITE and IPIsoft tracking algorithms, respectively) Ceseracciu et Walking BTS BTS 8 Maximum RMS differences range: 11.0° (ankle al. [107] SMART-D SMART-D dorsi/ plantar flexion) to 34.7° (hip internal/ (100 Hz) (200 Hz) external rotation) Sandau et al. Walking Monochrome cameras (75 Hz), Ariel 8 RMS differences in lower limb 3D angles [108] unconstrained articulated model fit Performance ranged between 1.8° (hip abduction/ to 3D point clouds (aided by full Analysis adduction) and 4.9° (hip internal/external body patterned suit) System rotation) Ong et al. Walking and Point Grey cameras (25 Hz) Motion 2 markerless, RMS differences ranged from 0.2° (knee [109] jogging Analysis 8 marker- abduction/adduction of jogging) to 1.0° (ankle Corporation based dorsi/plantar flexion of walking). Significant (100 Hz) differences between markerless and marker- based for the ankle joint angles. RMS root mean square perhaps involving imaging, as in previous work [76]. Add- performance of markerless systems have also been consid- ing markers to the validation images might also unduly bias ered, such as utilising force plate data to analyse centre of the performance of a markerless system under test, if the mass movement [113] and creating virtual environments algorithm detects the markers and uses them for its benefit (synthesised images) in which a predefined model moves or if the markers adversely affect the performance of the with known kinematics [93]. Although synthesised images markerless algorithm (altering silhouette shapes, for ex- can be invaluable for developing an algorithm (synthetic ample) [107]. As such, alternative methods of validating the images were used for generating training images for Table 3 Selection of published validation results against the HumanEva datasets Publication 3D joint position error (mm) Standard deviation of error (mm) Corazza et al. [86] 79.0 11.5 Amin et al. [94] 54.5 Belagiannis et al. [95] 68.3 Saini et al. [92] 45.7 5.3 Guo et al. [103] 46.8 Elhayek et al. [90] 66.5 Rhodin et al. [98] 54.6 24.2 Bogo et al. [89] 79.9 Colyer et al. Sports Medicine - Open (2018) 4:24 Page 12 of 15 discriminative approach (to get good initialisation and ro- bustness) with a robust silhouette-free kinematic model fitting approach for precision. A fast, approximate pose estimation system has previ- ously been combined with a slow, more-accurate tech- nique in order to provide basic parameters to athletics coaches and inform training in real-time [76]. This type of system may have utility in the applied field by allowing some of the primary, “top-level” biomechanical determi- nants (for example, step frequency and step length in gait) to be fed-back during normal training or rehabilitation sit- uations. Importantly, the more complex and computation- ally expensive kinematic variables (such as 3D joint angles, which require modelling of the body) may still be acquired. However, the likelihood is that more time-consuming, offline processing will be necessary. This two-part approach could help address the apparent dis- connect between sports science research and practice [114] as short participant preparation times and timely feedback from the system may increase the perceived (and actual) value of such studies to those operating in the ap- plied field. Importantly, the more complex kinematic in- Fig. 8 An example image from the HumanEva dataset used to formation could still be computed and communicated to validate markerless systems within computer vision. White dots applied practitioners across longer time frames, but indicate the location of tracked reflective markers and the cyan lines equally these data can be used in research studies to con- represent the defined skeleton model fit to the marker data. Although useful as an early benchmark for markerless tracking tinually progress our scientific understanding of human systems, the dataset has clear limitations for assessing the quality of movement. any markerless tracking results, especially in the context of It should be noted that resolution (both spatial and tem- biomechanics. Notice that the markers are attached to clothing, poral) will affect the accuracy of markerless systems in the marker clusters are not utilised, and the joint centres inferred from same way as it does for marker-based systems. However, the fitted skeleton are not closely aligned with how the person appears in the image (e.g. right elbow and hip joints). See further video-based automatic systems must also consider the fact information in the text that the size of the data captured will be considerably larger and thus, markerless systems may need to compromise ac- Microsoft’s Kinect pose tracker [68]), the idealised image curacy to make a deployable, fast system feasible. Such a data is unlikely to capture the noise and error sources of system requires large amounts of video data to be handled real imagery. efficiently and effectively, which is likely to necessitate the purchase of an expensive (perhaps specially engineered) Future of Markerless Approaches to Analyse Motion in video-based system (e.g. machine vision). Sports Biomechanics and Rehabilitation It is clear that a broad range of markerless technologies Conclusions have emerged from computer vision research over recent Vision-based motion analysis methods within sports and times, which have the potential to be applied across di- rehabilitation applications have evolved substantially over verse disciplines and settings. The priorities and require- recent times and have allowed biomechanical research to ments for a markerless motion capture system will contribute a vast amount of meaningful information to depend on the research area and the unique capture envir- thesefields. However, themost widespread kinematic data onment, and are thus non-uniform across disciplines. In capture techniques (marker-based technologies and manual sports biomechanics and rehabilitation applications, mo- digitisation) are not without their drawbacks. Considerable tion analysis systems must be highly accurate in order to developments in computer vision have sparked interest in detect subtle changes in motion, as well as being adapt- markerless motion analysis and its possible wider applica- able, non-invasive and unencumbering. With these system tions. Although this potential is promising, it is not yet clear requirements in mind, the current progression of tech- exactly what accuracy can be achieved and whether such nologies suggests that the future of practical markerless systems can be effectively and routinely utilised in motion capture will lie with techniques such as those pre- field-based (more externally valid) settings. Over the com- sented by Elhayek et al. [90], which fuse together a ing years, collaborative research between computer vision Colyer et al. Sports Medicine - Open (2018) 4:24 Page 13 of 15 experts and biomechanists is required to further develop 7. Sjödahl C, Jarnlo G-B, Söderberg B, Persson BM. Kinematic and kinetic gait analysis in the sagittal plane of trans-femoral amputees before and after markerless techniques so they are able to meet the unique special gait re-education. Prosthetics Orthot Int. 2002;26(2):101–12. practical and accuracy requirements of motion analysis 8. O'Rourke J, Badler NI. Model-based image analysis of human motion using within sports and rehabilitation contexts. constraint propagation. IEEE Trans Pattern Anal Mach Intell. 1980;2(6):522–36. 9. Leung MK, Yang Y-H. First sight: a human body outline labeling system. IEEE Abbreviations Trans Pattern Anal Mach Intell. 1995;17(4):359–77. 2D: Two-dimensional; 3D: Three-dimensional; DOF: Degrees of freedom; 10. Mündermann L, Corazza S, Andriacchi TP. The evolution of methods for the capture RMS: Root mean square; SMPL: Skinned Multi-Person Linear Model of human movement leading to markerless motion capture for biomechanical applications. J Neuroeng Rehabil. 2006; https://doi.org/10.1186/1743-0003-3-6 Funding 11. Bezodis IN, Kerwin DG, Salo AIT. Lower-limb mechanics during the support phase This review was funded by CAMERA, the RCUK Centre for the Analysis of of maximum-velocity sprint running. Med Sci Sports Exerc. 2008;40(4):707–15. Motion, Entertainment Research and Applications, EP/M023281/1. 12. Churchill SM, Salo AIT, Trewartha G. The effect of the bend on technique and performance during maximal effort sprinting. Sports Authors’ Contributions Biomechanics. 2015;14(1):106–21. SC, ME, DC and AS all participated in planning the conception and design of 13. Hiley MJ, Yeadon MR. Achieving consistent performance in a complex whole this review article. SC and ME completed the majority of draft writing. DC body movement: the Tkatchev on high bar. Hum Mov Sci. 2012;31(4):834–43. and AS provided critical revisions for the manuscript. All authors provided 14. Bezodis NE, Trewartha G, Wilson C, Irwin G. Contributions of the non-kicking-side the final approval for the final submitted version and agreed to be arm to rugby place-kicking technique. Sports Biomechanics. 2007;6:171–86. accountable for all aspects of the work. 15. Winter DA. Biomechanics and motor control of human movement. 2nd ed. New York: Wiley; 1990. Authors’ Information 16. Martin JC, Brown NAT. Joint-specific power production and fatigue during SC has a PhD and is a Post-Doctoral Research Associate in the CAMERA project maximal cycling. J Biomech. 2009;42(4):474–9. (see funding above) with expertise in analysing athletes’ technique. ME has a 17. Devita P, Hortobagyi T, Barrier J, Torry M, Glover KL, Speroni DL, et al. Gait PhD and is a Post-Doctoral Research Associate in the CAMERA project with adaptation before and after anterior cruciate ligament reconstruction expertise in computer vision with time spent in the computer industry before surgery. Med Sci Sports Exerc. 1997;29(7):853–9. joining back to academia. DC has a PhD and is a Professor in Computer Science 18. Camomilla V, Cereatti A, Cutti AG, Fantozzi S, Stagni R, Vannozzi G. specialising in computer vision. He is the principal investigator and the director Methodological factors affecting joint moments estimation in clinical gait of the CAMERA project. AS has a PhD and is a Reader (Associate Professor) in analysis: a systematic review. Biomed Eng Online. 2017;16(106); https://doi. Sports Biomechanics with expertise in analysing athletes’ technique. He org/10.1186/s12938-017-0396-x. is a co-investigator in the CAMERA project. 19. Matsas A, Taylor N, McBurney H. Knee joint kinematics from familiarised treadmill walking can be generalised to overground walking in young Ethics Approval and Consent to Participate unimpaired subjects. Gait Posture. 2000;11(1):46–53. Not applicable 20. Schmölzer B, Müller W. Individual flight styles in ski jumping: results obtained during Olympic Games competitions. J Biomech. 2005;38(5):1055–65. Competing Interests 21. Seminati E, Marzari A, Vacondio O, Minetti AE. Shoulder 3D range of motion The authors Steffi L. Colyer, Murray Evans, Darren P. Cosker and Aki I. T. Salo and humerus rotation in two volleyball spike techniques: injury prevention declare that they have no competing interests. and performance. Sports Biomechanics. 2015;14(2):216–31. 22. Zhang Z. Camera calibration. In: Ikeuchi K, editor. Computer vision: a reference guide. Boston: Springer US; 2014. p. 76–7. Publisher’sNote 23. Abdel-Aziz YI, Karara HM, Hauck M. Direct linear transformation from Springer Nature remains neutral with regard to jurisdictional claims in comparator coordinates into object space coordinates in close-range published maps and institutional affiliations. photogrammetry. Photogramm Eng Remote Sens. 2015;81(2):103–7. 24. Mayagoitia RE, Nene AV, Veltink PH. Accelerometer and rate gyroscope Author details measurement of kinematics: an inexpensive alternative to optical motion CAMERA—Centre for the Analysis of Motion, Entertainment Research and analysis systems. J Biomech. 2002;35(4):537–42. Applications, University of Bath, Bath BA2 7AY, UK. Department for Health, University of Bath, Bath BA2 7AY, UK. Department of Computer Science, 25. Ganter N, Krüger A, Gohla M, Witte K, Edelmann-Nusser J. Applicability of a University of Bath, Bath BA2 7AY, UK. full body inertial measurement system for kinematic analysis of the discus throw. Michigan: Proceedings of the 28th conference of the International Received: 23 January 2018 Accepted: 22 May 2018 Society of Biomechanics in Sports; 2010. 26. Eckardt F, Münz A, Witte K. Application of a full body inertial measurement system in dressage riding. J Equine Vet Sci. 2014;34:1294–9. References 27. de Magalhaes FA, Vannozzi G, Gatta G, Fantozzi S. Wearable inertial sensors in 1. Muybridge E. Complete human and animal locomotion (all 781 plates from swimming motion analysis: a systematic review. J Sports Sci. 2015;33(7):732–45. the 1887 animal locomotion). In: Cappozzo A, Marchetti M, Tosi V, editors. 28. Dadashi F, Crettenand F, Millet GP, Aminian K. Front-crawl instantaneous Biolocomotion: a century of research using moving pictures. Rome: velocity estimation using a wearable inertial measurement unit. Sensors. Promograph; 1979. p. 69. 2012;12(10):12927–39. 2. Cimolin V, Galli M. Summary measures for clinical gait analysis: a literature 29. Atha J. Current techniques for measuring motion. Appl Ergon. 1984;15(4):245–57. review. Gait Posture. 2014;39(4):1005–10. 30. Salo AIT, Grimshaw PN, Marar L. 3-D biomechanical analysis of sprint hurdles at 3. Cosker D, Eisert P, Helzle V. Facial capture and animation in visual effects. In: different competitive levels. Med Sci Sports Exerc. 1997;29(2):231–7. Magnor MA, Grau O, Sorkine-Hornung O, Theobalt C, editors. Digital 31. Manning ML, Irwin G, Gittoes MJR, Kerwin DG. Influence of longswing representations of the real world: how to capture, model, and render visual technique on the kinematics and key release parameters of the straddle reality. Boca Raton, FL: CRC Press; 2015. p. 311–21. Tkachev on uneven bars. Sports Biomechanics. 2011;10(3):161–73. 4. Bezodis NE, Salo AIT, Trewartha G. Relationships between lower-limb 32. Lees A, Graham-Smith P, Fowler N. A biomechanical analysis of the last kinematics and block phase performance in a cross section of sprinters. Eur stride, touchdown, and takeoff characteristics of the men’s long jump. J J Sport Sci. 2015;15(2):118–24. Appl Biomech. 1994;10:61–78. 5. Ford KR, Myer GD, Toms HE, Hewett TE. Gender differences in the kinematics of 33. Portus MR, Rosemond CD, Rath DA. Cricket: fast bowling arm actions and unanticipated cutting in young athletes. Med Sci Sports Exerc. 2005;37(1):124–9. the illegal delivery law in me’s high performance cricket matches. Sports 6. Devita P, Hortobagyi T, Barrier J. Gait biomechanics are not normal after Biomechanics. 2006;5(2):215–30. anterior cruciate ligament reconstruction and accelerated rehabilitation. Med Sci Sports Exerc. 1998;30(10):1481–8. 34. Procter P, Paul JP. Ankle joint biomechanics. J Biomech. 1982;15(9):627–34. Colyer et al. Sports Medicine - Open (2018) 4:24 Page 14 of 15 35. Ericson M. On the biomechanics of cycling. A study of joint and muscle load 61. Snyder KR, Earl JE, O'Connor KM, Ebersole KT. Resistance training is during exercise on the bicycle ergometer. Scand J Rehabil Med. 1986;16:1–43. accompanied by increases in hip strength and changes in lower extremity 36. Bobbert MF, Huijing PA, van Ingen Schenau GJ. A model of the human triceps biomechanics during running. Clin Biomech. 2009;24(1):26–34. surae muscle-tendon complex applied to jumping. J Biomech. 1986;19(11):887–98. 62. Lu TW, O'Connor JJ. Bone position estimation for skin marker co-ordinates 37. Wilson DJ,Smith BK,Gibson JK, Choe BK,GabaBC, Voelz JT. Accuracy of using global optimisation with joint constraints. J Biomech. 1999;32(2):129–34. digitization using automated and manual methods. Phys Ther. 1999;79(6):558–66. 63. Nedergaard NJ, Heinen F, Sloth S, Hébert-Losier K, Holmberg H-C, Kersting UG. The effect of light reflections from the snow on kinematic data collected using 38. Scholz JP, Millford JP. Accuracy and precision of the PEAK performance stereo-photogrammetry with passive markers. Sports Eng. 2014;17:97–102. technologies motion measurement system. J Mot Behav. 1993;25(1):2–7. 39. Salo AIT, Grimshaw PN. An examination of kinematic variability of motion 64. Moeslund TB, Hilton A, Krüger V. A survey of advances in vision-based human analysis in sprint hurdles. J Appl Biomech. 1998;14:211–22. motion capture and analysis. Comput Vis Image Underst. 2006;104:90–126. 40. Bezodis I, Salo A, Kerwin D. A longitudinal case study of step characteristics 65. Poppe R. Vision-based human motion analysis: an overview. Comput Vis in a world class sprint athlete. In: Kwon Y-H, Shim J, Shim JK, Shin I-S, Image Underst. 2007;108:4–18. editors. . Seoul: Proceedings of 26th international conference on 66. Holte MB, Tran C, Trivedi MM, Moeslund TB. Human pose estimation and biomechanics in Sports; 2008. activity recognition from multi-view videos: comparative explorations of 41. Bezodis NE, Salo AIT, Trewartha G. Choice of sprint start performance measure recent developments. Select Topics Signal Process. 2012;6:538–52. affects the performance-based ranking within a group of sprinters: which is 67. Yang SXM, Christiansen MS, Larsen PK, Alkjær T, Moeslund TB, Simonsen EB, the most appropriate measure? Sports Biomechanics. 2010;9(4):258–69. et al. Markerless motion capture systems for tracking of persons in forensic biomechanics: an overview. Comput Methods Biomechanics Biomed Eng: 42. Churchill AJG, Halligan PW, Wade DT. RIVCAM: a simple video-based Imaging Visual. 2014;2(1):46–65. kinematic analysis for clinical disorders of gait. Comput Methods Prog Biomed. 2002;69(3):197–209. 68. Shotton J, Fitzgibbon A, Blake A, Kipman A, Finocchio M, Moore R, et al. 43. Elliott B, Alderson J. Laboratory versus field testing in cricket bowling: a Real-time human pose recognition in parts from a single depth image. review of current and past practice in modelling techniques. Sports Colorado: Proceedings of the IEEE Conference on Computer Vision and Biomechanics. 2007;6(1):99–108. Pattern Recognition; 2011. 44. Elliott BC, Alderson JA, Denver ER. System and modelling errors in motion 69. Ye M, Zhang Q, Wang L, Zhu J, Yang R, Gall JA. Survey on human motion analysis: implications for the measurement of the elbow angle in cricket analysis from depth data. In: Grzegorzek M, Theobalt C, Koch R, Kolb A, bowling. J Biomech. 2007;40(12):2679–85. editors. Time-of-flight and depth imaging sensors, algorithms, and 45. Richards JG. The measurement of human motion: a comparison of applications. Berlin, Heidelberg: Springer; 2013. p. 149–87. commercially available systems. Hum Mov Sci. 1999;18(5):589–602. 70. Sarbolandi H, Lefloch D, Kolb A. Kinect range sensing: structured-light versus time-of-flight kinect. J Comput Vision Image Understand. 2015;139:1–20. 46. Ferrari A, Benedetti MG, Pavan E, Frigo C, Bettinelli D, Rabuffetti M, et al. 71. Wang Q, Kurillo G, Ofli F, Bajcsy R. Evaluation of pose tracking accuracy in Quantitative comparison of five current protocols in gait analysis. Gait the first and second generations of Microsoft Kinect. IEEE Int Confer Healthc Posture. 2008;28:207–16. Inform (ICHI). 2015:380–9. 47. Growney E, Meglan D, Johnson M, Cahalan T, An K-N. Repeated measures of adult normal walking using a video tracking system. Gait 72. Choppin S, Wheat J. The potential of the Microsoft Kinect in sports analysis Posture. 1997;6(2):147–62. and biomechanics. Sports Technol. 2013;6(2):78–85. 48. Tsushima H, Morris ME, McGinley J. Test-retest reliability and inter-tester 73. Schmitz A, Ye M, Boggess G, Shapiro R, Yang R, Noehren B. The reliability of kinematic data from a three-dimensional gait analysis system. J measurement of in vivo joint angles during a squat using a single camera Japan Phys Ther Assoc. 2003;6(1):9–17. markerless motion capture system as compared to a marker based system. 49. Cappozzo A, Catani F, Leardini A, Benedetti MG, Della Croce U. Position and Gait Posture. 2015;41:694–8. orientation in space of bones during movement: experimental artefacts. 74. Gavrila DM, Davis LS. 3-D model-based tracking of humans in action: a Clin Biomech. 1996;11(2):90–100. multi-view approach. San Juan, Puerto Rico: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 1996. 50. Stagni R, Fantozzi S, Cappello A, Leardini A. Quantification of soft tissue artefact 75. Stoll C, Hasler N, Gall J, Seidel HP, Theobalt C. Fast articulated motion in motion analysis by combining 3D fluoroscopy and stereophotogrammetry: tracking using a sums of Gaussians body model. Barcelona: Proceedings of a study on two subjects. Clin Biomech. 2005;20(3):320–9. the international conference on computer vision; 2011. 51. Maslen BA, Ackland TR. Radiographic study of skin displacement errors in the foot and ankle during standing. Clin Biomech. 1994;9:291–6. 76. El-Sallam A, Bennamoun M, Sohel F, Alderson J, Lyttle A, Rossi R. A low cost 52. Tranberg R, Karlsson D. The relative skin movement of the foot: a 2D 3D markerless system for the reconstruction of athletic techniques. Tampa: roentgen photogrammetry study. Clin Biomech. 1998;13(1):71–6. IEEE Workshop on Applications of Computer Vision; 2013. 53. Fuller J, Liu L-J, Mann RW. A comparison of lower-extremity skeletal 77. Allen B, Curless B, Popović Z. The space of human body shapes: reconstruction kinematics measured using skin- and pin-mounted markers. Hum Mov Sci. and parameterization from range scans. ACM Trans Graph. 2003;22(3):587–94. 1997;16(2–3):219–42. 78. Anguelov D, Srinivasan P, Koller D, Thrun S, Rodgers J, Davis J. SCAPE: shape 54. Benoit DL, Ramsey DM, Lamontagne M, Xu L, Wretenberg P, Renström P. completion and animation of people. ACM Trans Graph. 2005;24(3):408–16. 79. Loper M, Mahmood N, Romero J, Pons-Moll G, Black MJ. SMPL: a skinned Effect of skin movement artifact on knee kinematics during gait and cutting multi-person linear model. Proc of SIGGRAPH Asia. 2015;34(6):1–16. motions measured in vivo. Gait Posture. 2006;24(2):152–64. 55. Reinschmidt C, van den Bogert AJ, Nigg BM, Lundberg A, Murphy N. Effect 80. Akhter IB, Black MJ. Pose-conditioned joint angle limits for 3D human pose reconstruction. Boston: Proceedings of the IEEE Conference on Computer of skin movement on the analysis of skeletal knee joint motion during Vision and Pattern Recognition; 2015. running. J Biomech. 1997;30(7):729–32. 56. Leardini A, Chiari L, Della Croce U, Cappozzo A. Human movement analysis 81. Holden D, Saito J, Komura T, Joyce T. Learning motion manifolds with using stereophotogrammetry: part 3. Soft tissue artifact assessment and convolutional autoencoders. SIGGRAPH Asia Tech Briefs. 2015;18; https://doi. compensation. Gait Posture. 2005;21(2):212–25. org/10.1145/2820903.2820918. 57. Manal K, McClay I, Stanhope S, Richards J, Galinat B. Comparison of surface 82. Bouwmans T. Traditional and recent approaches in background modeling mounted markers and attachment methods in estimating tibial rotations for foreground detection: an overview. Comput Sci Rev. 2014;11-12:31–66. during walking: an in vivo study. Gait Posture. 2000;11(1):38–45. 83. Laurentini A. The visual hull concept for silhouette-based image 58. Cappozzo A, Catani F, Della Croce U, Leardini A. Position and orientation in understanding. IEEE Trans Pattern Anal Mach Intell. 1994;16(2):150–62. space of bones during movement: anatomical frame definition and 84. Furukawa Y, Hernández C. Multi-view stereo: a tutorial. Found Trends determination. Clin Biomech. 1995;10(4):171–8. Comput Graphics Vision. 2015;9(1–2):1–148. 59. Milner CE, Ferber R, Pollard CD, Hamill J, Davis IS. Biomechanical factors 85. Kanaujia A, Kittens N, Ramanathan N. Part segmentation of visual hull for 3D associated with tibial stress fracture in female runners. Med Sci Sports Exerc. human pose estimation. Oregon: IEEE Conference on Computer Vision and 2006;38(2):323–8. Pattern Recognition Workshops; 2013. 60. Chmielewski TL, Rudolph KS, Fitzgerald GK, Axe MJ, Snyder-Mackler L. 86. Corazza S, Mündermann L, Gambaretto E, Ferrigno G, Andriacchi TP. Biomechanical evidence supporting a differential response to acute ACL Markerless motion capture through visual hull, articulated ICP and subject injury. Clin Biomech. 2001;16(7):586–91. specific model generation. Int J Comput Vis. 2010;87:156–69. Colyer et al. Sports Medicine - Open (2018) 4:24 Page 15 of 15 87. Liu Y, Gall J, Stoll C, Dai Q, Seidel HP, Theobalt C. Markerless motion capture 111. Sigal L, Balan AO, Black MJ. HumanEva: synchronized video and motion of multiple characters using multiview image segmentation. IEEE Trans capture dataset and baseline algorithm for evaluation of articulated human Pattern Anal Mach Intell. 2013;35(11):2720–35. motion. Int J Comput Vis. 2010;87:4–27. 88. LeCun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015;521(7553):436–44. 112. Ionescu C, Papava D, Olaru V, Sminchisescu C. Human3.6M: large scale datasets and predictive methods for 3D human sensing in natural 89. Bogo F, Kanazawa A, Lassner C, Gehler P, Romero J, Black MJ. Keep it SMPL: environments. IEEE Trans Pattern Anal Mach Intell. 2014;36(7):1325–39. automatic estimation of 3D human pose and shape from a single image. Amsterdam: Proceedings of the European Conference on Computer Vision; 2016. 113. Corazza S, Andriacchi TP. Posturographic analysis through markerless motion capture without ground reaction forces measurement. J Biomech. 90. Elhayek A, de Aguiar E, Jain A, Tompson J, Pishchulin L, Andriluka M, et al. 2009;42(3):370–4. Efficient ConvNet-based marker-less motion capture in general scenes with 114. Bishop D. An applied research model for the sport sciences. Sports Med. a low number of cameras. Boston: Proceedings of the IEEE Conference on 2008;38(3):253–63. Computer Vision and Pattern Recognition; 2015. 91. Cao Z, Simon T, Wei SE, Sheikh Y. Realtime multi-person 2D pose estimation using part affinity fields. Comput Vision Pattern Recogn. 2016; https://doi. org/10.1109/CVPR.2017.143. 92. Saini S, Zakaria N, Rambli DRA, Sulaiman S. Markerless human motion tracking using hierarchical multi-swarm cooperative particle swarm optimization. PLoS One. 2015;10(5):1–22. 93. Corazza S, Mündermann L, Chaudhari AM, Demattio T, Cobelli C, Andriacchi TP. A markerless motion capture system to study musculoskeletal biomechanics: visual hull and simulated annealing approach. Ann Biomed Eng. 2006;34(6):1019–29. 94. Amin S, Andriluka M, Rohrback M, Schiele B. Multi-view pictorial structures for 3D human pose estimation. British Machine Vision Confer. 2013; https:// doi.org/10.5244/C.27.45. 95. Belagiannis V, Amin S, Andriluka M, Schiele B, Navab N, Ilic S. 3D pictorial structures for multiple human pose estimation. Ohio: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2014. 96. Burenius M, Sullivan J, Carlsson S. 3D pictorial structures for multiple view articulated pose estimation. Oregon: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2013. 97. Kazemi V, Burenius M, Azizipour H, Sullivan J. Multi-view body part recognition with random forests. Br Machine Vision Confer. 2013; https:// doi.org/10.5244/C.27.48. 98. Rhodin H, Robertini N, Casas D, Richardt C, Seidel H-P, Theobalt C. General automatic human shape and motion capture using volumetric contour cues. In: Leibe B, Matas J, Sebe N, Welling M, editors. . Amsterdam: Proceedings of the European Conference on Computer Vision; 2016. 99. Agarwal A, Triggs B. Recovering 3D human pose from monocular images. Pattern Anal Machine Intelligence. 2006;28:44–58. 100. Hong C, Yu K, Xie Y, Chen X. Multi-view deep learning for image-based pose recovery. Hangzhou: Proceedings of the IEEE 16th International Conference on Communication Technology; 2015. 101. Chen C, Yang Y, Nie F, Odobez J-M. 3D human pose recovery from image by efficient visual feature selection. Comput Vis Image Underst. 2011;115(3):290–9. 102. Babagholami-Mohamadabadi B, Jourabloo A, Zarghami A, Kasaei S. A Bayesian framework for sparse representation-based 3-D human pose estimation. IEEE Signal Proc Lett. 2014;21(3):297–300. 103. Guo Y, Chen Z, Yu J. Supervised spectral embedding for human pose estimation. Suzhou: Proceedings for the 5th International Conference of Intelligence Science and Big Data Engineering Image and Video Data Engineering; 2015. 104. Mehta D, Sridhar S, Sotnychenko O, Rhodin H, Shafiei M, Seidel H-P, et al. VNect: real-time 3D human pose estimation with a single RGB camera. ACM Trans Graph. 2017;36(4): https://doi.org/10.1145/3072959.3073596. 105. Elhayek A, Stoll C, Kim KI, Theobalt C. Outdoor human motion capture by simultaneous optimization of pose and camera parameters. Computer Graphics Forum. 2015;34(6):86–98. 106. Peters A, Galna B, Sangeux M, Morris M, Baker R. Quantification of soft tissue artifact in lower limb human motion analysis: a systematic review. Gait Posture. 2010;31:1–8. 107. Ceseracciu E, Sawacha Z, Cobelli C. Comparison of markerless and marker- based motion capture technologies through simultaneous data collection during gait: proof of concept. PLoS One. 2014;9(3):e87640. https://doi.org/ 10.1371/journal.pone.0087640 108. Sandau M, Koblauch H, Moeslund TB, Aanæs H, Alkjær T, Simonsen EB. Markerless motion capture can provide reliable 3D gait kinematics in the sagittal and frontal plane. Med Eng Phys. 2014;36(9):1168–75. 109. Ong A, Harris IS, Hamill J. The efficacy of a video-based marker-less tracking system for gait analysis. Comput Methods Biomechanics Biomed Eng. 2017; 20(10):1089–95. 110. Trewartha G, Yeadon MR, Knight JP. Marker-free tracking of aerial movements. In: Müller R, Gerber H, Stacoff A, editors. . Zürich: Proceedings of the 18th congress of the international society of Biomechanics; 2001.

Journal

Sports Medicine - OpenSpringer Journals

Published: Jun 5, 2018

References

You’re reading a free preview. Subscribe to read the entire article.


DeepDyve is your
personal research library

It’s your single place to instantly
discover and read the research
that matters to you.

Enjoy affordable access to
over 18 million articles from more than
15,000 peer-reviewed journals.

All for just $49/month

Explore the DeepDyve Library

Search

Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly

Organize

Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place.

Access

Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals.

Your journals are on DeepDyve

Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.

All the latest content is available, no embargo periods.

See the journals in your area

DeepDyve

Freelancer

DeepDyve

Pro

Price

FREE

$49/month
$360/year

Save searches from
Google Scholar,
PubMed

Create lists to
organize your research

Export lists, citations

Read DeepDyve articles

Abstract access only

Unlimited access to over
18 million full-text articles

Print

20 pages / month

PDF Discount

20% off