Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Global rigid registration of CT to video in laparoscopic liver surgery

Global rigid registration of CT to video in laparoscopic liver surgery Purpose Image-guidance systems have the potential to aid in laparoscopic interventions by providing sub-surface structure information and tumour localisation. The registration of a preoperative 3D image with the intraoperative laparoscopic video feed is an important component of image guidance, which should be fast, robust and cause minimal disruption to the surgical procedure. Most methods for rigid and non-rigid registration require a good initial alignment. However, in most research systems for abdominal surgery, the user has to manually rotate and translate the models, which is usually difficult to perform quickly and intuitively. Methods We propose a fast, global method for the initial rigid alignment between a 3D mesh derived from a preoperative CT of the liver and a surface reconstruction of the intraoperative scene. We formulate the shape matching problem as a quadratic assignment problem which minimises the dissimilarity between feature descriptors while enforcing geometrical consistency between all the feature points. We incorporate a novel constraint based on the liver contours which deals specifically with the challenges introduced by laparoscopic data. Results We validate our proposed method on synthetic data, on a liver phantom and on retrospective clinical data acquired during a laparoscopic liver resection. We show robustness over reduced partial size and increasing levels of deformation. Our results on the phantom and on the real data show good initial alignment, which can successfully converge to the correct position using fine alignment techniques. Furthermore, since we can pre-process the CT scan before surgery, the proposed method runs faster than current algorithms. Conclusion The proposed shape matching method can provide a fast, global initial registration, which can be further refined by fine alignment methods. This approach will lead to a more usable and intuitive image-guidance system for laparoscopic liver surgery. Keywords Image guidance · Laparoscopic liver surgery · Global registration · Shape matching · Surface descriptors · Computer-assisted surgery Introduction tion, constrained vantage point, limited field of view, poor haptic feedback and occluded anatomy [1]. Image guidance Minimally invasive surgery offers the patient major bene- aims to assist the clinicians in localising and tracking sub- fits over open surgery, including less trauma, less pain and surface structures such as abnormalities or major vessel trees. shorter hospital stays. However, these interventions present Thus, these systems have the potential to aid in surgical inter- several challenges for clinicians, such as weak depth percep- ventions through improved resection quality and a reduction in positive surgical margins [2]. The safety margin around a B Maria R. Robu possible tumour in current laparoscopic procedures is a min- [email protected] imum of 10 mm [3], so it is considered desirable to develop systems with accuracy below 5 mm on average [4,5]. Current Wellcome/EPSRC Centre for Interventional and Surgical Sciences, University College London, London, UK rigid registration methods achieve accuracies of approxi- mately 10 mm in phantom experiments [5–7]. Improving the Centre For Medical Image Computing, University College London, London, UK robustness, accessibility and reliability of image-guidance Division of Surgery and Interventional Science, University College London, London, UK 123 948 International Journal of Computer Assisted Radiology and Surgery (2018) 13:947–956 systems could potentially increase the number of patients of an additional CT or MRI scan immediately before the benefiting from minimally invasive surgeries. intervention. Most hospitals require an abdominal CT scan to be Surface acquisition of the intraoperative scene has been acquired before surgery for laparoscopic liver interventions. proposed as an alternative. Several strategies have been A 3D model of the liver, major vessel trees and any abnormal- developed using laser range scanners [6], optically tracked ities can be segmented from the CT scan. The registration of probes [11], time-of-flight (TOF) data [7] and stereo recon- the preoperative liver model and the intraoperative laparo- struction [5]. Once the surface is acquired, the clinician is scopic images is an essential step towards developing an required to delineate salient anatomical features leading to image-guidance system. Most methods in the literature can a point-based initial alignment [6,11] or to a more com- be divided between coarse alignment, defined as a global plex non-rigid optimisation framework [3]. Another option alignment which can match the surfaces irrespective of their to obtain the rigid alignment is to manually rotate and trans- initial transformation, and fine alignment techniques—in late the 3D preoperative image until it fits the intraoperative which a good initial alignment is already provided as a data [5]. While some level of user interaction is needed for starting point. In this paper, we focus on coarse alignment these approaches, it is generally more intuitive and faster to methods for surface-based registration. select salient features than to manipulate the six degrees of Furthermore, most methods are only applicable in open freedom associated with a rigid transform. surgery [4,6–8] as a large surface of the intraoperative scene Hybrid methods have been proposed using cone beam is required. However, surfaces acquired laparoscopically CT and fluoroscopy [15] as bridging modalities between the present the additional challenges that the camera has access to laparoscopic camera and the preoperative CT, which deliv- a restricted region of the abdomen leading to an even smaller ers an additional radiation dose to the patient. Feuerstein partial view, lack of reliable landmarks and significant defor- et al. [16] propose using intraoperative cone beam CT and mation from pneumoperitoneum [7]. We address and discuss optical tracking to register directly to the laparoscopic view the challenges inherent to laparoscopic surgeries which moti- without using preoperative information. While their methods vated our method. achieve promising results, they are based on advanced hard- We propose a fast, semi-automatic global alignment ware which might not be available in most clinical settings. method which achieves the initial alignment between the Finally, fully automated techniques have been proposed preoperative CT model of the liver surface and a surface in [7,17]. Fusaglia et al. [17] developed an exhaustive search reconstruction of the intraoperative scene. The resulting over the principal directions of the intraoperative surface, transformation could be further improved by fine alignment which is acquired using a laparoscopic laser pointer. While algorithms [9,10] in order to get a rigid [5,6,11] or a non-rigid their proposed approach is promising, it still introduces addi- alignment [3] between the two modalities. Our approach can tional tools into the clinical workflow. Dos Santos et al. [7] lead to a faster and more intuitive use of image-guidance introduced a novel automatic method to establish surface systems in laparoscopic surgeries. We show robustness to correspondences between the 3D preoperative mesh and the reduced partial sizes and increasing deformations in the intra- intraoperative surface acquired with a TOF camera in open operative model on synthetic data. Moreover, we evaluate the liver surgery. Their approach was validated on a phantom proposed method on a liver phantom and on retrospective data of the human liver and on an ex-vivo porcine liver with from a dataset acquired in a laparoscopic liver resection with accuracy better than 1 cm and computation time ranging promising results. from one minute to 5.5 h. While their phantom validation under deformation from breathing motion can be sufficient for open surgery, livers in laparoscopic interventions undergo Background significant general deformation due to pneumoperitoneum. The initial rigid registration of the preoperative 3D image and Furthermore, it is unclear how both methods [7,17] would the intraoperative scene has been explored through methods be translated to laparoscopic interventions since they rely on that rely on fiducials, user interaction and through fully auto- large surfaces of the liver being visible. mated methods. While promising results have been achieved in the liter- Several approaches propose the use of fiducials, either ature, we aim to develop an image-guidance system which on the patient skin [12] for needle guidance, or on the can handle the challenges of laparoscopic interventions and organ itself [13] for tracking in laparoscopic partial nephrec- is easy to integrate with the current clinical protocol without additional hardware or advanced cameras. Furthermore, the tomy. Another more robust option, which is applicable in laparoscopic interventions, would be to attach metabolisable system should be usable during surgical interventions, with minimal disruption and fast computation. fluorescent markers on the organ [14]. Such fiducials can be seen in both modalities. However, these strategies are disrup- tive to the clinical workflow since they require the acquisition 123 International Journal of Computer Assisted Radiology and Surgery (2018) 13:947–956 949 Fig. 1 An overview of our proposed global alignment framework, showing the preoperative and intraoperative steps Contributions Let M be the moving (preoperative CT mesh) model, and let T be the target (intraoperative point cloud) model. We propose a fast, semi-automatic method to obtain a global Sets of features, {m }⊂ M and {t }⊂ T , are selected on r s initial alignment between a 3D liver model extracted from both surfaces with f (·) as their corresponding descriptor. Let the preoperative CT scan and a surface reconstruction of the d (x , y) be the geodesic distance between any two points, x intraoperative scene. and y, on a surface. An existing formulation of shape matching is extended to Generally, it is difficult to match surfaces in laparoscopic incorporate an additional constraint based on the contours of liver surgery only based on descriptors since the surfaces the organ (the ridge line—see Fig. 1), which can be identified lack prominent, uniquely identifiable features. The use of on both surfaces with high confidence. Once the delineation geometric consistency between the correspondences on both of the liver ridge line is given in the two modalities, no further shapes can further constrain the registration problem. user interaction or initialisation is required for the alignment Shape matching can be formulated as a quadratic assign- stage. The proposed method is able to robustly estimate a ment problem (QAP): correspondence set between the two surfaces under defor- mation, sparse data, partial views and realistic noise levels. E (C ) = d( f (m ), f (t )) i i We validate our technique in a simulated environment to (m,t ) ∈C show robustness to partial data and deformation. Moreover, + (d (m , m ) − d (t , t )) (1) g i j g i j we provide quantitative results obtained on a liver phantom (m,t ) ∈C (m,t ) ∈C i j and qualitative results on retrospective data from a laparo- scopic liver resection to illustrate feasibility in a realistic where C ={(m, t ) }⊂ M T is the initial correspondence i × clinical setting. set composed of candidate pairs of feature points from the two surfaces, d( f (·), f (·)) is the distance between the feature descriptors and d (·, ·) is the geodesic distance between two correspondences on the same surface. This energy function Methods aims to output a set of correspondences for which the dissim- ilarity between the descriptors is minimised and the geodesic Figure 1 illustrates the main steps of the proposed pipeline. distances between pairs of correspondences are maintained. The input data include the segmentation of the 3D mesh from While this approach works well in the vision literature for the CT scan, a surface reconstruction represented as a point complex shapes [18], the intraoperative surfaces pose several cloud of the intraoperative data and the segmented contours challenges. It has been previously discussed in [7] that con- on both surfaces. The liver contour is defined as the ridge straining the correspondence set based only on the geodesic line visible in yellow in Fig. 1 on both the preoperative and distances between them is still ambiguous for almost flat sur- the intraoperative surfaces. faces, in which the same spatial configuration of features can 123 950 International Journal of Computer Assisted Radiology and Surgery (2018) 13:947–956 Fig. 2 Pairwise constraints on the moving, M (blue) and target, T (pink) models used for pruning the correspondence set be identified in multiple locations. The same behaviour was with distances as in Eq. 1, the pairwise terms are parametrised observed with our data. So, an additional constraint based on as consistency measures: the liver contour is proposed, which can be reliably observed on both models. The existing spectral matching framework is d (m , m ) d (t , t ) g i j g i j c(m , m , t , t ) = min , i j i j extended to incorporate the new term and robustly estimate d (t , t ) + ε d (m , m ) + ε g i j g i j a set of correspondences. (2) where ε is a small number to ensure the denominators are Optimisation not zero, c ∈[0, 1] and it quantifies how similar the geodesic distances are between the pairs (m , m ) and (t , t ). A pair In order to minimise E (C ) from Eq. 1, the shape alignment i j i j of correspondences is consistent from a geometric point of problem is formulated as graph matching [19]. Each node view if the ratio of the geodesic distances on each shape is consists of a candidate correspondence (i.e. (m, t ) ), and each closeto1[20]. However, in the presence of non-isometric edge connects two nodes (i.e. (m, t ) and (m, t ) ). Moreover, i j deformation of the data, correct correspondences might have if pair (m, t ) corresponds to (m, t ) , the pairwise constraints i j consistency values, c, lower than 1. So, the non-rigidity of imposed will quantify how consistent this association is from the data is taken into account by using the following function a geometrical point of view, thus providing weights for the for c: edges. Figure 2 highlights an example of a correct assign- ment. An affinity matrix, W , of the graph is built. The weights (c(m , m , t , t ) − 1) i j i j g(m , m , t , t ,σ ) = exp − i j i j associated with each node and edge will result in a strongly 2σ connected cluster for data with high consistency. On the other (3) hand, outlier nodes will be either weakly linked or linked in an unstructured way. In cases with a high number of out- where the parameter σ sets the amount of non-rigidity liers, large deformation or symmetry in the data, some wrong allowed for the correspondence set. Furthermore, the func- correspondences might be included in the main cluster. As a tion g also helps in separating the outliers by lowering the result, the initial correspondence set C is built by choosing for weights of highly unlikely candidate pairs. each target feature point {t }⊂ T , the closest k neighbour- M T Let B ⊂ M and B ⊂ T be the contour points on each ing descriptors on the moving surface {m }⊂ M, quickly surface. The closest contour point to each feature point x using kd trees. A spectral analysis method [19]isusedto on either M or T is computed as b = min(d (x , B)).The x g obtain the filtered correspondence set C from the matrix W . expression used for the proposed contour constraint is: In the next few paragraphs, the proposed formulation for W is detailed, which incorporates the additional term based on g (m , m , t , t ,σ ) the liver contours. b i j i j b The affinity matrix, W , should have values which are non- 1 = g(m , b , t , b ,σ ) + g(m , b , t , b ,σ ) i mi i ti b j mj j tj b negative, symmetric and increasing with higher similarity between the correspondences [19]. So, instead of working (4) 123 International Journal of Computer Assisted Radiology and Surgery (2018) 13:947–956 951 where σ allows some deformation between the candidate The intraoperative data collected during laparoscopic pairs and their corresponding contours. In practice, σ = σ surgery will most likely have some degree of sparsity— b d since they both represent variations in geodesic distances sparse point clouds [3], sparse data collection [4], sparse illustrating how restrictive the geometric pairwise constraints stereo reconstructed patches [5]. will be on the data. So, let S be a smooth interpolated surface of the intraopera- Finally, the affinity matrix, W , is built by placing the unary tive point cloud, T . The target feature points, {t }, and contour terms (similarity between descriptors sim( f (m ), f (t )))on points, B , can be expressed on the interpolated surface with i i the main diagonal and the pairwise constraints on the off- nearest neighbour computation. The geodesic distances on diagonal: S are computed using the fast marching algorithm [24,25]. This step is the most computationally expensive to compute W (i , j ) in our implementation. However, faster alternatives can be employed [26]. sim( f (m ), f (t )), i = j i i αg(m , m , t , t ,σ ) + (1 − α)g (m , m , t , t ,σ ), i = j i j i j d b i j i j b Estimating the rigid transform where α allows different weights for the importance of the two pairwise constraints. The proposed shape matching technique starts with a large set Spectral analysis on the initial formulation for E (C ) of correspondences, C, and spectral analysis prunes out the (Eq. 1) enforces high similarity between nodes (m, t ) and . The final set of correspondences is outliers, resulting in C i p (m, t ) , as well as approximately equal distances between not guaranteed to consist only of correct matches, especially them (d (m , m ) and d (t , t )). In addition, the proposed in cases with significant deformation. g i j g i j term weighs the edge connecting (m, t ) and (m, t ) higher Random sampling and consensus (RANSAC) [27](see i j if the distances d (m , b ), d (m , b ) are similar to Fig. 1) is used to get the best minimal solution {(m, t ) }∈ C g i mi g j mj i p d (t , b ), d (t , b ), respectively. As a result, the estimated out of the pruned set of correspondences C . The final pairs g i ti g j tj p correspondence set, C , is explicitly constrained to be con- {(m, t ) } are used for the least squares estimation of rotation p i sistent with the liver contours on both surfaces, M and T . and translation. The estimated transformation is considered to be a good fit if the root-mean-squared error (RMSE) Features and descriptors between the target and moving models is less than a thresh- old d and the difference between the normals is RANSAC Reliable landmarks are difficult to identify consistently less than a specific angle threshold a : dot(n , n )< normals m t between the two surfaces. The strategies used are farthest cos(a ) ∀ (n , n ) ∈ C . normals m t i p point sampling [21] and normal space sampling [22]. The former approach was chosen for a uniform distribution of the feature points on the surface. The latter aims to select Results samples such that the normals are distributed as evenly as possible, thus having fewer points in flat regions. The search Three sets of experiments were conducted to validate the space is constrained to only select features on the visible proposed method. Firstly, the robustness to the specific chal- surface of the moving mesh, M, in order to eliminate unfea- lenges present in laparoscopic interventions (partial views, sible solutions (see Fig. 1). TOLDI was chosen as a feature varying degrees of deformation) was tested on synthetic data. descriptor, because it was shown to be robust to data sampled Secondly, the proposed initial alignment method was quan- irregularly (which is the case for multiple stereo reconstruc- titatively validated in a liver phantom experiment. Finally, tion surfaces merged together), robust to varying levels of qualitative results are shown on retrospective clinical data noise and invariant to rigid transformations [23]. from a dataset acquired during a human liver resection. The same parameters are used for all our experiments Distances both on synthetic and on clinical data (σ = σ = 0.3, d b α = 0.6). The maximum number of iterations used for The geodesic distance represents the shortest distance on the RANSAC is 1000. The difference between the normals is surface between two points. If the surface changes topology set as a = 60 in order to account for some of the normals through holes or irregularities in the data, the geodesic dis- deformation. Similarly, d = 5 mm in the rigid case RANSAC tances might become unreliable. Another failure case would scenario (see “Robustness to reduced partial size” section) be observed if distant parts of the object come into contact and d = 10 mm for the remaining experiments. If RANSAC and create new shortest paths between feature points. How- no solution is found with the RMSE lower than d , RANSAC ever, it is unlikely the liver shape will change topology in the the transformation which resulted in the smallest error over initial stages of the surgery. all the iterations is used. The feature points, descriptors and 123 952 International Journal of Computer Assisted Radiology and Surgery (2018) 13:947–956 Fig. 3 Experiments on synthetic data. Top: robustness to reduced par- surface is used in the bottom experiment with increasing deformation tial size in the target model, T , bottom: robustness to increasing levels of levels. Color coding: moving model, M—blue, target model, T —pink deformation in T . The target model representing 23% of the total liver geodesic distances on the CT mesh are precomputed and –(SM + R1) The initial set, C, is pruned based on geodesic stored, in order to minimise the computation time during distances alone, following the spectral analysis technique surgery. The liver contours are currently manually delineated detailed above. RANSAC is applied on the pruned set, in a matter of seconds, and techniques to automate this step C . will be investigated in the future. –(SM + R2—ours) The proposed technique with both The proposed method was implemented in Matlab and pairwise constraints. C++, on a MacOS 10.11.2 laptop with an Intel Core i7 3.1 GHz processor. The libraries used as dependencies can Robustness to reduced partial size be found in [21,23,25,28]. The mesh processing applications MeshMixer and Meshlab [29] were used for visualisation We test the robustness of the proposed method to reduced and simulation purposes. partial views of the liver. For this experiment, there is a rigid transformation between the moving (M) and target (T ) mod- els, with all the remaining parameters fixed. Synthetic data In this experiment, the target model, T , is simulated by creating 10 partial views of decreasing sizes (from 43 to 7%) We validate the robustness of the proposed method to par- by cropping the original liver mesh M. This step was manu- tial views of the liver and to increasing deformation levels. ally performed in Meshlab. Each algorithm is run 500 times The mesh of a liver phantom (OpenCAS [8]) is used as the for each size. The mean and standard deviation of the result- moving model, M. The mean distance between the estimated ing errors are reported in Fig. 3-top. registration result and ground truth vertex correspondences is measured. Three algorithms are compared: Robustness to deformation – (R) RANSAC applied directly to the initial set of corre- We validate the robustness of the proposed algorithm to spondences, C. No pruning is applied. increasing levels of deformation in the data. A large defor- mation is applied with control points on the left lobe of the http://www.meshmixer.com. liver mesh M. Intermediate levels of deformation are gener- 123 International Journal of Computer Assisted Radiology and Surgery (2018) 13:947–956 953 Fig. 4 Phantom experiment. Our proposed global initial alignment is shown for a partial region of the deformed surface (left) and a partial sur- sufficient to allow potentially any fine alignment method to successfully face reconstruction from an intraoperative stereo laparoscopic camera converge. The TRE distribution after convergence of LM-ICP [10]is (right) Color coding: moving model, M—blue, target model, T —pink ated with vertex linear morphing between the original liver tial deformed surface is 7.94 mm after our proposed global shape and the deformed one. These steps were achieved in alignment, which is further reduced to 7.77 mm after LM- MeshMixer and Meshlab. Figure 3-bottom shows examples ICP. Similarly, in the case of the intraoperative partial surface, of the different deformation levels with 17 being the highest. the mean TRE of 28.62 mm after the global alignment is We choose the size of the partial view as 23% (Fig. 3- decreased to 12.10 mm after LM-ICP. The best case scenario top, middle shape) since this can be registered well by all for rigid registration would be point-based alignment of the algorithms tested. Visually, it represents a realistic size for a marker ball positions before and after deformation, with a laparoscopic view. mean TRE of 5.66 mm. In this experiment, the deformation level is the only vari- able. Each algorithm is run 500 times for each deformation Application in clinical data level. The mean and standard deviation are presented in Fig. 3-bottom. The proposed approach is demonstrated on clinical data from a video sequence acquired during a laparoscopic liver resec- tion. The 3D mesh of the liver surface was extracted from a Liver phantom CT scan before surgery. We use the SmartLiver system [5] to process the retrospective data. The liver is automatically The proposed method is validated using the OpenCAS [8] segmented in the laparoscopic video with a deep learning public dataset, which contains 3D meshes from an experi- framework [30]. Surface patches are collected to cover all ment in which a silicone liver phantom is deformed by an the visible surface in each video [31]. They are consequently indentation. The positions of small Teflon marker balls in merged together using optical tracking data. both the initial and deformed states of the phantom are given. Figure 5 shows the visual assessment between the manual Please refer to Suwelack et al. [8] for more details about how alignment performed on the SmartLiver GUI and the pro- the dataset was built. posed method. The last column illustrates an example of The 3D model of the liver phantom in its initial state is augmented reality in laparoscopic liver surgery after LM- used as the moving model, M. The proposed coarse regis- ICP is applied to the proposed alignment. tration method is tested in two scenarios. Firstly, a partial view of the deformed liver phantom is used as the target model, T (Fig. 4-left), which tests the performance under Discussion deformation, partial and sparse data. Secondly, a partial sur- face reconstructed from an intraoperative stereo endoscopic The results from the first experiment show that when the camera (Fig. 4-right) is used as T . On top of deformation intraoperative surface is large enough, all three methods and partial data, this scenario also tests the proposed method have comparable results. However, having surfaces with size in realistic noise levels from a stereo reconstruction. After smaller than 23% of the whole mesh becomes challenging the global alignment is estimated with the proposed method, for both R and SM + R1. From what we noticed in our Levenberg–Marquardt iterative closest point (LM-ICP)[10] is applied. The distribution of target registration error (TRE) in mm is computed for both cases. The mean TRE for the par- www.visiblepatient.com. 123 954 International Journal of Computer Assisted Radiology and Surgery (2018) 13:947–956 Fig. 5 Global alignment on clinical data from a dataset acquired during ing LM-ICP on the proposed alignment. Color coding Overlay: liver a laparoscopic liver resection. Color coding Alignment: moving model, tumour—green, vessels—purple, liver contour—yellow M—blue, target model, T —pink. The overlay is computed after apply- datasets, such surfaces are characteristic for videos acquired to find three good pairs of correspondences from small sets on the left lobe in laparoscopic interventions with restricted (approximately 10 correspondences, depending on the data). camera movement. Note how the proposed method has less A quantitative evaluation of the proposed algorithm is per- variance in the solutions even for smaller partial shapes. The formed on a phantom dataset with partial size, deformation additional pairwise term which incorporates the boundaries and realistic noise levels. The partial surface used in Fig. 4- of both M and T makes the problem less ambiguous, as right is illustrative for an intraoperative scenario, since the opposed to just using the geodesic distances between pairs data are collected using a stereo laparoscope. The proposed of correspondences. method succeeds in providing a good initial alignment, and The second experiment illustrates robustness to increas- it is shown that further fine alignment methods (such as LM- ing deformation levels in the partial views. Similarly to the ICP) can successfully converge towards the correct location. previous experiment, the proposed method is more consis- The current errors are comparable to the literature in the rigid tent across different deformations, with less variation in the case scenario [5–7]. Since most fine registration algorithms solutions it provides. This is mostly due to the fact that the set can converge successfully if the coarse alignment is within a of correspondences, C , obtained from the proposed method fewcm[9], the proposed method achieves results within the contains fewer outliers than the other methods. desired range. In order to decrease the errors further, we will These methods are compared with RANSAC because it investigate non-rigid refinement methods in the future. is a popular algorithm for finding correspondences between We show promising results on a retrospective video point sets related by parametric transformations. If RANSAC acquisition from a laparoscopic liver surgery. The proposed is applied directly on the set of correspondences C,itis method is compared against a manual alignment performed unable to obtain a good alignment. This is mostly due to on the SmartLiver GUI. Qualitative results are provided to the fact that the initial set will contain a high number of out- illustrate that the proposed method manages to correctly liers, due to the low descriptiveness of the data. Moreover, identify the liver region in a challenging environment with for partial shape sizes characteristic to laparoscopic surgeries realistic noise levels, significant deformation and small par- (less than 23% in Fig. 3), there are multiple locations on the tial views. Note that the proposed method aims to estimate a liver which result in a good fit. However, by allowing for coarse surface alignment, which can then be further refined deformation in the proposed pruning technique, a set of cor- with a local algorithm. For example, Fig. 5-right shows an respondences, C , is obtained in the correct region of interest, overlay computed by applying LM-ICP on the coarse align- which can be further refined by RANSAC. This approach ment estimated by our method. Furthermore, the current is also more computationally efficient since RANSAC has computational time required to compute the initial alignment 123 International Journal of Computer Assisted Radiology and Surgery (2018) 13:947–956 955 between surfaces is approximately 20 s with non-optimised Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecomm code, which makes it feasible for clinical usage. ons.org/licenses/by/4.0/), which permits unrestricted use, distribution, Our experiments were all performed using the same choice and reproduction in any medium, provided you give appropriate credit of parameters, suggesting the proposed method is not very to the original author(s) and the source, provide a link to the Creative sensitive to variations. However, in the future we would like Commons license, and indicate if changes were made. to investigate their influence. Although the experiments pre- sented here show promising results, we would like to validate the proposed technique on clinical data from more patients References to test its robustness with respect to liver surface variations. Furthermore, we are looking into ways to automate the liver 1. Bartoli A, Collins T, Bourdel N, Canis M (2012) Computer assisted contour selection, such that no user interaction is needed and minimally invasive surgery: Is medical computer vision the answer the whole process is fully automated. to improving laparosurgery? J Med Hypotheses 79(6):858–863 2. Hughes-Hallett A, Pratt P, Dilley J, Vale J, Darzi A, Mayer E (2015) Augmented reality: 3d image-guided surgery. Cancer Imag- ing 15(1):O8 3. Plantefève R, Peterlik I, Haouchine N, Cotin S (2016) Patient- specific biomechanical modeling for guidance during minimally- Conclusion invasive hepatic surgery. Ann Biomed Eng 44(1):139–153 4. Rucker DC, Wu Y, Clements LW, Ondrake JE, Pheiffer TS, We propose a fast and global method for surface-based reg- Simpson AL, Jarnagin WR, Miga MI (2014) A mechanics-based istration of a 3D liver mesh extracted from a preoperative nonrigid registration method for liver surgery using sparse intraop- erative data. IEEE Trans Med Imaging 33(1):147–158 CT scan of the liver and the surface reconstruction of the 5. Thompson S, Totz J, Song Y, Johnsen S, Stoyanov D, Ourselin S, intraoperative laparoscopic video feed. We have validated Gurusamy K, Scheider C, Davidson B, Hawkes D, Clarkson MJ its performance with respect to the challenges characteris- (2015) Accuracy validation of an image guided laparoscopy system tic to laparoscopic surfaces on synthetic data, on a phantom for liver resection. In: SPIE, vol 9415, p 941509 6. Clements LW, Chapman WC, Dawant BM, Galloway RL, Miga MI dataset and on retrospective clinical data. We conclude that (2008) Robust surface registration using salient anatomical features the proposed method could potentially be used as an auto- for image-guided liver surgery: algorithm and validation. J Med matic way of obtaining a good initial alignment between the Phys 35(6):2528 two surfaces, given the required features. Moreover, it does 7. dos Santos TR, Seitel A, Kilgus T, Suwelack S, Wekerle AL, Kenngott H, Speidel S, Schlemmer HP, Meinzer HP, Heimann T, not require any advanced hardware, which makes it accessi- Maier-Hein L (2014) Pose-independent surface matching for intra- ble and comparatively easy to translate to a clinical setting. operative soft-tissue marker-less registration. J Med Image Anal 18(7):1101–1114 Acknowledgements This work is supported by the CRUK (http:// 8. Suwelack S, Röhl S, Bodenstedt S, Reichard D, Dillmann R, dos www.cancerresearchuk.org) and EPSRC Comprehensive Cancer Imag- Santos T, Maier-Hein L, Wagner M, Wünscher J, Kenngott H, ing Centre at KCL and UCL, and EPSRC CDT in Medical Imag- Müller BP, Speidel S (2014) Physics-based shape matching for ing [EP/L016478/1]. DH, MJC, DS were supported by the Well- intraoperative image guidance. J Med Phys 41(11):111901 come/EPSRC [203145Z/16/Z]. DS receives funding from EPSRC 9. Maier-Hein L, Franz AM, dos Santos TR, Schmidt M, Fangerau [EP/P012841/1]. BD was supported by the NIHR Biomedical Research M, Meinzer HP, Fitzpatrick JM (2012) Convergent iterative Centre at University College London Hospitals NHS Foundations Trust closest-point algorithm to accomodate anisotropic and inhomoge- and University College London. The imaging data used for this work nous localization error. IEEE Trans Pattern Anal Mach Intell were obtained with funding from the Health Innovation Challenge Fund 34(8):1520–1532 [HICF-T4-317], a parallel funding partnership between the Wellcome 10. Fitzgibbon AW (2003) Robust registration of 2d and 3d point sets. Trust and the Department of Health. The views expressed in this publica- J Image Vis Comput 21(13):1145–1153 tion are those of the author(s) and not necessarily those of the Wellcome 11. Cash DM, Miga MI, Glasgow SC, Dawant BM, Clements LW, Cao Trust or the Department of Health. Z, Galloway RL, Chapman WC (2007) Concepts and preliminary data toward the realization of image-guided liver surgery. J Gas- trointest Surg 11(7):844–859 Compliance with ethical standards 12. Nicolau SA, Pennec X, Soler L, Buy X, Gangi A, Ayache N, Marescaux J (2009) An augmented reality system for liver ther- Conflict of interest The authors declare that they have no conflict of mal ablation: design and evaluation on clinical cases. J Med Image interest. Anal 13(3):494–506 13. Teber D, Guven S, Simpfendörfer T, Baumhauer M, Güven EO, Human and animal rights All procedures performed in studies involv- Yencilek F, Gözen AS, Rassweiler J (2009) Augmented reality: a ing human participants were in accordance with the ethical standards of new tool to improve surgical accuracy during laparoscopic partial the institutional and/or national research committee and with the 1964 nephrectomy? Preliminary in vitro and in vivo results. Eur Urol Helsinki declaration and its later amendments or comparable ethical 56(2):332–338 standards. 14. Wild E, Teber D, Schmid D, Simpfendörfer T, Müller M, Baranski AC, Kenngott H, Kopka K, Maier-Hein L (2016) Robust augmented reality guidance with fluorescent markers in laparoscopic surgery. Informed consent Informed consent was obtained from all individual IJCARS 11(6):899–907 participants included in the study. 123 956 International Journal of Computer Assisted Radiology and Surgery (2018) 13:947–956 15. Mountney P, Fallert J, Nicolau S, Soler L, Mewes PW (2014) 25. Jacobson A (2015) gptoolbox: geometry processing toolbox. An augmented reality framework for soft tissue surgery. MICCAI https://github.com/alecjacobson/gptoolbox 17:423–431 26. Crane K, Weischedel C, Wardetzky M (2013) Geodesics in heat: 16. Feuerstein M, Mussack T, Heining SM, Navab N (2007) a new approach to computing distance based on heat flow. ACM Registration-free laparoscope augmentation for intra-operative Trans Gr 32(5):152:1–152:11 liver resection planning. In: Medical imaging 2007 27. Fischler MA, Bolles RC (1981) Random sample consensus: 17. Fusaglia M, Hess H, Schwalbe M, Peterhans M, Tinguely P, Weber paradigm for model fitting with applications to image analysis and S, Lu H (2016) A clinically applicable laser-based image-guided automated cartography. Commun ACM 24(6):381–395 system for laparoscopic liver procedures. IJCARS 11(8):1499– 28. D’Errico J (2016) Surface fitting using gridfit. https://uk. 1513 mathworks.com/matlabcentral/fileexchange/8998-surface-fitting- 18. Raviv D, Dubrovina A, Kimmel R (2012) Hierarchical matching using-gridfit of non-rigid shapes. In: SSVM, pp 604–615 29. Cignoni P, Callieri M, Corsini M, Dellepiane M, Ganovelli F, 19. Leordeanu M, Hebert M (2005) A spectral technique for correspon- Ranzuglia G (2008) MeshLab: an open-source mesh processing dence problems using pairwise constraints. ICCV 2:1482–1489 tool. In: Eurographics Italian chapter conference 20. Huang QX, Adams B, Wicke M, Guibas LJ (2008) Non-rigid 30. Gibson ED, Robu MR, Thompson SA, Edwards PE, Schneider registration under isometric deformations. Comput Gr Forum C, Gurusamy K, Davidson BR, Hawkes DJ, Barrat DC, Clarkson 27(5):1449–1457 MJ (2017) Deep residual networks for automatic segmentation of 21. Peyré G (2011) The numerical tours of signal processing. Comput laparoscopic videos of the liver. In: SPIE, vol 10135, pp 101351M– Sci Eng 13(4):94–97 101351M–6 22. Gelfand N, Ikemoto L, Rusinkiewicz S, Levoy M (2003) Geometri- 31. Totz J, Thompson S, Stoyanov D, Gurusamy K, Davidson B, cally stable sampling for the ICP algorithm. In: 3DIM, pp 260–267 Hawkes DJ, Clarkson MJ (2014) Fast semi-dense surface recon- 23. Yang J, Zhang Q, Xiao Y, Cao Z (2017) Toldi: an effective and struction from stereoscopic video in laparoscopic surgery. IPCAI robust approach for 3d local shape description. Pattern Recognit 8498:206–215 65:175–187 24. Sethian JA (1996) A fast marching level set method for monoton- ically advancing fronts. PNAS 93(4):1591–1595 http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png International Journal of Computer Assisted Radiology and Surgery Springer Journals

Global rigid registration of CT to video in laparoscopic liver surgery

Download PDF
10 pages

Loading next page...
 
/lp/springer_journal/global-rigid-registration-of-ct-to-video-in-laparoscopic-liver-surgery-RKQPtj2y3P

References (32)

Publisher
Springer Journals
Copyright
Copyright © 2018 by The Author(s)
Subject
Medicine & Public Health; Imaging / Radiology; Surgery; Health Informatics; Computer Imaging, Vision, Pattern Recognition and Graphics; Computer Science, general
ISSN
1861-6410
eISSN
1861-6429
DOI
10.1007/s11548-018-1781-z
pmid
29736801
Publisher site
See Article on Publisher Site

Abstract

Purpose Image-guidance systems have the potential to aid in laparoscopic interventions by providing sub-surface structure information and tumour localisation. The registration of a preoperative 3D image with the intraoperative laparoscopic video feed is an important component of image guidance, which should be fast, robust and cause minimal disruption to the surgical procedure. Most methods for rigid and non-rigid registration require a good initial alignment. However, in most research systems for abdominal surgery, the user has to manually rotate and translate the models, which is usually difficult to perform quickly and intuitively. Methods We propose a fast, global method for the initial rigid alignment between a 3D mesh derived from a preoperative CT of the liver and a surface reconstruction of the intraoperative scene. We formulate the shape matching problem as a quadratic assignment problem which minimises the dissimilarity between feature descriptors while enforcing geometrical consistency between all the feature points. We incorporate a novel constraint based on the liver contours which deals specifically with the challenges introduced by laparoscopic data. Results We validate our proposed method on synthetic data, on a liver phantom and on retrospective clinical data acquired during a laparoscopic liver resection. We show robustness over reduced partial size and increasing levels of deformation. Our results on the phantom and on the real data show good initial alignment, which can successfully converge to the correct position using fine alignment techniques. Furthermore, since we can pre-process the CT scan before surgery, the proposed method runs faster than current algorithms. Conclusion The proposed shape matching method can provide a fast, global initial registration, which can be further refined by fine alignment methods. This approach will lead to a more usable and intuitive image-guidance system for laparoscopic liver surgery. Keywords Image guidance · Laparoscopic liver surgery · Global registration · Shape matching · Surface descriptors · Computer-assisted surgery Introduction tion, constrained vantage point, limited field of view, poor haptic feedback and occluded anatomy [1]. Image guidance Minimally invasive surgery offers the patient major bene- aims to assist the clinicians in localising and tracking sub- fits over open surgery, including less trauma, less pain and surface structures such as abnormalities or major vessel trees. shorter hospital stays. However, these interventions present Thus, these systems have the potential to aid in surgical inter- several challenges for clinicians, such as weak depth percep- ventions through improved resection quality and a reduction in positive surgical margins [2]. The safety margin around a B Maria R. Robu possible tumour in current laparoscopic procedures is a min- [email protected] imum of 10 mm [3], so it is considered desirable to develop systems with accuracy below 5 mm on average [4,5]. Current Wellcome/EPSRC Centre for Interventional and Surgical Sciences, University College London, London, UK rigid registration methods achieve accuracies of approxi- mately 10 mm in phantom experiments [5–7]. Improving the Centre For Medical Image Computing, University College London, London, UK robustness, accessibility and reliability of image-guidance Division of Surgery and Interventional Science, University College London, London, UK 123 948 International Journal of Computer Assisted Radiology and Surgery (2018) 13:947–956 systems could potentially increase the number of patients of an additional CT or MRI scan immediately before the benefiting from minimally invasive surgeries. intervention. Most hospitals require an abdominal CT scan to be Surface acquisition of the intraoperative scene has been acquired before surgery for laparoscopic liver interventions. proposed as an alternative. Several strategies have been A 3D model of the liver, major vessel trees and any abnormal- developed using laser range scanners [6], optically tracked ities can be segmented from the CT scan. The registration of probes [11], time-of-flight (TOF) data [7] and stereo recon- the preoperative liver model and the intraoperative laparo- struction [5]. Once the surface is acquired, the clinician is scopic images is an essential step towards developing an required to delineate salient anatomical features leading to image-guidance system. Most methods in the literature can a point-based initial alignment [6,11] or to a more com- be divided between coarse alignment, defined as a global plex non-rigid optimisation framework [3]. Another option alignment which can match the surfaces irrespective of their to obtain the rigid alignment is to manually rotate and trans- initial transformation, and fine alignment techniques—in late the 3D preoperative image until it fits the intraoperative which a good initial alignment is already provided as a data [5]. While some level of user interaction is needed for starting point. In this paper, we focus on coarse alignment these approaches, it is generally more intuitive and faster to methods for surface-based registration. select salient features than to manipulate the six degrees of Furthermore, most methods are only applicable in open freedom associated with a rigid transform. surgery [4,6–8] as a large surface of the intraoperative scene Hybrid methods have been proposed using cone beam is required. However, surfaces acquired laparoscopically CT and fluoroscopy [15] as bridging modalities between the present the additional challenges that the camera has access to laparoscopic camera and the preoperative CT, which deliv- a restricted region of the abdomen leading to an even smaller ers an additional radiation dose to the patient. Feuerstein partial view, lack of reliable landmarks and significant defor- et al. [16] propose using intraoperative cone beam CT and mation from pneumoperitoneum [7]. We address and discuss optical tracking to register directly to the laparoscopic view the challenges inherent to laparoscopic surgeries which moti- without using preoperative information. While their methods vated our method. achieve promising results, they are based on advanced hard- We propose a fast, semi-automatic global alignment ware which might not be available in most clinical settings. method which achieves the initial alignment between the Finally, fully automated techniques have been proposed preoperative CT model of the liver surface and a surface in [7,17]. Fusaglia et al. [17] developed an exhaustive search reconstruction of the intraoperative scene. The resulting over the principal directions of the intraoperative surface, transformation could be further improved by fine alignment which is acquired using a laparoscopic laser pointer. While algorithms [9,10] in order to get a rigid [5,6,11] or a non-rigid their proposed approach is promising, it still introduces addi- alignment [3] between the two modalities. Our approach can tional tools into the clinical workflow. Dos Santos et al. [7] lead to a faster and more intuitive use of image-guidance introduced a novel automatic method to establish surface systems in laparoscopic surgeries. We show robustness to correspondences between the 3D preoperative mesh and the reduced partial sizes and increasing deformations in the intra- intraoperative surface acquired with a TOF camera in open operative model on synthetic data. Moreover, we evaluate the liver surgery. Their approach was validated on a phantom proposed method on a liver phantom and on retrospective data of the human liver and on an ex-vivo porcine liver with from a dataset acquired in a laparoscopic liver resection with accuracy better than 1 cm and computation time ranging promising results. from one minute to 5.5 h. While their phantom validation under deformation from breathing motion can be sufficient for open surgery, livers in laparoscopic interventions undergo Background significant general deformation due to pneumoperitoneum. The initial rigid registration of the preoperative 3D image and Furthermore, it is unclear how both methods [7,17] would the intraoperative scene has been explored through methods be translated to laparoscopic interventions since they rely on that rely on fiducials, user interaction and through fully auto- large surfaces of the liver being visible. mated methods. While promising results have been achieved in the liter- Several approaches propose the use of fiducials, either ature, we aim to develop an image-guidance system which on the patient skin [12] for needle guidance, or on the can handle the challenges of laparoscopic interventions and organ itself [13] for tracking in laparoscopic partial nephrec- is easy to integrate with the current clinical protocol without additional hardware or advanced cameras. Furthermore, the tomy. Another more robust option, which is applicable in laparoscopic interventions, would be to attach metabolisable system should be usable during surgical interventions, with minimal disruption and fast computation. fluorescent markers on the organ [14]. Such fiducials can be seen in both modalities. However, these strategies are disrup- tive to the clinical workflow since they require the acquisition 123 International Journal of Computer Assisted Radiology and Surgery (2018) 13:947–956 949 Fig. 1 An overview of our proposed global alignment framework, showing the preoperative and intraoperative steps Contributions Let M be the moving (preoperative CT mesh) model, and let T be the target (intraoperative point cloud) model. We propose a fast, semi-automatic method to obtain a global Sets of features, {m }⊂ M and {t }⊂ T , are selected on r s initial alignment between a 3D liver model extracted from both surfaces with f (·) as their corresponding descriptor. Let the preoperative CT scan and a surface reconstruction of the d (x , y) be the geodesic distance between any two points, x intraoperative scene. and y, on a surface. An existing formulation of shape matching is extended to Generally, it is difficult to match surfaces in laparoscopic incorporate an additional constraint based on the contours of liver surgery only based on descriptors since the surfaces the organ (the ridge line—see Fig. 1), which can be identified lack prominent, uniquely identifiable features. The use of on both surfaces with high confidence. Once the delineation geometric consistency between the correspondences on both of the liver ridge line is given in the two modalities, no further shapes can further constrain the registration problem. user interaction or initialisation is required for the alignment Shape matching can be formulated as a quadratic assign- stage. The proposed method is able to robustly estimate a ment problem (QAP): correspondence set between the two surfaces under defor- mation, sparse data, partial views and realistic noise levels. E (C ) = d( f (m ), f (t )) i i We validate our technique in a simulated environment to (m,t ) ∈C show robustness to partial data and deformation. Moreover, + (d (m , m ) − d (t , t )) (1) g i j g i j we provide quantitative results obtained on a liver phantom (m,t ) ∈C (m,t ) ∈C i j and qualitative results on retrospective data from a laparo- scopic liver resection to illustrate feasibility in a realistic where C ={(m, t ) }⊂ M T is the initial correspondence i × clinical setting. set composed of candidate pairs of feature points from the two surfaces, d( f (·), f (·)) is the distance between the feature descriptors and d (·, ·) is the geodesic distance between two correspondences on the same surface. This energy function Methods aims to output a set of correspondences for which the dissim- ilarity between the descriptors is minimised and the geodesic Figure 1 illustrates the main steps of the proposed pipeline. distances between pairs of correspondences are maintained. The input data include the segmentation of the 3D mesh from While this approach works well in the vision literature for the CT scan, a surface reconstruction represented as a point complex shapes [18], the intraoperative surfaces pose several cloud of the intraoperative data and the segmented contours challenges. It has been previously discussed in [7] that con- on both surfaces. The liver contour is defined as the ridge straining the correspondence set based only on the geodesic line visible in yellow in Fig. 1 on both the preoperative and distances between them is still ambiguous for almost flat sur- the intraoperative surfaces. faces, in which the same spatial configuration of features can 123 950 International Journal of Computer Assisted Radiology and Surgery (2018) 13:947–956 Fig. 2 Pairwise constraints on the moving, M (blue) and target, T (pink) models used for pruning the correspondence set be identified in multiple locations. The same behaviour was with distances as in Eq. 1, the pairwise terms are parametrised observed with our data. So, an additional constraint based on as consistency measures: the liver contour is proposed, which can be reliably observed on both models. The existing spectral matching framework is d (m , m ) d (t , t ) g i j g i j c(m , m , t , t ) = min , i j i j extended to incorporate the new term and robustly estimate d (t , t ) + ε d (m , m ) + ε g i j g i j a set of correspondences. (2) where ε is a small number to ensure the denominators are Optimisation not zero, c ∈[0, 1] and it quantifies how similar the geodesic distances are between the pairs (m , m ) and (t , t ). A pair In order to minimise E (C ) from Eq. 1, the shape alignment i j i j of correspondences is consistent from a geometric point of problem is formulated as graph matching [19]. Each node view if the ratio of the geodesic distances on each shape is consists of a candidate correspondence (i.e. (m, t ) ), and each closeto1[20]. However, in the presence of non-isometric edge connects two nodes (i.e. (m, t ) and (m, t ) ). Moreover, i j deformation of the data, correct correspondences might have if pair (m, t ) corresponds to (m, t ) , the pairwise constraints i j consistency values, c, lower than 1. So, the non-rigidity of imposed will quantify how consistent this association is from the data is taken into account by using the following function a geometrical point of view, thus providing weights for the for c: edges. Figure 2 highlights an example of a correct assign- ment. An affinity matrix, W , of the graph is built. The weights (c(m , m , t , t ) − 1) i j i j g(m , m , t , t ,σ ) = exp − i j i j associated with each node and edge will result in a strongly 2σ connected cluster for data with high consistency. On the other (3) hand, outlier nodes will be either weakly linked or linked in an unstructured way. In cases with a high number of out- where the parameter σ sets the amount of non-rigidity liers, large deformation or symmetry in the data, some wrong allowed for the correspondence set. Furthermore, the func- correspondences might be included in the main cluster. As a tion g also helps in separating the outliers by lowering the result, the initial correspondence set C is built by choosing for weights of highly unlikely candidate pairs. each target feature point {t }⊂ T , the closest k neighbour- M T Let B ⊂ M and B ⊂ T be the contour points on each ing descriptors on the moving surface {m }⊂ M, quickly surface. The closest contour point to each feature point x using kd trees. A spectral analysis method [19]isusedto on either M or T is computed as b = min(d (x , B)).The x g obtain the filtered correspondence set C from the matrix W . expression used for the proposed contour constraint is: In the next few paragraphs, the proposed formulation for W is detailed, which incorporates the additional term based on g (m , m , t , t ,σ ) the liver contours. b i j i j b The affinity matrix, W , should have values which are non- 1 = g(m , b , t , b ,σ ) + g(m , b , t , b ,σ ) i mi i ti b j mj j tj b negative, symmetric and increasing with higher similarity between the correspondences [19]. So, instead of working (4) 123 International Journal of Computer Assisted Radiology and Surgery (2018) 13:947–956 951 where σ allows some deformation between the candidate The intraoperative data collected during laparoscopic pairs and their corresponding contours. In practice, σ = σ surgery will most likely have some degree of sparsity— b d since they both represent variations in geodesic distances sparse point clouds [3], sparse data collection [4], sparse illustrating how restrictive the geometric pairwise constraints stereo reconstructed patches [5]. will be on the data. So, let S be a smooth interpolated surface of the intraopera- Finally, the affinity matrix, W , is built by placing the unary tive point cloud, T . The target feature points, {t }, and contour terms (similarity between descriptors sim( f (m ), f (t )))on points, B , can be expressed on the interpolated surface with i i the main diagonal and the pairwise constraints on the off- nearest neighbour computation. The geodesic distances on diagonal: S are computed using the fast marching algorithm [24,25]. This step is the most computationally expensive to compute W (i , j ) in our implementation. However, faster alternatives can be employed [26]. sim( f (m ), f (t )), i = j i i αg(m , m , t , t ,σ ) + (1 − α)g (m , m , t , t ,σ ), i = j i j i j d b i j i j b Estimating the rigid transform where α allows different weights for the importance of the two pairwise constraints. The proposed shape matching technique starts with a large set Spectral analysis on the initial formulation for E (C ) of correspondences, C, and spectral analysis prunes out the (Eq. 1) enforces high similarity between nodes (m, t ) and . The final set of correspondences is outliers, resulting in C i p (m, t ) , as well as approximately equal distances between not guaranteed to consist only of correct matches, especially them (d (m , m ) and d (t , t )). In addition, the proposed in cases with significant deformation. g i j g i j term weighs the edge connecting (m, t ) and (m, t ) higher Random sampling and consensus (RANSAC) [27](see i j if the distances d (m , b ), d (m , b ) are similar to Fig. 1) is used to get the best minimal solution {(m, t ) }∈ C g i mi g j mj i p d (t , b ), d (t , b ), respectively. As a result, the estimated out of the pruned set of correspondences C . The final pairs g i ti g j tj p correspondence set, C , is explicitly constrained to be con- {(m, t ) } are used for the least squares estimation of rotation p i sistent with the liver contours on both surfaces, M and T . and translation. The estimated transformation is considered to be a good fit if the root-mean-squared error (RMSE) Features and descriptors between the target and moving models is less than a thresh- old d and the difference between the normals is RANSAC Reliable landmarks are difficult to identify consistently less than a specific angle threshold a : dot(n , n )< normals m t between the two surfaces. The strategies used are farthest cos(a ) ∀ (n , n ) ∈ C . normals m t i p point sampling [21] and normal space sampling [22]. The former approach was chosen for a uniform distribution of the feature points on the surface. The latter aims to select Results samples such that the normals are distributed as evenly as possible, thus having fewer points in flat regions. The search Three sets of experiments were conducted to validate the space is constrained to only select features on the visible proposed method. Firstly, the robustness to the specific chal- surface of the moving mesh, M, in order to eliminate unfea- lenges present in laparoscopic interventions (partial views, sible solutions (see Fig. 1). TOLDI was chosen as a feature varying degrees of deformation) was tested on synthetic data. descriptor, because it was shown to be robust to data sampled Secondly, the proposed initial alignment method was quan- irregularly (which is the case for multiple stereo reconstruc- titatively validated in a liver phantom experiment. Finally, tion surfaces merged together), robust to varying levels of qualitative results are shown on retrospective clinical data noise and invariant to rigid transformations [23]. from a dataset acquired during a human liver resection. The same parameters are used for all our experiments Distances both on synthetic and on clinical data (σ = σ = 0.3, d b α = 0.6). The maximum number of iterations used for The geodesic distance represents the shortest distance on the RANSAC is 1000. The difference between the normals is surface between two points. If the surface changes topology set as a = 60 in order to account for some of the normals through holes or irregularities in the data, the geodesic dis- deformation. Similarly, d = 5 mm in the rigid case RANSAC tances might become unreliable. Another failure case would scenario (see “Robustness to reduced partial size” section) be observed if distant parts of the object come into contact and d = 10 mm for the remaining experiments. If RANSAC and create new shortest paths between feature points. How- no solution is found with the RMSE lower than d , RANSAC ever, it is unlikely the liver shape will change topology in the the transformation which resulted in the smallest error over initial stages of the surgery. all the iterations is used. The feature points, descriptors and 123 952 International Journal of Computer Assisted Radiology and Surgery (2018) 13:947–956 Fig. 3 Experiments on synthetic data. Top: robustness to reduced par- surface is used in the bottom experiment with increasing deformation tial size in the target model, T , bottom: robustness to increasing levels of levels. Color coding: moving model, M—blue, target model, T —pink deformation in T . The target model representing 23% of the total liver geodesic distances on the CT mesh are precomputed and –(SM + R1) The initial set, C, is pruned based on geodesic stored, in order to minimise the computation time during distances alone, following the spectral analysis technique surgery. The liver contours are currently manually delineated detailed above. RANSAC is applied on the pruned set, in a matter of seconds, and techniques to automate this step C . will be investigated in the future. –(SM + R2—ours) The proposed technique with both The proposed method was implemented in Matlab and pairwise constraints. C++, on a MacOS 10.11.2 laptop with an Intel Core i7 3.1 GHz processor. The libraries used as dependencies can Robustness to reduced partial size be found in [21,23,25,28]. The mesh processing applications MeshMixer and Meshlab [29] were used for visualisation We test the robustness of the proposed method to reduced and simulation purposes. partial views of the liver. For this experiment, there is a rigid transformation between the moving (M) and target (T ) mod- els, with all the remaining parameters fixed. Synthetic data In this experiment, the target model, T , is simulated by creating 10 partial views of decreasing sizes (from 43 to 7%) We validate the robustness of the proposed method to par- by cropping the original liver mesh M. This step was manu- tial views of the liver and to increasing deformation levels. ally performed in Meshlab. Each algorithm is run 500 times The mesh of a liver phantom (OpenCAS [8]) is used as the for each size. The mean and standard deviation of the result- moving model, M. The mean distance between the estimated ing errors are reported in Fig. 3-top. registration result and ground truth vertex correspondences is measured. Three algorithms are compared: Robustness to deformation – (R) RANSAC applied directly to the initial set of corre- We validate the robustness of the proposed algorithm to spondences, C. No pruning is applied. increasing levels of deformation in the data. A large defor- mation is applied with control points on the left lobe of the http://www.meshmixer.com. liver mesh M. Intermediate levels of deformation are gener- 123 International Journal of Computer Assisted Radiology and Surgery (2018) 13:947–956 953 Fig. 4 Phantom experiment. Our proposed global initial alignment is shown for a partial region of the deformed surface (left) and a partial sur- sufficient to allow potentially any fine alignment method to successfully face reconstruction from an intraoperative stereo laparoscopic camera converge. The TRE distribution after convergence of LM-ICP [10]is (right) Color coding: moving model, M—blue, target model, T —pink ated with vertex linear morphing between the original liver tial deformed surface is 7.94 mm after our proposed global shape and the deformed one. These steps were achieved in alignment, which is further reduced to 7.77 mm after LM- MeshMixer and Meshlab. Figure 3-bottom shows examples ICP. Similarly, in the case of the intraoperative partial surface, of the different deformation levels with 17 being the highest. the mean TRE of 28.62 mm after the global alignment is We choose the size of the partial view as 23% (Fig. 3- decreased to 12.10 mm after LM-ICP. The best case scenario top, middle shape) since this can be registered well by all for rigid registration would be point-based alignment of the algorithms tested. Visually, it represents a realistic size for a marker ball positions before and after deformation, with a laparoscopic view. mean TRE of 5.66 mm. In this experiment, the deformation level is the only vari- able. Each algorithm is run 500 times for each deformation Application in clinical data level. The mean and standard deviation are presented in Fig. 3-bottom. The proposed approach is demonstrated on clinical data from a video sequence acquired during a laparoscopic liver resec- tion. The 3D mesh of the liver surface was extracted from a Liver phantom CT scan before surgery. We use the SmartLiver system [5] to process the retrospective data. The liver is automatically The proposed method is validated using the OpenCAS [8] segmented in the laparoscopic video with a deep learning public dataset, which contains 3D meshes from an experi- framework [30]. Surface patches are collected to cover all ment in which a silicone liver phantom is deformed by an the visible surface in each video [31]. They are consequently indentation. The positions of small Teflon marker balls in merged together using optical tracking data. both the initial and deformed states of the phantom are given. Figure 5 shows the visual assessment between the manual Please refer to Suwelack et al. [8] for more details about how alignment performed on the SmartLiver GUI and the pro- the dataset was built. posed method. The last column illustrates an example of The 3D model of the liver phantom in its initial state is augmented reality in laparoscopic liver surgery after LM- used as the moving model, M. The proposed coarse regis- ICP is applied to the proposed alignment. tration method is tested in two scenarios. Firstly, a partial view of the deformed liver phantom is used as the target model, T (Fig. 4-left), which tests the performance under Discussion deformation, partial and sparse data. Secondly, a partial sur- face reconstructed from an intraoperative stereo endoscopic The results from the first experiment show that when the camera (Fig. 4-right) is used as T . On top of deformation intraoperative surface is large enough, all three methods and partial data, this scenario also tests the proposed method have comparable results. However, having surfaces with size in realistic noise levels from a stereo reconstruction. After smaller than 23% of the whole mesh becomes challenging the global alignment is estimated with the proposed method, for both R and SM + R1. From what we noticed in our Levenberg–Marquardt iterative closest point (LM-ICP)[10] is applied. The distribution of target registration error (TRE) in mm is computed for both cases. The mean TRE for the par- www.visiblepatient.com. 123 954 International Journal of Computer Assisted Radiology and Surgery (2018) 13:947–956 Fig. 5 Global alignment on clinical data from a dataset acquired during ing LM-ICP on the proposed alignment. Color coding Overlay: liver a laparoscopic liver resection. Color coding Alignment: moving model, tumour—green, vessels—purple, liver contour—yellow M—blue, target model, T —pink. The overlay is computed after apply- datasets, such surfaces are characteristic for videos acquired to find three good pairs of correspondences from small sets on the left lobe in laparoscopic interventions with restricted (approximately 10 correspondences, depending on the data). camera movement. Note how the proposed method has less A quantitative evaluation of the proposed algorithm is per- variance in the solutions even for smaller partial shapes. The formed on a phantom dataset with partial size, deformation additional pairwise term which incorporates the boundaries and realistic noise levels. The partial surface used in Fig. 4- of both M and T makes the problem less ambiguous, as right is illustrative for an intraoperative scenario, since the opposed to just using the geodesic distances between pairs data are collected using a stereo laparoscope. The proposed of correspondences. method succeeds in providing a good initial alignment, and The second experiment illustrates robustness to increas- it is shown that further fine alignment methods (such as LM- ing deformation levels in the partial views. Similarly to the ICP) can successfully converge towards the correct location. previous experiment, the proposed method is more consis- The current errors are comparable to the literature in the rigid tent across different deformations, with less variation in the case scenario [5–7]. Since most fine registration algorithms solutions it provides. This is mostly due to the fact that the set can converge successfully if the coarse alignment is within a of correspondences, C , obtained from the proposed method fewcm[9], the proposed method achieves results within the contains fewer outliers than the other methods. desired range. In order to decrease the errors further, we will These methods are compared with RANSAC because it investigate non-rigid refinement methods in the future. is a popular algorithm for finding correspondences between We show promising results on a retrospective video point sets related by parametric transformations. If RANSAC acquisition from a laparoscopic liver surgery. The proposed is applied directly on the set of correspondences C,itis method is compared against a manual alignment performed unable to obtain a good alignment. This is mostly due to on the SmartLiver GUI. Qualitative results are provided to the fact that the initial set will contain a high number of out- illustrate that the proposed method manages to correctly liers, due to the low descriptiveness of the data. Moreover, identify the liver region in a challenging environment with for partial shape sizes characteristic to laparoscopic surgeries realistic noise levels, significant deformation and small par- (less than 23% in Fig. 3), there are multiple locations on the tial views. Note that the proposed method aims to estimate a liver which result in a good fit. However, by allowing for coarse surface alignment, which can then be further refined deformation in the proposed pruning technique, a set of cor- with a local algorithm. For example, Fig. 5-right shows an respondences, C , is obtained in the correct region of interest, overlay computed by applying LM-ICP on the coarse align- which can be further refined by RANSAC. This approach ment estimated by our method. Furthermore, the current is also more computationally efficient since RANSAC has computational time required to compute the initial alignment 123 International Journal of Computer Assisted Radiology and Surgery (2018) 13:947–956 955 between surfaces is approximately 20 s with non-optimised Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecomm code, which makes it feasible for clinical usage. ons.org/licenses/by/4.0/), which permits unrestricted use, distribution, Our experiments were all performed using the same choice and reproduction in any medium, provided you give appropriate credit of parameters, suggesting the proposed method is not very to the original author(s) and the source, provide a link to the Creative sensitive to variations. However, in the future we would like Commons license, and indicate if changes were made. to investigate their influence. Although the experiments pre- sented here show promising results, we would like to validate the proposed technique on clinical data from more patients References to test its robustness with respect to liver surface variations. Furthermore, we are looking into ways to automate the liver 1. Bartoli A, Collins T, Bourdel N, Canis M (2012) Computer assisted contour selection, such that no user interaction is needed and minimally invasive surgery: Is medical computer vision the answer the whole process is fully automated. to improving laparosurgery? J Med Hypotheses 79(6):858–863 2. Hughes-Hallett A, Pratt P, Dilley J, Vale J, Darzi A, Mayer E (2015) Augmented reality: 3d image-guided surgery. Cancer Imag- ing 15(1):O8 3. Plantefève R, Peterlik I, Haouchine N, Cotin S (2016) Patient- specific biomechanical modeling for guidance during minimally- Conclusion invasive hepatic surgery. Ann Biomed Eng 44(1):139–153 4. Rucker DC, Wu Y, Clements LW, Ondrake JE, Pheiffer TS, We propose a fast and global method for surface-based reg- Simpson AL, Jarnagin WR, Miga MI (2014) A mechanics-based istration of a 3D liver mesh extracted from a preoperative nonrigid registration method for liver surgery using sparse intraop- erative data. IEEE Trans Med Imaging 33(1):147–158 CT scan of the liver and the surface reconstruction of the 5. Thompson S, Totz J, Song Y, Johnsen S, Stoyanov D, Ourselin S, intraoperative laparoscopic video feed. We have validated Gurusamy K, Scheider C, Davidson B, Hawkes D, Clarkson MJ its performance with respect to the challenges characteris- (2015) Accuracy validation of an image guided laparoscopy system tic to laparoscopic surfaces on synthetic data, on a phantom for liver resection. In: SPIE, vol 9415, p 941509 6. Clements LW, Chapman WC, Dawant BM, Galloway RL, Miga MI dataset and on retrospective clinical data. We conclude that (2008) Robust surface registration using salient anatomical features the proposed method could potentially be used as an auto- for image-guided liver surgery: algorithm and validation. J Med matic way of obtaining a good initial alignment between the Phys 35(6):2528 two surfaces, given the required features. Moreover, it does 7. dos Santos TR, Seitel A, Kilgus T, Suwelack S, Wekerle AL, Kenngott H, Speidel S, Schlemmer HP, Meinzer HP, Heimann T, not require any advanced hardware, which makes it accessi- Maier-Hein L (2014) Pose-independent surface matching for intra- ble and comparatively easy to translate to a clinical setting. operative soft-tissue marker-less registration. J Med Image Anal 18(7):1101–1114 Acknowledgements This work is supported by the CRUK (http:// 8. Suwelack S, Röhl S, Bodenstedt S, Reichard D, Dillmann R, dos www.cancerresearchuk.org) and EPSRC Comprehensive Cancer Imag- Santos T, Maier-Hein L, Wagner M, Wünscher J, Kenngott H, ing Centre at KCL and UCL, and EPSRC CDT in Medical Imag- Müller BP, Speidel S (2014) Physics-based shape matching for ing [EP/L016478/1]. DH, MJC, DS were supported by the Well- intraoperative image guidance. J Med Phys 41(11):111901 come/EPSRC [203145Z/16/Z]. DS receives funding from EPSRC 9. Maier-Hein L, Franz AM, dos Santos TR, Schmidt M, Fangerau [EP/P012841/1]. BD was supported by the NIHR Biomedical Research M, Meinzer HP, Fitzpatrick JM (2012) Convergent iterative Centre at University College London Hospitals NHS Foundations Trust closest-point algorithm to accomodate anisotropic and inhomoge- and University College London. The imaging data used for this work nous localization error. IEEE Trans Pattern Anal Mach Intell were obtained with funding from the Health Innovation Challenge Fund 34(8):1520–1532 [HICF-T4-317], a parallel funding partnership between the Wellcome 10. Fitzgibbon AW (2003) Robust registration of 2d and 3d point sets. Trust and the Department of Health. The views expressed in this publica- J Image Vis Comput 21(13):1145–1153 tion are those of the author(s) and not necessarily those of the Wellcome 11. Cash DM, Miga MI, Glasgow SC, Dawant BM, Clements LW, Cao Trust or the Department of Health. Z, Galloway RL, Chapman WC (2007) Concepts and preliminary data toward the realization of image-guided liver surgery. J Gas- trointest Surg 11(7):844–859 Compliance with ethical standards 12. Nicolau SA, Pennec X, Soler L, Buy X, Gangi A, Ayache N, Marescaux J (2009) An augmented reality system for liver ther- Conflict of interest The authors declare that they have no conflict of mal ablation: design and evaluation on clinical cases. J Med Image interest. Anal 13(3):494–506 13. Teber D, Guven S, Simpfendörfer T, Baumhauer M, Güven EO, Human and animal rights All procedures performed in studies involv- Yencilek F, Gözen AS, Rassweiler J (2009) Augmented reality: a ing human participants were in accordance with the ethical standards of new tool to improve surgical accuracy during laparoscopic partial the institutional and/or national research committee and with the 1964 nephrectomy? Preliminary in vitro and in vivo results. Eur Urol Helsinki declaration and its later amendments or comparable ethical 56(2):332–338 standards. 14. Wild E, Teber D, Schmid D, Simpfendörfer T, Müller M, Baranski AC, Kenngott H, Kopka K, Maier-Hein L (2016) Robust augmented reality guidance with fluorescent markers in laparoscopic surgery. Informed consent Informed consent was obtained from all individual IJCARS 11(6):899–907 participants included in the study. 123 956 International Journal of Computer Assisted Radiology and Surgery (2018) 13:947–956 15. Mountney P, Fallert J, Nicolau S, Soler L, Mewes PW (2014) 25. Jacobson A (2015) gptoolbox: geometry processing toolbox. An augmented reality framework for soft tissue surgery. MICCAI https://github.com/alecjacobson/gptoolbox 17:423–431 26. Crane K, Weischedel C, Wardetzky M (2013) Geodesics in heat: 16. Feuerstein M, Mussack T, Heining SM, Navab N (2007) a new approach to computing distance based on heat flow. ACM Registration-free laparoscope augmentation for intra-operative Trans Gr 32(5):152:1–152:11 liver resection planning. In: Medical imaging 2007 27. Fischler MA, Bolles RC (1981) Random sample consensus: 17. Fusaglia M, Hess H, Schwalbe M, Peterhans M, Tinguely P, Weber paradigm for model fitting with applications to image analysis and S, Lu H (2016) A clinically applicable laser-based image-guided automated cartography. Commun ACM 24(6):381–395 system for laparoscopic liver procedures. IJCARS 11(8):1499– 28. D’Errico J (2016) Surface fitting using gridfit. https://uk. 1513 mathworks.com/matlabcentral/fileexchange/8998-surface-fitting- 18. Raviv D, Dubrovina A, Kimmel R (2012) Hierarchical matching using-gridfit of non-rigid shapes. In: SSVM, pp 604–615 29. Cignoni P, Callieri M, Corsini M, Dellepiane M, Ganovelli F, 19. Leordeanu M, Hebert M (2005) A spectral technique for correspon- Ranzuglia G (2008) MeshLab: an open-source mesh processing dence problems using pairwise constraints. ICCV 2:1482–1489 tool. In: Eurographics Italian chapter conference 20. Huang QX, Adams B, Wicke M, Guibas LJ (2008) Non-rigid 30. Gibson ED, Robu MR, Thompson SA, Edwards PE, Schneider registration under isometric deformations. Comput Gr Forum C, Gurusamy K, Davidson BR, Hawkes DJ, Barrat DC, Clarkson 27(5):1449–1457 MJ (2017) Deep residual networks for automatic segmentation of 21. Peyré G (2011) The numerical tours of signal processing. Comput laparoscopic videos of the liver. In: SPIE, vol 10135, pp 101351M– Sci Eng 13(4):94–97 101351M–6 22. Gelfand N, Ikemoto L, Rusinkiewicz S, Levoy M (2003) Geometri- 31. Totz J, Thompson S, Stoyanov D, Gurusamy K, Davidson B, cally stable sampling for the ICP algorithm. In: 3DIM, pp 260–267 Hawkes DJ, Clarkson MJ (2014) Fast semi-dense surface recon- 23. Yang J, Zhang Q, Xiao Y, Cao Z (2017) Toldi: an effective and struction from stereoscopic video in laparoscopic surgery. IPCAI robust approach for 3d local shape description. Pattern Recognit 8498:206–215 65:175–187 24. Sethian JA (1996) A fast marching level set method for monoton- ically advancing fronts. PNAS 93(4):1591–1595

Journal

International Journal of Computer Assisted Radiology and Surgery – Springer Journals

Published: May 7, 2018

There are no references for this article.