Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Mechanical assembly assistance using marker-less augmented reality system

Mechanical assembly assistance using marker-less augmented reality system This paper aims to propose a real-time augmented reality (AR)-based assembly assistance system using a coarse-to-fine marker-less tracking strategy. The system automatically adapts to tracking requirement when the topological structure of the assembly changes after each assembly step.Design/methodology/approachThe prototype system’s process can be divided into two stages: the offline preparation stage and online execution stage. In the offline preparation stage, planning results (assembly sequence, parts position, rotation, etc.) and image features [gradient and oriented FAST and rotated BRIEF (ORB)features] are extracted automatically from the assembly planning process. In the online execution stage, too, image features are extracted and matched with those generated offline to compute the camera pose, and planning results stored in XML files are parsed to generate the assembly instructions for manipulators. In the prototype system, the working range of template matching algorithm, LINE-MOD, is first extended by using depth information; then, a fast and robust marker-less tracker that combines the modified LINE-MOD algorithm and ORB tracker is designed to update the camera pose continuously. Furthermore, to track the camera pose stably, a tracking strategy according to the characteristic of assembly is presented herein.FindingsThe tracking accuracy and time of the proposed marker-less tracking approach were evaluated, and the results showed that the tracking method could run at 30 fps and the position and pose tracking accuracy was slightly superior to ARToolKit.Originality/valueThe main contributions of this work are as follows: First, the authors present a coarse-to-fine marker-less tracking method that uses modified state-of-the-art template matching algorithm, LINE-MOD, to find the coarse camera pose. Then, a feature point tracker ORB is activated to calculate the accurate camera pose. The whole tracking pipeline needs, on average, 24.35 ms for each frame, which can satisfy the real-time requirement for AR assembly. On basis of this algorithm, the authors present a generic tracking strategy according to the characteristics of the assembly and develop a generic AR-based assembly assistance platform. Second, the authors present a feature point mismatch-eliminating rule based on the orientation vector. By obtaining stable matching feature points, the proposed system can achieve accurate tracking results. The evaluation of the camera position and pose tracking accuracy result show that the study’s method is slightly superior to ARToolKit markers. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Assembly Automation Emerald Publishing

Mechanical assembly assistance using marker-less augmented reality system

Download PDF
11 pages

Loading next page...
 
/lp/emerald-publishing/mechanical-assembly-assistance-using-marker-less-augmented-reality-ce8QVBM4Zz
Publisher
Emerald Publishing
Copyright
© Emerald Publishing Limited
ISSN
0144-5154
DOI
10.1108/aa-11-2016-152
Publisher site
See Article on Publisher Site

Abstract

This paper aims to propose a real-time augmented reality (AR)-based assembly assistance system using a coarse-to-fine marker-less tracking strategy. The system automatically adapts to tracking requirement when the topological structure of the assembly changes after each assembly step.Design/methodology/approachThe prototype system’s process can be divided into two stages: the offline preparation stage and online execution stage. In the offline preparation stage, planning results (assembly sequence, parts position, rotation, etc.) and image features [gradient and oriented FAST and rotated BRIEF (ORB)features] are extracted automatically from the assembly planning process. In the online execution stage, too, image features are extracted and matched with those generated offline to compute the camera pose, and planning results stored in XML files are parsed to generate the assembly instructions for manipulators. In the prototype system, the working range of template matching algorithm, LINE-MOD, is first extended by using depth information; then, a fast and robust marker-less tracker that combines the modified LINE-MOD algorithm and ORB tracker is designed to update the camera pose continuously. Furthermore, to track the camera pose stably, a tracking strategy according to the characteristic of assembly is presented herein.FindingsThe tracking accuracy and time of the proposed marker-less tracking approach were evaluated, and the results showed that the tracking method could run at 30 fps and the position and pose tracking accuracy was slightly superior to ARToolKit.Originality/valueThe main contributions of this work are as follows: First, the authors present a coarse-to-fine marker-less tracking method that uses modified state-of-the-art template matching algorithm, LINE-MOD, to find the coarse camera pose. Then, a feature point tracker ORB is activated to calculate the accurate camera pose. The whole tracking pipeline needs, on average, 24.35 ms for each frame, which can satisfy the real-time requirement for AR assembly. On basis of this algorithm, the authors present a generic tracking strategy according to the characteristics of the assembly and develop a generic AR-based assembly assistance platform. Second, the authors present a feature point mismatch-eliminating rule based on the orientation vector. By obtaining stable matching feature points, the proposed system can achieve accurate tracking results. The evaluation of the camera position and pose tracking accuracy result show that the study’s method is slightly superior to ARToolKit markers.

Journal

Assembly Automation – Emerald Publishing

Published: Jan 23, 2018

Keywords: Tracking; Augmented reality; Assembly; Marker-less; Line-mod; ORB

References

  • Accuracy in optical tracking with fiducial markers: an accuracy function for ARToolKit
    Abawi, D.F.; Joachim, B.; Ralf, D.
  • Providing guidance for maintenance operations using automatic markerless augmented reality system
    Alvarez, H.; Aguinaga, I.; Borro, D.
  • A survey of augmented reality
    Azuma, R.T.
  • Decision support for sequence generation in an assembly oriented design environment
    Barnes, C.J.; Jared, G.E.M.; Swift, K.G.
  • SURF: speeded up robust features
    Bay, H.; Tuytelaars, T.; Van Gool, L.
  • Large-scale direct slam for omnidirectional cameras
    Caruso, D.; Engel, J.; Cremers, D.
  • Markless tracking based on natural feature for augmented reality
    Chen, Z.; Li, X.
  • A novel representation of parts for accurate 3d object detection and tracking in monocular images
    Crivellaro, A.; Rad, M.; Verdie, Y.; Yi, K.M.; Fua, P.; Lepetit, V.
  • Real-time learning and detection of 3D texture-less objects: a scalable approach
    Damen, D.; Bunnun, P.; Calway, A.; Mayol-Cuevas, W.
  • LSD-SLAM: large-scale direct monocular SLAM
    Engel, J.; Schöps, T.; Cremers, D.
  • Rigid object tracking algorithms for low-cost AR devices
    Garrett, T.; Debernardis, S.; Radkowski, R.; Oliver, J.H.
  • Gradient response maps for real-time detection of textureless objects
    Hinterstoisser, S.; Cagniart, C.; Ilic, S.; Sturm, P.; Navab, N.; Fua, P.
  • Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes
    Hinterstoisser, S.; Lepetit, V.; Ilic, S.; Holzer, S.; Bradski, G.; Konolige, K.; Nava, N.
  • Pn learning: bootstrapping binary classifiers by structural constraints
    Kalal, Z.; Matas, J.; Mikolajczyk, K.
  • Robust model-based 3d object recognition by combining feature matching with tracking
    Kim, S.; Kweon, I.; Kim, I.
  • Multithreaded hybrid feature tracking for markerless augmented reality
    Lee, T.; Hollerer, T.
  • Randomized trees for real-time keypoint recognition
    Lepetit, V.; Lagger, P.; Fua, P.
  • BRISK: Binary robust invariant scalable keypoints
    Leutenegger, S.; Chli, M.; Siegwart, R.Y.
  • A robust O (n) solution to the perspective-n-point problem
    Li, S.; Xu, C.; Xie, M.
  • Distinctive image features from scale-invariant keypoints
    Lowe, D.G.
  • Fast and globally convergent pose estimation from video images
    Lu, C.P.; Hager, G.D.; Mjolsness, E.
  • An affine invariant interest point detector
    Mikolajczyk, K.; Schmid, C.
  • A coarse-to-fine model for 3D pose estimation and sub-category recognition
    Mottaghi, R.; Xiang, Y.; Savarese, S.
  • ORB-SLAM: tracking and mapping recognizable features
    Mur-Artal, R.; Tardós, J.D.
  • Integrated product design and assembly planning in an augmented reality environment
    Ng, L.X.; Wang, Z.B.; Ong, S.K.; Nee, A.Y.C.
  • Real-time model-based rigid object pose estimation and tracking combining dense and sparse visual cues
    Pauwels, K.; Rubio, L.; Diaz, J.; Ros, E.
  • From contours to 3d object detection and pose estimation
    Payet, N.; Todorovic, S.
  • Real-time 3d model-based tracking: combining edge and texture information
    Pressigout, M.; Marchand, E.
  • Pwp3d: real-time segmentation and tracking of 3d objects
    Prisacariu, V.A.; Reid, I.D.
  • A point cloud-based method for object alignment verification for augmented reality applications
    Radkowski, R.
  • ORB: an efficient alternative to SIFT or SURF
    Rublee, E.; Rabaud, V.; Konolige, K.; Bradski, B.
  • Bold features to detect texture-less objects
    Tombari, F.; Franchi, A.; Di Stefano, L.
  • Deformable random dot markers
    Uchiyama, H.; Marchand, E.
  • Stable real-time 3d tracking using online and offline information
    Vacchetti, L.; Lepetit, V.; Fua, P.
  • Multi-modal augmented-reality assembly guidance based on bare-hand interface
    Wang, X.; Ong, S.K.; Nee, A.Y.C.
  • Real-virtual components interaction for assembly simulation and planning
    Wang, X.; Ong, S.K.; Nee, A.Y.C.
  • A pipeline for rapidly incorporating real objects into a mixed environment
    Wang, X.; Kotranza, A.; Quarles, J.; Lok, B.
  • Assembly planning and evaluation in an augmented reality environment
    Wang, Z.B.; Ng, L.X.; Ong, S.K.; Nee, A.Y.C.
  • Global optimal searching for textureless 3D object tracking
    Wang, G.; Wang, B.; Zhong, F.; Qin, X.; Chen, B.
  • Intelligent augmented reality training for motherboard assembly
    Westerfield, G.; Mitrovic, A.; Billinghurst, M.
  • Fusion of vision and gyro tracking for robust augmented reality registration
    You, S.; Neumann, U.
  • An authorable context-aware augmented reality system to assist the maintenance technicians
    Zhu, J.; Ong, S.K.; Nee, A.Y.C.
  • A survey of tracking technology for virtual environments
    Baillot, Y.; Davis, L.; Rolland, J.
  • Large-scale direct SLAM with stereo cameras
    Engel, J.; Stückler, J.; Cremers, D.