TY - JOUR
AU1 - Liao, Pu
AU2 - Guixiong, Liu
AB - Introduction The butt welds of pressure vessels A and B are important factors that affect stress. The weld surface consists of four parameters: weld width, reinforcement, undercut, and misalignment, as specified in the relevant standards. These parameters directly reflect the stress concentration at the welding position, and their measurement is an important for evaluation the welding quality [1,2]. Currently, relatively mature parameter measurement methods are manually completed using a magnifying glass, weld inspection ruler, angle ruler, and other tools. However, these methods exhibit low accuracy, low efficiency, large workload, and high labor intensity. Given the good characteristics of machine vision inspection methods, there have been various degrees of research and applications in welding process inspection or welding seam surface parameters. Based on the different imaging light sources, machine vision weld inspection methods can be classified into two branches: passive and active visual weld inspection. The passive vision welding seam detection method involves imaging in a natural light environment, using clustering, template matching, Hough transformation, and other algorithms to identify and locate the weld defect information (undercut, wrong edge, etc.) in the image. For example, Kumar used a BP neural network to identify and classify MIG butt welding surface defects according to the standard EN25817 [3]. Apostolos (2015) used the Sobel edge detection algorithm to obtain the undercut edge feature of the weld image and matched the undercut feature template with the feature extracted from the edge of the image, which was detected to realize weld undercut feature recognition [4]. Ding (2016) proposed the use of the circular Hough transformation to locate weld images after edge extraction for determining the location of weld defects [5]. Passive visual weld inspection method has been mainly used in the field of weld defect recognition and classification. However, owing to the poor reflection ability of the weld parameters, it is impossible to calculate the specific values of the weld size parameters in natural-light weld imaging. Active vision imaging refers to imaging under an artificial light source (including ultraviolet light, infrared light, and laser.). The artificial light source applied to weld inspection is the most structured laser, which can be used to inspect and capture the laser line on the welded surface. The parameter information of the curved welded surface can be expressed as the feature points of the curve. [6,7]. The extraction of the feature points of the welding parameter measurement from the welding seam laser line is important for active vision welding seam measurements [8]. Current mainstream feature point determination methods include the feature point extraction method based on curve slope analysis [9], corner detection method curve inflection point [10], and curve processing and fitting [11]. The feature point extraction methods based on curve slope analysis are as follows: In 2017, Muhammad calculated the first-order derivative of each pixel in each laser curve in an image coordinate system [12]. If the derivative value exceeded the threshold value, it was determined to be a feature point. However, this method is only appropriate for extracting simple-shaped sheet joint weld feature points, and it is not applicable to the extraction of feature points for cylindrical longitudinal welds. Researchers from Tsinghua University have used the third-order Savitzky–Golay filter to calculate the second-derivative value of the laser fringe curve [13]. If the calculated value exceeded the threshold value, then it was determined as an undercut feature point. In [14], a structured light-vision sensor with a narrowband filter was developed. After the centerline extraction of the sensor output image, the weld width and undercut feature points are extracted by calculating the second-order derivative of the laser stripe centerline. However, the algorithm can only measure the parameter width and undercut. This method is better than the previous curve method, but it is sensitive to noise. The feature point extraction methods based on the corner detection method curve inflection points are as follows. In [15], Kovacevic used a corner detection algorithm to extract the feature points of the weld toe and reinforcement on both sides of the weld of U-shaped, V-shaped, square, and lap joints. However, this method does not consider the feature point extraction of undercut defects. In [16], Hessian eigenvectors were used to locate the position of the weld centerline inflection point as a feature point, where the parameter width and reinforcement can be calculated. Consequently, it cannot be used to measure the undercut and misalignment parameters. Feature point extraction methods, based on curve processing and fitting, include performing a continuous sym8 wavelet transformation on the laser curve in the image coordinate system and using the local maximum value of the sym8 wavelet coefficient as the feature point of the weld undercut [17]. However, a smoothly changing feature point (reinforcement, width) cannot be obtained using this method because it produces sharp protruding points in the curve (undercut defects, etc.). A curve fitting plus derivative method was proposed in [18], which uses the zero point of the second derivative of the fitted curve as the characteristic point of the apex of the V-shaped plate welded joint. In [19], a structured light vision sensor was used to determine feature points using NURBS curve segmentation and fitting. The influence of image noise was significantly reduced via fitting because the derivative of the fitted curve was used as opposed to direct determination of the slope of the curve. However, the fitting curve differed from the curve, which in turn caused feature point position errors. The representative products of active visual weld inspection include the weld measuring instrument used for various groove welds devised by Servo-Robot of Canada [20], which can measure the reinforcement, width, and undercut, and the seam tracking system developed by the British MetaVision Company [21], which can monitor the longitudinal and circumferential seam welding process of welding robots. It is evident that when compared to the passive visual weld inspection method, the active visual weld detection method exhibits a better reflection of the image characteristics of the weld parameters and lower inspection environment requirements. This has become the mainstream method for weld visual tracking, identification, and inspection. However, the applicability of the current feature-point extraction algorithm should be improved. It is not possible for any algorithm to simultaneously detect the four parameters of weld width, reinforcement, undercut depth, and misalignment. In recent years, deep learning has advanced rapidly in machine vision research. The processing methods for extracting specified feature points from images are classified as pose estimation algorithms. The Deep-Pose network for image feature estimation was first proposed by Google in 2014 [22]. The front end of the network uses deep convolutional neural networks (CNN) to extract image feature information at multiple scales, and the back end uses multi-scale features of the convolutional layer. The output is connected to the fully connected layers (FC), and the coordinate extraction task of the feature points in the image coordinate system is completed. Owing to the limitation in CNN’s feature extraction performance at the time, the Deep-Pose accuracy was low. Megvii Technology (2018) proposed a cascaded pyramid network (CPN) [23], which was classified into Global-Net and Refine-Net. Global-Net uses a pyramid structure to extract CNN to extract image multiscale features, and Refine-Net structure regresses on image feature points. To accurately estimate the coordinate information of feature points, the complex and inefficient FC layer in Deep-Pose was abandoned, and a simple and efficient regression method of the deconvolution layer and mean pooling layer structure was used. To date, this is the best network for feature point extraction. A simple and effective bottom-up structure of DeepLabCut has been proposed in the literature [24] for pose estimation of animals, and the network exhibits good migration performance and can be applied to feature point extraction of other objects. In [25], a residual step network structure was proposed to fuse features of the same spatial dimension (intra-level features) to further optimize the key point locations. To solve the problem of severe degradation of feature map resolution in the calculation of traditional CNN structures, [26] proposed an HR-net that can maintain a high-resolution feature map output during the convolution calculation. Later, [27] designed a bottom-up human pose estimation structure based on HR-net. Reference [28] proposed a TokenPose network structure, which replaces the image convolution calculation with the transformer structure in the NLP research field. The network feature extraction AP is close to the literature result, but the network structure is lighter, and it is only suitable for human pose estimation. Therefore, this study attempts to combine a deep learning image feature extraction method with active visual weld inspection technology to provide a new research idea for the field of weld inspection. The main contributions of this study are as follows: We analyzed the reasons due to which the standard welding seam surface parameter measurement method cannot be used for actual pressure pipeline inspections, and we proposed a measurement index for welding seam surface parameters to account for the coexistence of numerous surface parameters. 2. A deep convolutional neural network-based image parameter feature point extraction approach can simultaneously extract all parameter feature points in a single image. 3. We proposed a method based on the third-order NURBS curve for enhancing the image data of the weld surface profile of a pressure vessel, which can significantly reduce the number of deep network data collection tasks. The remainder of this paper is organized as follows. In the section on modeling and numerical analyses, the design details of a weld surface device based on a laser profile sensor are provided and an overview of the weld surface parameter calculation algorithms are outlined. Various experimental results are provided to confirm the validity of the proposed methods in the results section. Finally, conclusions are provided in the discussion section. Modeling and numerical analyses Weld surface parameters measuring device based on laser profile sensor A laser profile sensor, electric slide-in Z-axis, manual slide in Y-axis, and computer are shown in Fig 1 as a system for measuring the weld surface parameters based on a structured light model. A KEYENCE LJ-V7080 laser profile sensor with a 32-mm built-in camera and built-in laser with a wavelength of 405 nm were used in the experiment. The measuring ranges of the sensor in the X-and Y-axis directions were 20 mm and 46 mm, respectively. The sensor was installed directly above the cylinder for measurement, and its imaging distance was within this range. The one-line-shaped laser emitted by the sensor hit the surface of the pressure vessel to be measured. The camera inside the sensor captured images near the linear laser range in real time, and the point cloud data were output to the computer via an internal algorithm. The sensor was fixed on the Z-axis electric guide rail, and the line laser emission surface was parallel to the cross-section of the pressure vessel cylinder. The Z-axis electric slide rail was connected to the Y-axis manual slide rail via a bracket, allowing the sensor to move in both directions. The laser sensor and pressure pipe cylinder were kept tangential to the movement direction during the detection procedure. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 1. Device for measuring weld surface parameters, which consists of a laser profile sensor, electric slide-in Z-axis, manual slide in Y-axis, and computer. https://doi.org/10.1371/journal.pone.0267743.g001 The laser profile sensor-based welding seam surface profile parameter detection system described in this study is shown in Fig 2. The entire detection process is as follows: ① the laser profile sensor emits a laser that is perpendicular to the weld surface of the pressure vessel; ② point set S generated by the sensor is preprocessed to generate the weld contour curve image G; ③ deep learning network, and the characteristic points of the weld parameters are output at coordinates P (x, y); ④ according to the proposed numerical calculation index of the weld surface parameters, the numerical calculation of the parameters is completed. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 2. Flowchart of the laser profile sensor-based welding seam surface profile parameter detection system. https://doi.org/10.1371/journal.pone.0267743.g002 In general, the weld contour curve image G is generated from the weld contour point set S, and this method requires that it can highlight and accurately reflect the surface contour characteristics of the weld. The common image-production method first creates an m×n×3 three-channel back matrix Iback. Then, to generate image G, each contour point set was filled with a black background image in the form of monochrome pixels. Although the picture background (black background) and content (contour pixels) scales are extremely small, as low as 1/10000, this image-generating method can precisely depict the surface contour properties of the weld. The model can be over-fitted if the image is used for subsequent deep learning training. Therefore, after the black background image is created using the above method, the following formula is used to generate a contour curve image G with a large ratio between the image background and content. Let Pix be the RGB value of the contour point color and 〈⋅〉 be the calculation method of rounding down. Then, G can be expressed as follows: (1) Weld surface parameter measurement index with multiple defect parameters The butt joint longitudinal and girth weld reinforcements, as well as three measurement surface parameters (width, undercut, and misalignment), were defined in accordance with the AWS A3.0 "Definition of Standard Welding Terms" [29], ISO 5817 "Welding Joints" [30], AWS D1.1 "Welding Specification for Steel Structures" [31], and ASME VIII "Boiler and Pressure Vessel Manufacturing Code" standards [32]. A schematic of the four-parameter measurement requirements for butt welds is shown in Fig 3. The parameter corresponding to weld width is defined as the distance between the two weld toes according to the AWS A3.0 "Definition of Standard Welding Terms," which is the junction between the weld surface and base metal. The weld reinforcement hre is a parameter in which the weld metal exceeds the height of the fillet welding groove. The weld parameter undercut hun_cut denotes the size of grooves or depressions produced along with the base metal of the weld toe owing to improper selection of welding parameters or incorrect operation methods. Fig 3(A) shows the definitions of weld width lwidth, weld reinforcement hre, and weld undercut hun_cut in the standard. Weld misalignment is defined by the ASME VIII "Boiler and Pressure Vessel Manufacturing Code" standard as the phenomenon of dislocation and unevenness due to the deformation of the welding deviation and other factors during welding. The parameter weld misalignment halign denotes the size and amplitude of the misalignment as shown in Fig 3(B). Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 3. Schematic diagram of parameter definition of butt weld 4. (a) Definition of weld reinforcement, width, and undercut. (b) Definition of weld misalignment. https://doi.org/10.1371/journal.pone.0267743.g003 The existence of weld defects alone, which is a measurement index under the ideal weld condition, is an example of each weld appearance parameter defined and detailed in the standard state. An initially formed weld can exhibit multiple coexisting defects. In the case of weld defects (such as wrong under the influence of edge volume), the definition of the welding seam parameter measurement index in the standard example diagram is no longer applicable. Hence, the appearance parameters of welds at the cross-sectional position of the welds under the conditions of normal welds without defects, single-defect welds, and multi-defect welds are discussed in this study with reference to measurement diagrams of weld appearance parameters in relevant standards. Fig 4 shows a schematic of the weld parameter measurement indicators for various weld shapes. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 4. Schematic diagrams of welding seam parameter measurement index under different welding seam shapes. (a) Normal weld. (b) Welds with the wrong sides. (c) Welds with single undercut and misalignment. (d) Welds with double undercut and misalignment. https://doi.org/10.1371/journal.pone.0267743.g004 The parameter reinforcement feature point Pre was determined as the highest point of the welding area of the profile curve. The parameter width feature points and are determined as the intersection of the welding curve and the base metal curve, i.e., the welding toes on both sides, as shown in Fig 4(A) and 4(B). The misalignment feature points and coincide with the width feature points in the cases of no defect parameters and single misalignment defects. However, given that the weld toe on the undercut side disappears in the case of a defective undercut, the parameter width feature point is modified to the junction point of the undercut depression curve and welding curve. The misalignment feature point at this time is the intersection point between the undercut curve and base material curve as shown in Fig 4(C) and 4(D), respectively. The parameter feature points in normal welds are extreme points or corner points, as shown in the above feature point selection example, and the traditional curve extreme and reciprocal analysis methods can complete the feature point extraction task. The characteristic points of the cross-section curve width are weak owing to undercut defects. It is difficult to simultaneously extract all of the characteristic points based on traditional curve analysis methods, and to date, no scholars have proposed a method for simultaneously extracting all of the appearance parameters. Consequently, in terms of image processing feature analysis, in this study, we employed deep learning image semantic segmentation methods to classify weld defects and extracted four parameter feature points of welds in laser curve images. Design of image feature point extraction network based on CNN The structural diagram of the coding-decoding-free image feature point extraction network (EDE-net), proposed for this study, is depicted in Fig 5. The input of this network is the preprocessed laser profile image of the weld, and the output of the network is the pixel position of the parameter feature point. The coding part of the network is composed of the CNN backbone. First, the CNN backbone output feature map outputs an n-dimensional feature map after upsampling at the branch of the decoding part. Then, the location of the feature point, roughly extracted by the network, can be expressed as (i = 1,…, n). On branch two of the decoding part, the output feature map with dimensions of 2n is sampled and processed, and feature point correction information is finely extracted. Finally, the position of the parameter feature point in the input image, namely , can be obtained by combining the information of the two output feature maps and . Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 5. Network structure diagram of EDE-net. https://doi.org/10.1371/journal.pone.0267743.g005 The EDE-net coding structure output feature map up-sampling methods include bilinear interpolation, deconvolution, and depooling methods. If the image up-sampling multiple is kupsample, then the input scale is ih×iw×3 image I, the up-sampling output scale is ih*kupsample×iw*kupsample×3 image O, and the pixel position (kpx, kpy) in the image O pixel value is sampled and mapped to the pixel position (px, py) of the image I pixel value . The up-sampled image O after bilinear interpolation is determined as follows: (2) where , , and 〈⋅〉 are round-down calculations. Based on the formula, it is evident that the bilinear interpolation upsampling method should traverse each pixel, which features a large amount of calculation and slower running speed. The deconvolution upsampling mechanism is shown in Fig 6. The adjacent and surrounding pixels in Image I were interpolated and filled with pixels (usually the pixel value was 0). To obtain the inverse convolution output image O, the convolution kernel and supplemented images were used for the convolution calculation. The pooling kernel was set according to the maximum and average pooling methods, and the de-pooling up-sampling mechanism was the same as the deconvolution up-sampling image supplement method. The up-sampling magnification of the deconvolution and de-pooling methods was lower, but the amount of calculation was less. In deconvolution, the convolution kernel can participate in the entire network training and update the weights, whereas the weights of the pooling kernel cannot be changed. Therefore, EDE-net uses the deconvolution calculation as the up-sampling method. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 6. Deconvolution up-sampling mechanism diagram. https://doi.org/10.1371/journal.pone.0267743.g006 The EDE-net network branch is the task of rough extraction of feature point positions. Based on the concept of completely convolutional network semantic segmentation, the deep feature map output by the convolutional network was processed via a single layer with a stride of 2 and a 3 × 3 deconvolution kernel with a number of feature points n. The output dimension of the feature map was reduced to n, with each dimension corresponding to the approximate position information of each feature point in the input weld centerline image. Each pixel value is the probability that the feature point is at that position. The theoretical output result, M, of the feature extraction module branch is shown in Fig 7. If the theoretical regression object M is the only single-point position of the characteristic point in the laser centerline image of the weld, then the position outside the characteristic point in M is a negative sample. If the positive and negative samples are seriously out of balance, overfitting can occur during network training. Therefore, the feature point distance threshold T was introduced, and the pixel distance from the feature point position was set as positive samples within T. This in turn solved the imbalance problem. The theoretical output feature map M can be obtained as follows: (3) Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 7. Theoretical output results of EDE-net branch 1. https://doi.org/10.1371/journal.pone.0267743.g007 The distances between the feature point and background within the threshold T can be treated as a binary classification task. Hence, the focal loss of the feature classification task can be utilized as the loss function as shown in the output feature map of the branch-network theory. The adjustment parameters α and γ are compatible with the feature classification task Focal-loss [33]. Therefore, the branch-loss function LM is as follows: (4) where matrix S elements are all one. EDE-net branch 2 is a feature point position–correction task. The convolutional network’s deep feature map is processed through a single layer with a stride of 2, scale of 3×3, and dimension of 2n deconvolution kernels. This yields a 2n-dimensional feature map . The theoretical output results N of EDE-net network branch 2 are shown in Fig 8. Based on the approximate location of M feature points, the branch two-theory output feature map N adds a feature point location correction value. This implies that the position value of the corresponding element of N2i-1 is the true value of Mi, and N2i is the pixel difference between the theoretical feature points and X-Y-axis of the image coordinate system at the element’s position. Therefore, the theoretical outputs N2i-1 and N2i branch 2 are defined as follows: (5) where c denotes the ratio of the input image scale to the output feature map scale of upsampling. The feature point position correction task is a numerical regression task. Therefore, the Huber loss is used to establish the loss function: (6) Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 8. Theoretical output results of EDE-net branch 2. https://doi.org/10.1371/journal.pone.0267743.g008 Then, the EDE-net network loss function can be obtained as LEDE = LN+LM. Image data enhancement method for pressure vessel weld surface profile based on third-order NURBS curve A CNN requires a large number of datasets for training, and the data capacity of the training set is directly related to the CNN’s ability to extract feature points. The conventional approach of producing training sets involves acquiring contour images with an active vision imaging device and manually labeling the location coordinates of the feature points in each image. To avoid this, in the current study, we provide a surface parameter simulation method with the coexistence of multiple defects on the weld surface of the pressure vessel. This in turn allows variation of the types of weld parameters, number of feature points, and parameter values, and thereby, effectively reduces the number of training set collection tasks. A simulation diagram of the normal weld curve is shown in Fig 9. The parent material area of the curve in the image coordinate system (Fmetal(x,y)) is expressed as follows: (7) where Setwidth denotes the weld width parameter, Rstand denotes the diameter of the pressure vessel cylinder, Lpic denotes the imaging area length, and Wpic denotes the width. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 9. Simulation of normal weld curve. https://doi.org/10.1371/journal.pone.0267743.g009 The simulation feature point position of the normal weld contour curve is presented in Table 1 when the simulation reinforcement is Setre. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 1. Simulation feature point position of normal weld contour curve. https://doi.org/10.1371/journal.pone.0267743.t001 A non-uniform rational B-spline (NURBS) curve [34] simulation weld zone curve was constructed based on the aforementioned three points: Pre, , and . Assuming that the control point of the third-order NURBS curve is k = 3, di = [x,y]T, wi is the weight factor of the curve control point, and w0, w2>0, the remaining wi≥0, Fweld(u) can be expressed as follows: (8) (9) Furthermore, di information can be inversely calculated with three NURBS curve data points: Pre, , and . Let the node vectorbe when the curve is opened and the control point be n = m+k-1 = 5 such that the curve passes through the first and last control points. The node vector should exhibit k+1 repeatability, and it is set using the accumulation chord length method. This implies that u0 = u1 + u2 = u3 = 0, u5 = u6 + u7 = u8 = 1, , and the setting of the weight factor wi of the curve control point are as follows: (10) The beginning and end points of the curve control points were consistent with those of the data points throughout the reverse solution procedure. The connection points of the curved segment correspond to the nodes of the NURBS curve-defining domain. Therefore, if the data point qi corresponds to the node value ui+3(i = 0,1,2), then the solution condition for di is as follows: (11) To complete the control-point solution provided by the tangent vector boundary conditions, the following two equations must be included: (12) By combining the above equations, we can complete the coordinate position solution di and construct curve Fweld(u). Defect parameter undercut simulation. If the undercut width, depth, and offset are δun_cut, Setun_cut, and Δun_cut, respectively, then they indicate that the undercut feature point and width feature point are separated in the X-axis direction. Then, the value range of x in the curve of of the parent metal area on the right is modified to . The positions of the simulated feature points of the undercut weld profile curve with defect parameters are listed in Table 2. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 2. Location of the feature point in the simulation of the undercut weld profile curve with the defect parameter. https://doi.org/10.1371/journal.pone.0267743.t002 To perform the single-defect undercut weld contour curve simulation, five-point coordinates Pre, , , Pun_cut, and Pmis_align can be used as the NURBS curve data points as shown in Fig 10. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 10. Simulation of weld profile curve with single defect parameter undercut Setun_cut. https://doi.org/10.1371/journal.pone.0267743.g010 Defect parameter misalignment simulation. In this case, the width feature points on both sides of the weld coincide with the deviation feature point, and curve of the base material area on the left is consistent with the simulation of the normal weld. The curve of the base material area on the right by Setmis_align along the Y-axis of the image coordinate system and on the right side of the weld is as follows: (13) The single-defect misaligned weld profile curve simulation was performed using the coordinates Pre, , and . The three points were used as NURBS curve data points as shown in Fig 11. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 11. Simulation of weld profile curve with single defect parameter misalignment Setalign. https://doi.org/10.1371/journal.pone.0267743.g011 Weld surface parameters measuring device based on laser profile sensor A laser profile sensor, electric slide-in Z-axis, manual slide in Y-axis, and computer are shown in Fig 1 as a system for measuring the weld surface parameters based on a structured light model. A KEYENCE LJ-V7080 laser profile sensor with a 32-mm built-in camera and built-in laser with a wavelength of 405 nm were used in the experiment. The measuring ranges of the sensor in the X-and Y-axis directions were 20 mm and 46 mm, respectively. The sensor was installed directly above the cylinder for measurement, and its imaging distance was within this range. The one-line-shaped laser emitted by the sensor hit the surface of the pressure vessel to be measured. The camera inside the sensor captured images near the linear laser range in real time, and the point cloud data were output to the computer via an internal algorithm. The sensor was fixed on the Z-axis electric guide rail, and the line laser emission surface was parallel to the cross-section of the pressure vessel cylinder. The Z-axis electric slide rail was connected to the Y-axis manual slide rail via a bracket, allowing the sensor to move in both directions. The laser sensor and pressure pipe cylinder were kept tangential to the movement direction during the detection procedure. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 1. Device for measuring weld surface parameters, which consists of a laser profile sensor, electric slide-in Z-axis, manual slide in Y-axis, and computer. https://doi.org/10.1371/journal.pone.0267743.g001 The laser profile sensor-based welding seam surface profile parameter detection system described in this study is shown in Fig 2. The entire detection process is as follows: ① the laser profile sensor emits a laser that is perpendicular to the weld surface of the pressure vessel; ② point set S generated by the sensor is preprocessed to generate the weld contour curve image G; ③ deep learning network, and the characteristic points of the weld parameters are output at coordinates P (x, y); ④ according to the proposed numerical calculation index of the weld surface parameters, the numerical calculation of the parameters is completed. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 2. Flowchart of the laser profile sensor-based welding seam surface profile parameter detection system. https://doi.org/10.1371/journal.pone.0267743.g002 In general, the weld contour curve image G is generated from the weld contour point set S, and this method requires that it can highlight and accurately reflect the surface contour characteristics of the weld. The common image-production method first creates an m×n×3 three-channel back matrix Iback. Then, to generate image G, each contour point set was filled with a black background image in the form of monochrome pixels. Although the picture background (black background) and content (contour pixels) scales are extremely small, as low as 1/10000, this image-generating method can precisely depict the surface contour properties of the weld. The model can be over-fitted if the image is used for subsequent deep learning training. Therefore, after the black background image is created using the above method, the following formula is used to generate a contour curve image G with a large ratio between the image background and content. Let Pix be the RGB value of the contour point color and 〈⋅〉 be the calculation method of rounding down. Then, G can be expressed as follows: (1) Weld surface parameter measurement index with multiple defect parameters The butt joint longitudinal and girth weld reinforcements, as well as three measurement surface parameters (width, undercut, and misalignment), were defined in accordance with the AWS A3.0 "Definition of Standard Welding Terms" [29], ISO 5817 "Welding Joints" [30], AWS D1.1 "Welding Specification for Steel Structures" [31], and ASME VIII "Boiler and Pressure Vessel Manufacturing Code" standards [32]. A schematic of the four-parameter measurement requirements for butt welds is shown in Fig 3. The parameter corresponding to weld width is defined as the distance between the two weld toes according to the AWS A3.0 "Definition of Standard Welding Terms," which is the junction between the weld surface and base metal. The weld reinforcement hre is a parameter in which the weld metal exceeds the height of the fillet welding groove. The weld parameter undercut hun_cut denotes the size of grooves or depressions produced along with the base metal of the weld toe owing to improper selection of welding parameters or incorrect operation methods. Fig 3(A) shows the definitions of weld width lwidth, weld reinforcement hre, and weld undercut hun_cut in the standard. Weld misalignment is defined by the ASME VIII "Boiler and Pressure Vessel Manufacturing Code" standard as the phenomenon of dislocation and unevenness due to the deformation of the welding deviation and other factors during welding. The parameter weld misalignment halign denotes the size and amplitude of the misalignment as shown in Fig 3(B). Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 3. Schematic diagram of parameter definition of butt weld 4. (a) Definition of weld reinforcement, width, and undercut. (b) Definition of weld misalignment. https://doi.org/10.1371/journal.pone.0267743.g003 The existence of weld defects alone, which is a measurement index under the ideal weld condition, is an example of each weld appearance parameter defined and detailed in the standard state. An initially formed weld can exhibit multiple coexisting defects. In the case of weld defects (such as wrong under the influence of edge volume), the definition of the welding seam parameter measurement index in the standard example diagram is no longer applicable. Hence, the appearance parameters of welds at the cross-sectional position of the welds under the conditions of normal welds without defects, single-defect welds, and multi-defect welds are discussed in this study with reference to measurement diagrams of weld appearance parameters in relevant standards. Fig 4 shows a schematic of the weld parameter measurement indicators for various weld shapes. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 4. Schematic diagrams of welding seam parameter measurement index under different welding seam shapes. (a) Normal weld. (b) Welds with the wrong sides. (c) Welds with single undercut and misalignment. (d) Welds with double undercut and misalignment. https://doi.org/10.1371/journal.pone.0267743.g004 The parameter reinforcement feature point Pre was determined as the highest point of the welding area of the profile curve. The parameter width feature points and are determined as the intersection of the welding curve and the base metal curve, i.e., the welding toes on both sides, as shown in Fig 4(A) and 4(B). The misalignment feature points and coincide with the width feature points in the cases of no defect parameters and single misalignment defects. However, given that the weld toe on the undercut side disappears in the case of a defective undercut, the parameter width feature point is modified to the junction point of the undercut depression curve and welding curve. The misalignment feature point at this time is the intersection point between the undercut curve and base material curve as shown in Fig 4(C) and 4(D), respectively. The parameter feature points in normal welds are extreme points or corner points, as shown in the above feature point selection example, and the traditional curve extreme and reciprocal analysis methods can complete the feature point extraction task. The characteristic points of the cross-section curve width are weak owing to undercut defects. It is difficult to simultaneously extract all of the characteristic points based on traditional curve analysis methods, and to date, no scholars have proposed a method for simultaneously extracting all of the appearance parameters. Consequently, in terms of image processing feature analysis, in this study, we employed deep learning image semantic segmentation methods to classify weld defects and extracted four parameter feature points of welds in laser curve images. Design of image feature point extraction network based on CNN The structural diagram of the coding-decoding-free image feature point extraction network (EDE-net), proposed for this study, is depicted in Fig 5. The input of this network is the preprocessed laser profile image of the weld, and the output of the network is the pixel position of the parameter feature point. The coding part of the network is composed of the CNN backbone. First, the CNN backbone output feature map outputs an n-dimensional feature map after upsampling at the branch of the decoding part. Then, the location of the feature point, roughly extracted by the network, can be expressed as (i = 1,…, n). On branch two of the decoding part, the output feature map with dimensions of 2n is sampled and processed, and feature point correction information is finely extracted. Finally, the position of the parameter feature point in the input image, namely , can be obtained by combining the information of the two output feature maps and . Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 5. Network structure diagram of EDE-net. https://doi.org/10.1371/journal.pone.0267743.g005 The EDE-net coding structure output feature map up-sampling methods include bilinear interpolation, deconvolution, and depooling methods. If the image up-sampling multiple is kupsample, then the input scale is ih×iw×3 image I, the up-sampling output scale is ih*kupsample×iw*kupsample×3 image O, and the pixel position (kpx, kpy) in the image O pixel value is sampled and mapped to the pixel position (px, py) of the image I pixel value . The up-sampled image O after bilinear interpolation is determined as follows: (2) where , , and 〈⋅〉 are round-down calculations. Based on the formula, it is evident that the bilinear interpolation upsampling method should traverse each pixel, which features a large amount of calculation and slower running speed. The deconvolution upsampling mechanism is shown in Fig 6. The adjacent and surrounding pixels in Image I were interpolated and filled with pixels (usually the pixel value was 0). To obtain the inverse convolution output image O, the convolution kernel and supplemented images were used for the convolution calculation. The pooling kernel was set according to the maximum and average pooling methods, and the de-pooling up-sampling mechanism was the same as the deconvolution up-sampling image supplement method. The up-sampling magnification of the deconvolution and de-pooling methods was lower, but the amount of calculation was less. In deconvolution, the convolution kernel can participate in the entire network training and update the weights, whereas the weights of the pooling kernel cannot be changed. Therefore, EDE-net uses the deconvolution calculation as the up-sampling method. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 6. Deconvolution up-sampling mechanism diagram. https://doi.org/10.1371/journal.pone.0267743.g006 The EDE-net network branch is the task of rough extraction of feature point positions. Based on the concept of completely convolutional network semantic segmentation, the deep feature map output by the convolutional network was processed via a single layer with a stride of 2 and a 3 × 3 deconvolution kernel with a number of feature points n. The output dimension of the feature map was reduced to n, with each dimension corresponding to the approximate position information of each feature point in the input weld centerline image. Each pixel value is the probability that the feature point is at that position. The theoretical output result, M, of the feature extraction module branch is shown in Fig 7. If the theoretical regression object M is the only single-point position of the characteristic point in the laser centerline image of the weld, then the position outside the characteristic point in M is a negative sample. If the positive and negative samples are seriously out of balance, overfitting can occur during network training. Therefore, the feature point distance threshold T was introduced, and the pixel distance from the feature point position was set as positive samples within T. This in turn solved the imbalance problem. The theoretical output feature map M can be obtained as follows: (3) Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 7. Theoretical output results of EDE-net branch 1. https://doi.org/10.1371/journal.pone.0267743.g007 The distances between the feature point and background within the threshold T can be treated as a binary classification task. Hence, the focal loss of the feature classification task can be utilized as the loss function as shown in the output feature map of the branch-network theory. The adjustment parameters α and γ are compatible with the feature classification task Focal-loss [33]. Therefore, the branch-loss function LM is as follows: (4) where matrix S elements are all one. EDE-net branch 2 is a feature point position–correction task. The convolutional network’s deep feature map is processed through a single layer with a stride of 2, scale of 3×3, and dimension of 2n deconvolution kernels. This yields a 2n-dimensional feature map . The theoretical output results N of EDE-net network branch 2 are shown in Fig 8. Based on the approximate location of M feature points, the branch two-theory output feature map N adds a feature point location correction value. This implies that the position value of the corresponding element of N2i-1 is the true value of Mi, and N2i is the pixel difference between the theoretical feature points and X-Y-axis of the image coordinate system at the element’s position. Therefore, the theoretical outputs N2i-1 and N2i branch 2 are defined as follows: (5) where c denotes the ratio of the input image scale to the output feature map scale of upsampling. The feature point position correction task is a numerical regression task. Therefore, the Huber loss is used to establish the loss function: (6) Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 8. Theoretical output results of EDE-net branch 2. https://doi.org/10.1371/journal.pone.0267743.g008 Then, the EDE-net network loss function can be obtained as LEDE = LN+LM. Image data enhancement method for pressure vessel weld surface profile based on third-order NURBS curve A CNN requires a large number of datasets for training, and the data capacity of the training set is directly related to the CNN’s ability to extract feature points. The conventional approach of producing training sets involves acquiring contour images with an active vision imaging device and manually labeling the location coordinates of the feature points in each image. To avoid this, in the current study, we provide a surface parameter simulation method with the coexistence of multiple defects on the weld surface of the pressure vessel. This in turn allows variation of the types of weld parameters, number of feature points, and parameter values, and thereby, effectively reduces the number of training set collection tasks. A simulation diagram of the normal weld curve is shown in Fig 9. The parent material area of the curve in the image coordinate system (Fmetal(x,y)) is expressed as follows: (7) where Setwidth denotes the weld width parameter, Rstand denotes the diameter of the pressure vessel cylinder, Lpic denotes the imaging area length, and Wpic denotes the width. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 9. Simulation of normal weld curve. https://doi.org/10.1371/journal.pone.0267743.g009 The simulation feature point position of the normal weld contour curve is presented in Table 1 when the simulation reinforcement is Setre. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 1. Simulation feature point position of normal weld contour curve. https://doi.org/10.1371/journal.pone.0267743.t001 A non-uniform rational B-spline (NURBS) curve [34] simulation weld zone curve was constructed based on the aforementioned three points: Pre, , and . Assuming that the control point of the third-order NURBS curve is k = 3, di = [x,y]T, wi is the weight factor of the curve control point, and w0, w2>0, the remaining wi≥0, Fweld(u) can be expressed as follows: (8) (9) Furthermore, di information can be inversely calculated with three NURBS curve data points: Pre, , and . Let the node vectorbe when the curve is opened and the control point be n = m+k-1 = 5 such that the curve passes through the first and last control points. The node vector should exhibit k+1 repeatability, and it is set using the accumulation chord length method. This implies that u0 = u1 + u2 = u3 = 0, u5 = u6 + u7 = u8 = 1, , and the setting of the weight factor wi of the curve control point are as follows: (10) The beginning and end points of the curve control points were consistent with those of the data points throughout the reverse solution procedure. The connection points of the curved segment correspond to the nodes of the NURBS curve-defining domain. Therefore, if the data point qi corresponds to the node value ui+3(i = 0,1,2), then the solution condition for di is as follows: (11) To complete the control-point solution provided by the tangent vector boundary conditions, the following two equations must be included: (12) By combining the above equations, we can complete the coordinate position solution di and construct curve Fweld(u). Defect parameter undercut simulation. If the undercut width, depth, and offset are δun_cut, Setun_cut, and Δun_cut, respectively, then they indicate that the undercut feature point and width feature point are separated in the X-axis direction. Then, the value range of x in the curve of of the parent metal area on the right is modified to . The positions of the simulated feature points of the undercut weld profile curve with defect parameters are listed in Table 2. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 2. Location of the feature point in the simulation of the undercut weld profile curve with the defect parameter. https://doi.org/10.1371/journal.pone.0267743.t002 To perform the single-defect undercut weld contour curve simulation, five-point coordinates Pre, , , Pun_cut, and Pmis_align can be used as the NURBS curve data points as shown in Fig 10. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 10. Simulation of weld profile curve with single defect parameter undercut Setun_cut. https://doi.org/10.1371/journal.pone.0267743.g010 Defect parameter misalignment simulation. In this case, the width feature points on both sides of the weld coincide with the deviation feature point, and curve of the base material area on the left is consistent with the simulation of the normal weld. The curve of the base material area on the right by Setmis_align along the Y-axis of the image coordinate system and on the right side of the weld is as follows: (13) The single-defect misaligned weld profile curve simulation was performed using the coordinates Pre, , and . The three points were used as NURBS curve data points as shown in Fig 11. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 11. Simulation of weld profile curve with single defect parameter misalignment Setalign. https://doi.org/10.1371/journal.pone.0267743.g011 Defect parameter undercut simulation. If the undercut width, depth, and offset are δun_cut, Setun_cut, and Δun_cut, respectively, then they indicate that the undercut feature point and width feature point are separated in the X-axis direction. Then, the value range of x in the curve of of the parent metal area on the right is modified to . The positions of the simulated feature points of the undercut weld profile curve with defect parameters are listed in Table 2. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 2. Location of the feature point in the simulation of the undercut weld profile curve with the defect parameter. https://doi.org/10.1371/journal.pone.0267743.t002 To perform the single-defect undercut weld contour curve simulation, five-point coordinates Pre, , , Pun_cut, and Pmis_align can be used as the NURBS curve data points as shown in Fig 10. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 10. Simulation of weld profile curve with single defect parameter undercut Setun_cut. https://doi.org/10.1371/journal.pone.0267743.g010 Defect parameter misalignment simulation. In this case, the width feature points on both sides of the weld coincide with the deviation feature point, and curve of the base material area on the left is consistent with the simulation of the normal weld. The curve of the base material area on the right by Setmis_align along the Y-axis of the image coordinate system and on the right side of the weld is as follows: (13) The single-defect misaligned weld profile curve simulation was performed using the coordinates Pre, , and . The three points were used as NURBS curve data points as shown in Fig 11. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 11. Simulation of weld profile curve with single defect parameter misalignment Setalign. https://doi.org/10.1371/journal.pone.0267743.g011 Results Experiment on EDE-net performance of different backbone networks Test CNN backbone selection. The accuracy of the overall network feature point extraction is affected by the feature extraction performance of the CNN backbone network in the image feature point extraction network based on encoding-decoding. Among the common CNN networks, including AlexNet [35], VGG [36], Res-Net [37], and Inception [38], AlexNet and VGG networks are linear and branchless structures. When the network layers are deeper, they are more difficult to develop, and common problems, such as gradient disappearance and explosion, can occur. To solve the aforementioned problems, Res-Net introduces the residual unit residual in the convolutional layer and realizes identity mapping by constructing direct connections. The network does not degrade as the depth of the network convolutional layer increases owing to the continued stacking. The inception network proposed a structure to obtain close feature extraction capabilities with fewer network layers. The feature map was produced using several convolutions and pooling kernels, and the results were stacked to reduce the number of layers in the network. Resnet50, Resnet101, Resnet152, InceptionV4, and Inception-Res-net were used as the backbones of the CNN based on the encoding-decoding deep feature-point extraction network. Table 3 lists the tested CNN network information, where Top1 accuracy is the CNN structure in the image net image classification result. This can be used as the performance level of the network. The trainable parameters indicate the complexity of the network training owing to the involvement of many parameters, such as the standard of performance, higher accuracy, and better network performance. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 3. Tested CNN network information. https://doi.org/10.1371/journal.pone.0267743.t003 Selection of training sets. Based on the existence of the undercut defect parameter, the number of feature points of the weld parameters in images are 3, 5, and 7, namely, datasets D3, D5, and D7. To collect boiler butt type B and pressure pipeline type A welds with diameters of 1300 and 255 mm, respectively, a Keyence LJV-7080 sensor was used. The set of surface contour points comprised the training and test sets. Fig 12 shows the simulation-generated contour image effect and previously acquired contour maps. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 12. Acquisition of contour images to conduct simulations to generate contour images. (a) No defect parameter welds profile data set D3. (b) Weld profile data set D3 for parameter simulation of defects with a single misalignment error. (c) Weld profile data set D5 for parameter simulation of single undercut defect. (d) Weld profile data set D5 for parameter simulation of undercut and misalignment defect parameters. (The undercut is on the misalignment side.) (e) Weld profile data set D5 for parameter simulation of undercut and misalignment defect parameters. (The undercut is not on the misalignment side.) (f) Weld profile data set D7 with parameter simulation of undercut defects on both sides. (g) Weld profile data set D7 for simulation of the misalignment and undercut parameters on both sides (h) Actual collection data set D3. (j) Actual collection data set D5. (j) Actual collection data set D7. https://doi.org/10.1371/journal.pone.0267743.g012 To improve the data from the aforementioned photographs, an affine transformation was adopted. Let the image size be WD×HD, image rotation angle be β, and image scaling size be hscale. Then, the affine matrix is determined as Wwarp as follows: (14) One hundred physical collection datasets D3, D5, D7, and 100 simulation-generated datasets D3, D5, and D7 were selected, with a random rotation angle β = 0–30° and random scaling size hscale = 0.5~0.8. Table 4 contains the training of hyperparameter information, and the network training loss function is used as the evaluation index to determine the ideal CNN structure. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 4. Training hyper-parameter information. https://doi.org/10.1371/journal.pone.0267743.t004 Fig 13(A)–13(C) show the trend charts of the network loss function extracted from different CNN structures based on the encoding-decoding depth feature points. In the figure, it can be observed that (a) the CNN network structure is more difficult to train without fine-tuning migration, resulting loss does not converge, and network model fails. As the number of training steps increased, the training difficulty of the CNN structure after fine-tuning dramatically decreased, and the ultimate convergence effect improved. (b) Network training is more effective when the layers of the same CNN structure are much deeper. (c) As the performance improves, the accuracy rate of the CNN structure increases. For instance, the Inception-Res-net network structure exhibited a Top1 accuracy of 80.4. The training loss function is reduced when compared to that of the ResNet series CNN. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 13. Loss function trend chart of different CNN backbones based on the encoding–decoding depth feature point extraction network. (a) Comparison of transfer learning and non-transfer learning. (b) Comparison of loss functions of different network backbones. (c) Final convergence of the network. https://doi.org/10.1371/journal.pone.0267743.g013 Experiment on the actual pressure vessel weld of measurement results The experiment was initiated by selecting 150 physical collection datasets and 60 simulation datasets D3, D5, and D7 with a random rotation angle β = 0–30° and random scaling size hscale = 0.5–0.8. Among the datasets, 90 physical collection datasets and 60 physical collection training sets with 60simulation-generated training sets were used to train DeepLabCut, HR-net, and EDE-net (Inception Res-net), respectively. Finally, the remaining 60 physical collection datasets D3, D5, and D7 were used as the test set, and the accuracy of the parameter feature point extraction was used as the network performance evaluation index. All the parameter feature points were regarded as feature points of the same nature if the Euclidean distance between the artificially marked parameter feature point and network output feature point was used as the evaluation index. The parameter reinforcement and undercut feature points are extreme-value natural points. The characteristic reflection is stronger when compared to the parameter width and natural point of the wrong edge inflection point. Given that the feature points of the weld parameters differ in marking and extraction difficulty, using the Euclidean distance as the evaluation index is no longer appropriate. Consequently, the object key-point similarity (OKS) weighted Euclidean distance was introduced as the characteristic evaluation index of the network output parameters. It is defined as follows: (15) where denotes the presence of an undercut in the weld image, denotes the square of the Euclidean distance between the artificially marked parameter feature point and network output feature point, denotes the pixel area of the weld curve in the image, denotes the artificially marked feature point and real position deviation information of the feature point, and is replaced by the mean square error in the numerical calculation. Currently, average precision (AP) is used to evaluate the same task in a deep network and performance evaluation index of different network structures. The feature points of the OKS and AP indicators contain AP evaluation methods. The correlation between the tasks is as follows: (16) where Ts denotes the OKS threshold. The network output feature points deviate from the actual feature points. The accuracy of the feature point extraction can be improved further if the network output feature points are returned to the laser curve using the following equation. where (xout−yout) denotes the coordinate of the characteristic point of the welded seam output by the network, (xCor,yCor) denotes the coordinate of the characteristic point of welding on the laser line after correction, and (xR−yR) denotes the coordinate of any point on the contour line. The network results are presented in Table 5. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 5. AP results of feature points extracted from different CNNs. https://doi.org/10.1371/journal.pone.0267743.t005 As shown in Table 5, EDE-net (Inception-Resnet) before and after correction of AP, AP0.5, AP07 is better than Deep Lab-Cut,and HR-net network, and these corrections can significantly improve the feature point. The extraction accuracies were similar for the two training sets. However, the training method for actual measurements and simulations can effectively reduce the data-collection workload. Table 6 lists the absolute error and standard deviation data for all the parameter feature points. The measurement resolutions of the X-axis and Y-axis of the sensor were 0.005 and 0.001 mm, respectively. If we select a 99.73% confidence interval [μ-3σ, μ+3σ], then the confidence interval of the reinforcement feature point extraction error is [-10.958, 10.8629], and the theoretical measurement accuracy can be as high as 0.011 mm. Similarly, for other parameters, in which the measurement accuracies are mentioned, the confidence interval of the width feature point extraction error was [-9.182, 13.069], and the theoretical measurement accuracy was as high as 0.065 mm. The confidence interval of the extraction error of the undercut feature points was [-10.245, 8.885], and the theoretical measurement accuracy was as high as 0.011 mm. The confidence interval of the error of the feature point extraction of the amount of error was [-11.833, 11.833], and the theoretical measurement accuracy was as high as 0.012 mm. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 6. Feature point extraction error information. https://doi.org/10.1371/journal.pone.0267743.t006 Experiment on EDE-net performance of different backbone networks Test CNN backbone selection. The accuracy of the overall network feature point extraction is affected by the feature extraction performance of the CNN backbone network in the image feature point extraction network based on encoding-decoding. Among the common CNN networks, including AlexNet [35], VGG [36], Res-Net [37], and Inception [38], AlexNet and VGG networks are linear and branchless structures. When the network layers are deeper, they are more difficult to develop, and common problems, such as gradient disappearance and explosion, can occur. To solve the aforementioned problems, Res-Net introduces the residual unit residual in the convolutional layer and realizes identity mapping by constructing direct connections. The network does not degrade as the depth of the network convolutional layer increases owing to the continued stacking. The inception network proposed a structure to obtain close feature extraction capabilities with fewer network layers. The feature map was produced using several convolutions and pooling kernels, and the results were stacked to reduce the number of layers in the network. Resnet50, Resnet101, Resnet152, InceptionV4, and Inception-Res-net were used as the backbones of the CNN based on the encoding-decoding deep feature-point extraction network. Table 3 lists the tested CNN network information, where Top1 accuracy is the CNN structure in the image net image classification result. This can be used as the performance level of the network. The trainable parameters indicate the complexity of the network training owing to the involvement of many parameters, such as the standard of performance, higher accuracy, and better network performance. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 3. Tested CNN network information. https://doi.org/10.1371/journal.pone.0267743.t003 Selection of training sets. Based on the existence of the undercut defect parameter, the number of feature points of the weld parameters in images are 3, 5, and 7, namely, datasets D3, D5, and D7. To collect boiler butt type B and pressure pipeline type A welds with diameters of 1300 and 255 mm, respectively, a Keyence LJV-7080 sensor was used. The set of surface contour points comprised the training and test sets. Fig 12 shows the simulation-generated contour image effect and previously acquired contour maps. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 12. Acquisition of contour images to conduct simulations to generate contour images. (a) No defect parameter welds profile data set D3. (b) Weld profile data set D3 for parameter simulation of defects with a single misalignment error. (c) Weld profile data set D5 for parameter simulation of single undercut defect. (d) Weld profile data set D5 for parameter simulation of undercut and misalignment defect parameters. (The undercut is on the misalignment side.) (e) Weld profile data set D5 for parameter simulation of undercut and misalignment defect parameters. (The undercut is not on the misalignment side.) (f) Weld profile data set D7 with parameter simulation of undercut defects on both sides. (g) Weld profile data set D7 for simulation of the misalignment and undercut parameters on both sides (h) Actual collection data set D3. (j) Actual collection data set D5. (j) Actual collection data set D7. https://doi.org/10.1371/journal.pone.0267743.g012 To improve the data from the aforementioned photographs, an affine transformation was adopted. Let the image size be WD×HD, image rotation angle be β, and image scaling size be hscale. Then, the affine matrix is determined as Wwarp as follows: (14) One hundred physical collection datasets D3, D5, D7, and 100 simulation-generated datasets D3, D5, and D7 were selected, with a random rotation angle β = 0–30° and random scaling size hscale = 0.5~0.8. Table 4 contains the training of hyperparameter information, and the network training loss function is used as the evaluation index to determine the ideal CNN structure. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 4. Training hyper-parameter information. https://doi.org/10.1371/journal.pone.0267743.t004 Fig 13(A)–13(C) show the trend charts of the network loss function extracted from different CNN structures based on the encoding-decoding depth feature points. In the figure, it can be observed that (a) the CNN network structure is more difficult to train without fine-tuning migration, resulting loss does not converge, and network model fails. As the number of training steps increased, the training difficulty of the CNN structure after fine-tuning dramatically decreased, and the ultimate convergence effect improved. (b) Network training is more effective when the layers of the same CNN structure are much deeper. (c) As the performance improves, the accuracy rate of the CNN structure increases. For instance, the Inception-Res-net network structure exhibited a Top1 accuracy of 80.4. The training loss function is reduced when compared to that of the ResNet series CNN. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 13. Loss function trend chart of different CNN backbones based on the encoding–decoding depth feature point extraction network. (a) Comparison of transfer learning and non-transfer learning. (b) Comparison of loss functions of different network backbones. (c) Final convergence of the network. https://doi.org/10.1371/journal.pone.0267743.g013 Test CNN backbone selection. The accuracy of the overall network feature point extraction is affected by the feature extraction performance of the CNN backbone network in the image feature point extraction network based on encoding-decoding. Among the common CNN networks, including AlexNet [35], VGG [36], Res-Net [37], and Inception [38], AlexNet and VGG networks are linear and branchless structures. When the network layers are deeper, they are more difficult to develop, and common problems, such as gradient disappearance and explosion, can occur. To solve the aforementioned problems, Res-Net introduces the residual unit residual in the convolutional layer and realizes identity mapping by constructing direct connections. The network does not degrade as the depth of the network convolutional layer increases owing to the continued stacking. The inception network proposed a structure to obtain close feature extraction capabilities with fewer network layers. The feature map was produced using several convolutions and pooling kernels, and the results were stacked to reduce the number of layers in the network. Resnet50, Resnet101, Resnet152, InceptionV4, and Inception-Res-net were used as the backbones of the CNN based on the encoding-decoding deep feature-point extraction network. Table 3 lists the tested CNN network information, where Top1 accuracy is the CNN structure in the image net image classification result. This can be used as the performance level of the network. The trainable parameters indicate the complexity of the network training owing to the involvement of many parameters, such as the standard of performance, higher accuracy, and better network performance. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 3. Tested CNN network information. https://doi.org/10.1371/journal.pone.0267743.t003 Selection of training sets. Based on the existence of the undercut defect parameter, the number of feature points of the weld parameters in images are 3, 5, and 7, namely, datasets D3, D5, and D7. To collect boiler butt type B and pressure pipeline type A welds with diameters of 1300 and 255 mm, respectively, a Keyence LJV-7080 sensor was used. The set of surface contour points comprised the training and test sets. Fig 12 shows the simulation-generated contour image effect and previously acquired contour maps. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 12. Acquisition of contour images to conduct simulations to generate contour images. (a) No defect parameter welds profile data set D3. (b) Weld profile data set D3 for parameter simulation of defects with a single misalignment error. (c) Weld profile data set D5 for parameter simulation of single undercut defect. (d) Weld profile data set D5 for parameter simulation of undercut and misalignment defect parameters. (The undercut is on the misalignment side.) (e) Weld profile data set D5 for parameter simulation of undercut and misalignment defect parameters. (The undercut is not on the misalignment side.) (f) Weld profile data set D7 with parameter simulation of undercut defects on both sides. (g) Weld profile data set D7 for simulation of the misalignment and undercut parameters on both sides (h) Actual collection data set D3. (j) Actual collection data set D5. (j) Actual collection data set D7. https://doi.org/10.1371/journal.pone.0267743.g012 To improve the data from the aforementioned photographs, an affine transformation was adopted. Let the image size be WD×HD, image rotation angle be β, and image scaling size be hscale. Then, the affine matrix is determined as Wwarp as follows: (14) One hundred physical collection datasets D3, D5, D7, and 100 simulation-generated datasets D3, D5, and D7 were selected, with a random rotation angle β = 0–30° and random scaling size hscale = 0.5~0.8. Table 4 contains the training of hyperparameter information, and the network training loss function is used as the evaluation index to determine the ideal CNN structure. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 4. Training hyper-parameter information. https://doi.org/10.1371/journal.pone.0267743.t004 Fig 13(A)–13(C) show the trend charts of the network loss function extracted from different CNN structures based on the encoding-decoding depth feature points. In the figure, it can be observed that (a) the CNN network structure is more difficult to train without fine-tuning migration, resulting loss does not converge, and network model fails. As the number of training steps increased, the training difficulty of the CNN structure after fine-tuning dramatically decreased, and the ultimate convergence effect improved. (b) Network training is more effective when the layers of the same CNN structure are much deeper. (c) As the performance improves, the accuracy rate of the CNN structure increases. For instance, the Inception-Res-net network structure exhibited a Top1 accuracy of 80.4. The training loss function is reduced when compared to that of the ResNet series CNN. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 13. Loss function trend chart of different CNN backbones based on the encoding–decoding depth feature point extraction network. (a) Comparison of transfer learning and non-transfer learning. (b) Comparison of loss functions of different network backbones. (c) Final convergence of the network. https://doi.org/10.1371/journal.pone.0267743.g013 Experiment on the actual pressure vessel weld of measurement results The experiment was initiated by selecting 150 physical collection datasets and 60 simulation datasets D3, D5, and D7 with a random rotation angle β = 0–30° and random scaling size hscale = 0.5–0.8. Among the datasets, 90 physical collection datasets and 60 physical collection training sets with 60simulation-generated training sets were used to train DeepLabCut, HR-net, and EDE-net (Inception Res-net), respectively. Finally, the remaining 60 physical collection datasets D3, D5, and D7 were used as the test set, and the accuracy of the parameter feature point extraction was used as the network performance evaluation index. All the parameter feature points were regarded as feature points of the same nature if the Euclidean distance between the artificially marked parameter feature point and network output feature point was used as the evaluation index. The parameter reinforcement and undercut feature points are extreme-value natural points. The characteristic reflection is stronger when compared to the parameter width and natural point of the wrong edge inflection point. Given that the feature points of the weld parameters differ in marking and extraction difficulty, using the Euclidean distance as the evaluation index is no longer appropriate. Consequently, the object key-point similarity (OKS) weighted Euclidean distance was introduced as the characteristic evaluation index of the network output parameters. It is defined as follows: (15) where denotes the presence of an undercut in the weld image, denotes the square of the Euclidean distance between the artificially marked parameter feature point and network output feature point, denotes the pixel area of the weld curve in the image, denotes the artificially marked feature point and real position deviation information of the feature point, and is replaced by the mean square error in the numerical calculation. Currently, average precision (AP) is used to evaluate the same task in a deep network and performance evaluation index of different network structures. The feature points of the OKS and AP indicators contain AP evaluation methods. The correlation between the tasks is as follows: (16) where Ts denotes the OKS threshold. The network output feature points deviate from the actual feature points. The accuracy of the feature point extraction can be improved further if the network output feature points are returned to the laser curve using the following equation. where (xout−yout) denotes the coordinate of the characteristic point of the welded seam output by the network, (xCor,yCor) denotes the coordinate of the characteristic point of welding on the laser line after correction, and (xR−yR) denotes the coordinate of any point on the contour line. The network results are presented in Table 5. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 5. AP results of feature points extracted from different CNNs. https://doi.org/10.1371/journal.pone.0267743.t005 As shown in Table 5, EDE-net (Inception-Resnet) before and after correction of AP, AP0.5, AP07 is better than Deep Lab-Cut,and HR-net network, and these corrections can significantly improve the feature point. The extraction accuracies were similar for the two training sets. However, the training method for actual measurements and simulations can effectively reduce the data-collection workload. Table 6 lists the absolute error and standard deviation data for all the parameter feature points. The measurement resolutions of the X-axis and Y-axis of the sensor were 0.005 and 0.001 mm, respectively. If we select a 99.73% confidence interval [μ-3σ, μ+3σ], then the confidence interval of the reinforcement feature point extraction error is [-10.958, 10.8629], and the theoretical measurement accuracy can be as high as 0.011 mm. Similarly, for other parameters, in which the measurement accuracies are mentioned, the confidence interval of the width feature point extraction error was [-9.182, 13.069], and the theoretical measurement accuracy was as high as 0.065 mm. The confidence interval of the extraction error of the undercut feature points was [-10.245, 8.885], and the theoretical measurement accuracy was as high as 0.011 mm. The confidence interval of the error of the feature point extraction of the amount of error was [-11.833, 11.833], and the theoretical measurement accuracy was as high as 0.012 mm. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 6. Feature point extraction error information. https://doi.org/10.1371/journal.pone.0267743.t006 Discussion The measurement indicators for the surface parameters of a single weld specified in the relevant verification standards for pressure vessels cannot be effectively used in the measurement of an actual weld surface profile where multiple defects coexist. In this study, the appearance characteristics of the weld surface parameters were measured in the form of image feature points, and algorithm design ideas of the regression image from the feature point coordinates were proposed using the excellent nonlinear mapping ability of CNN networks. An image feature point extraction network based on deep learning was designed to simultaneously extract all the parameter feature points. For the network training measurement, a method based on the 3rd NURBS curve simulation of a realistic weld surface profile was proposed to enhance the training data. Finally, an experimental device was designed to collect the surface data of A and B butt welds, and the deep learning network proposed in this study was compared with the DeepLabCut and HR-net methods under different training sets. The results show that the difference between the training output of the training set network after data enhancement and training set network output AP, which is completely measured, is low. However, the data enhancement method can effectively reduce the workload of sample collection, and the theoretical accuracy of parameter measurement can be realized within 0.065 mm.
TI - Pressure vessel-oriented visual inspection method based on deep learning
JF - PLoS ONE
DO - 10.1371/journal.pone.0267743
DA - 2022-05-02
UR - https://www.deepdyve.com/lp/public-library-of-science-plos-journal/pressure-vessel-oriented-visual-inspection-method-based-on-deep-SdLpsGLpFg
SP - e0267743
VL - 17
IS - 5
DP - DeepDyve
ER -