6DOF Simulation of a Floating Structure Model Using a Single Video

Abstract

This paper purposes on stimulating the dynamic behavior of a floating structure model with the image processing and the close-range photogrammetry, instead of the contact sensors. Previously, the movement of structures was presented by the exterior orientation estimation from a single camera following the space resection. The inverse resection yields to 6 orientation parameters of the floating structure, with respect to the camera coordinates system. The single camera solution of interest in applications is characterized by the restriction in terms of costs, unfavorable observation conditions, or synchronization demands when using multiple cameras. This paper discusses the theoretical determinations of camera exterior orientation by using the least squares adjustment, applied of the values from the DLT (Direct Linear Transformation) and the photogrammetric resection. This proposed method is applied to monitor motions of a floating model. The results of 6DOF (Six Degrees of Freedom) from the inverse resection were signified that applying appropriate initial values from DLT in the least square adjustment is effective in obtaining precise exterior orientation parameters. Therefore, the proposed method can be concluded as an efficient solution to simulate movements of the floating structure.

1. Introduction

There are two methods generally used to measure the displacement or vibration of structures: the contact type, and the non-contact type. For the contact type, the mode requires mounting of sensors to a rigid body to acquire the motion signal, but this is always challenging in field tests. Meanwhile, the non-contact method requires relatively expensive devices, and in many cases has a limited measurement distance (Lee and Rhee, 2013).

Photogrammetry is a technology for measuring 2D or 3D shapes from photos. Golparvar-Fard et al. (2011) successfully researched “structure from motion,” by implementing an image processing technique, that of based on the point cloud application. In this approach, however, the model object must be in stationary status during the process, so it is not suitable for modeling a moving object in real time. With the subject of measuring the movement of an object, Jeon et al. (2010) applied image processing to measure the vibrations of a beam with a camera, but the obtained result was limited to 2D measurement, and it was necessary to use a high-speed camera.

The Pure photogrammetric solutions for the object tracking upon the two or more synchronized cameras have advantages on providing the absolute object coordinates of targets. Ozbek et al. (2010) monitored a large wind turbine by use of a photogrammetry method that required a PONTOS system, consisting of four CCD (Charge Coupled Device) cameras. Fathi and Brilakis (2013) utilized two video cameras for measuring dimensions of a roof base on the structure following motions with the necessity of matching time stamp of each video frame, recorded from each camera.

The estimation of camera orientation by space resection is a common task in photogrammetry, regarding to the indirect determination of position and orientation parameters. The exterior orientation of a camera with respect to the three-dimensional coordinates system can be obtained by linear or non-linear solutions, based on measurements of image coordinates of reference points. The linear resection method is based on DLT (Abdel-Aziz and Karara, 1971; Marzan, 1975), or projective geometry (Harley and Zisserman, 2003), while the linear functions require a minimum of six control points, without needs of approximate values of unknown orientation parameters. In fact, the most accurate solutions are required the use of redundant observations and the least square techniques (Mikhail et al., 2001). The best-known non-linear method is the space resection, which is established upon the least square solution of linearized collinearity equations, so as this method requires a minimum of three reference points, and approximate values for six unknown parameters.

In general, the object tracking is a challenging problem, as well as an important task within the field of a computer vision, especially for applications that require the location or shape of an object in every frame, such as security and surveillance, video communication, traffic management, human-computer interaction, and medical imaging. The developments of computer science, achieved the availability of high quality and inexpensive video cameras and increased demands for automated video analysis, have generated a great deal of interest in the object movements tracking (Yilmaz et al., 2006). In this research, the problem of automatic video tracking following a moving target of floating object over the video sequence was introduced.

The study mainly purposed on solving the problem for measuring the dynamic behavior of floating structures with an inexpensive video camera. Firstly, the methodology of image-based motion measuring was performed in Section 2. At next, the experimental verification to investigate the algorithms and analyses results, obtained from the regular wave test, were shown in Section 3, and then the dynamic motion of floating model was successfully obtained through 6DOF simulation using a single video.

 

2. Movement Measurement by Using a Video

Fig. 1 shows detailed procedures of the proposed method to measure movements of a floating structure with the video in images processing and close-range photogrammetry. The following approaches were implemented to calculate positions and orientations of the camera, and 6DOF of the structure.

(1) Image processing-based tracking reference points (2) Space resection using DLT (3) Adjustment using the least squares method (4) 6DOF by inverse resection

Fig. 1.Movement measurement procedure by a camera

2.1. Image processing for tracking of reference points

A series of images, obtained from the camera, was conversed its position of interest that allows structure displacement measurements; thus, tracking the target is the main task in this research. Specifically, the floating structure was controlled by a set of points, namely targeted points, during experiments. In this paper, the tracking function was performed by the KLT (Kanade-Lucas- Tomasi) feature-tracking algorithm (Tomasi and Kanade, 1991; Shi and Tomasi, 1994). Those point detectors were applied to find points of interest in a small targeted region, which have expressive textures in each respective location. The KLT tracker evaluated the quality of tracked patches, by computing the affine transformation between the corresponding patches in consecutive frames (Yilmaz et al., 2006). The well-known Matlab from the MathWorks (2014) is appropriate software to utilize for this purpose. Accordingly, the consequent target tracking was described by displacements of points in the image coordinates system.

2.2. Movement observation of floating structure using close-range photogrammetry

2.2.1 Initial values of exterior orientation

The methodology proposed in this research is based on the theory of space resection. The collinearity equations given in Eq. (1) constitute the mathematical foundation of photogrammetry. A set of reference points was used by their coordinate relationship in an object system and image coordinate system, to calculate the EOPs (Exterior Orientation Parameters) of the camera, including the perspective center position (XL, YL, ZL), and three rotation angles (ω, φ, κ ) with respect to the x, y and z-axes (Mikhail et al., 2001).

where xa, ya: Coordinates of point in image coordinates, x0, y0: Coordinates of the principal point in the image plane, XA, YA, ZA: Coordinates of the object point, XL, YL, ZL: Coordinates of the exposure station, f : Camera focal length, ω, φ, κ : Rotation angles with respect to the x, y and z axes, m11, m12,... m33 : Elements of the rotation matrix.

The research objective was approached by assumptions of the fixed rigid structure as well as moving camera; hence, motions of the structure would be exhibited by the camera transformation. Accordingly, the determination of six exterior orientation parameters from the method of single image resection was recognized for each video frame. The DLT method is based on the collinear equations, which are extended by an affine transformation of image coordinates. Abdel-Aziz and Karara (1971) made the use of a standard photogrammetry method to solve for the EOP and the IOP (Interior Orientation Parameter) of a single image relative to a set of 3D coordinates. The transformation equation of DLT is given by Eq. (2) (Luhmann et al., 2006).

Whereas x, y: Image coordinates, X, Y, Z: Coordinates of the reference points in the object system, L1 – L11: DLT parameters to be estimated from the parameters of interior orientation and exterior orientation.

2.2.2 Least squares adjustment using control points

The least square solution of linearized collinearity equations from the non-linear method is considered the most accurate solution of space resection. If a suitable number of control points are available, the calculated exterior orientation by DLT can be applied as initial values of unknown parameters in the least square adjustment solution for finding the six exterior orientation parameters. The interior orientation (principal point, focal length and additional camera distortion) can be precisely determined by the well-known laboratory test field.

Accordingly, the general least squared adjustment method was utilized to compute the adjusted parameters of exterior orientation. The collinearity equations can be rewritten as Eq. (3):

The image coordinates x and y are considered the observation or measurements, while the elements of interior orientation x0, y0 and f are considered known from the calibration. The remaining variables are considered unknown parameters. Consequently, the linearized form of Eq. (3) is given by Eq. (4) (Mikhail et al., 2001)

where v = [vx vy]T: Image coordinate residuals, B: The matris of partial derivatives of the two functions in Eq. (3) with respect to each of six exterior orientation elements, and three coordinates from the object point, Δ : The vector of nine corrections to approximate the parameters.

2.2.3 Six degrees of freedom by inverse resection

The relationship between coordinates of a point X in the object system, and x* in the camera system (Luhmann et al., 2006), is given by Eq. (5)

The translation vector, X0, and rotation matrix, R, form 6DOF of the camera are respected to the object system. The coordinates x* of a control point on the object, corresponding to the camera system, are calculated by Eq. (6)

The camera remained for the entire video acquisition, so the changes in spatial position and orientation of an object at the camera coordinates system can be obtained by the repeated inverse space resections, as in Eq. (6). A simulation program has been created from the research to execute the calculation of algorithm, as presented above.

 

3. Experiment Verification

3.1. Model experiment

The working performance of proposed methodology was verified by applying it to the measurement of rigid floating model movements. Fig. 2 shows the model of experiments that was carried out at the Coastal and Harbor Experiment Center of Chonnam National University. The dimensions of an experimental model can be seen in the drawing of floating structure model in Fig. 2(a). The video acquisition was performed with a camera Nikon D7000 24 fps to observe the motion process of a structure model for 231 seconds of duration time, while the wave generator’s setting value was 3.3 sec. As aforementioned, a set of reference points on the floating body was appropriate to track in image processing and calculate the EOPs. The disposition of these reference points with known coordinates is illustrated in Fig. 2(b).

Fig. 2.Experimental model; (a) Model drawing, and (b) Floating structure model

3.2. Results and analysis

As it is described above, the application of image processing in tracking points of interest was performed with the video data. Fig. 3 illustrates the matching point results between the first image and second image by 793 points with the SURF (Speed Up Robust Features). The affine transformation was used to determine the translation and rotation between two images. The transformation parameters are shown in Table 1. Fig. 4 shows a reasonable displacement of each target point corresponding to the xand y-axis in the image coordinates system. The accuracy of target point matching is summarized in Table 2, which has the maximum errors of 0.65 and 0.86 pixels for the column and row in image coordinates, respectively. These obtained results can be adequately acceptable within the range of 1 pixel error for all control points.

Fig. 3.Interest points and matched pairs; (a) First image, and (b) Second image

Table 1.Affine transformation results

Table 2.Matching pairs and transformation errors

Fig. 4.Automatic tracking results of reference point using image processing

The interior orientation parameters of the camera can be found in Table 3:

Table 3.Camera calibration parameters

Fig. 5 illustrates the results in the displacement quantity of observation mode, which is based on space resection by DLT, from relative coordinates of control points in both coordinate systems. The rotation angles by the X, Y, and Z-axes are correspondingly around 3.0, 2.8, and 1.6 degrees, as shown in Fig. 5(b); and the amplitudes of oscillation by axes are around 320, 350, and 450 mm, following Fig. 5(a).

Fig. 5.Displacement of floating model measured by DLT; (a) Translation, and (b) Rotation

The precision of exterior orientation of a camera can be expected to increase by applying the adjustment method. The results of space resection by DLT were used as initial values of six unknowns in the least square adjustment solution of linearized collinearity equations, as referred above. Figs. 6(a) and (b) show the adjustment results from exterior orientation, with oscillatory amplitudes of 500, 400, 250 mm and 3.1, 3.2, 1.4 degrees, following X, Y, and Z-axes and rotation angles, respectively. Additionally, the behavior of image-based displacement is stable with a cycle of 3.296 sec. By comparing the wave generator’s setting value of 3.3 sec, those results can be determined as similar.

Fig. 6.EOP by least squares adjustment; (a) Translation, and (b) Rotation

Therefore, the camera was stationary during the video acquisition, and the spatial motion of the structure can be fully determined by the inverse space resection. Figs. 7(a) and (b) show the results of floating structure movement with oscillatory amplitudes of 54, 49 and 96 mm by X, Y, and Z-axes, and rotation angles of 3.2, 3.4 and 0.58 degree, respectively. Following that, the performance of translation and rotation in 3-D coordinates can be seen in Figs. 7(c) and (d).

Fig. 7.6DOF by inverse resection; (a) Translation, (b) Rotation, (c) 3-D translation, and (d) 3-D rotation

In order to verify the accuracy of determined results, the displacements of floating model in vertical direction was compared from the setting values of wave height (see Table 4). In the experiment, the wave height was set up at 100 mm. According to Table 4, the errors of maximum value and mean value are 7.818 mm and 4.028 mm correspondingly. The RMSE (Root Mean Square Error) of 4.210 mm can be considered acceptable.

Table 4.Absolute accuracy of proposed method

 

4. Conclusion

This paper proposed a methodology to simulate the movement of a floating structure, using a video camera. This technique is based on image processing and space resection in photogrammetry, to provide an effective solution for determining the dynamic behavior of a moving structure. The results of exterior orientation by DLT can be used as initial values of unknown parameters, in least squares adjustment solution of space resection. The inverse resection procedure yields 6DOF parameters of the structure. In order to verify this research, the experiment was carried out with a floating model, using a digital camera with 24 fps video.

Additionally, this method can be considered as costeffective, simple, and safe procedure in surveying, replaced existing sensors.

Further research will be conducted for fully integrating the modeling methodology with more than one camera in the movement observation, to obtain motions of a dynamic object at each control point.