A Ship-Wake Joint Detection Using Sentinel-2 Imagery

Abstract

Ship detection is widely used in areas such as maritime security, maritime traffic, fisheries management, illegal fishing, and border control, and ship detection is important for rapid response and damage minimization as ship accident rates increase due to recent increases in international maritime traffic. Currently, according to a number of global and national regulations, ships must be equipped with automatic identification system (AIS), which provide information such as the location and speed of the ship periodically at regular intervals. However, most small vessels (less than 300 tons) are not obligated to install the transponder and may not be transmitted intentionally or accidentally. There is even a case of misuse of the ship'slocation information. Therefore, in this study, ship detection was performed using high-resolution optical satellite images that can periodically remotely detect a wide range and detectsmallships. However, optical images can cause false-alarm due to noise on the surface of the sea, such as waves, or factors indicating ship-like brightness, such as clouds and wakes. So, it is important to remove these factors to improve the accuracy of ship detection. In this study, false alarm wasreduced, and the accuracy ofship detection wasimproved by removing wake.As a ship detection method, ship detection was performed using machine learning-based random forest (RF), and convolutional neural network (CNN) techniquesthat have been widely used in object detection fieldsrecently, and ship detection results by the model were compared and analyzed. In addition, in this study, the results of RF and CNN were combined to improve the phenomenon of ship disconnection and the phenomenon of small detection. The ship detection results of thisstudy are significant in that they improved the limitations of each model while maintaining accuracy. In addition, if satellite images with improved spatial resolution are utilized in the future, it is expected that ship and wake simultaneous detection with higher accuracy will be performed.

1. Introduction

The recent increase in international sea traffic has led to a rise in ship accidents, making ship detection crucial for rapid response and damage minimization (Zou and Shi, 2016; Liu et al., 2017; Heiselberg, 2016). While automatic identification system (AIS) and other shipborne transponders have been implemented to provide location information for certain classes of ships, small vessels under 300 tons are not obligated to carry them and may not transmit their location intentionally or accidentally (Kanjir et al., 2018). Furthermore, there have been cases of illegal ships intentionally falsifying their location information (Heiselberg, 2016). Therefore, non-cooperative detection systems such as satellite-based ship detection are necessary for comprehensive and accurate monitoring of maritime activities, including maritime security, maritime transportation, fisheries management, illegal fishing, and border control.

Satellite-based sensors have the advantages of remote detection, global reach, regular updates, and high data collection volumes (Kanjir et al., 2018). Therefore, the use of satellite images is the most economical and essential tool for detecting ships in the ocean (Kanjir et al., 2018).

Ship detection utilizes a range of imaging technologies, including optical and reflected infrared, hyperspectral, thermal infrared, and radar. While synthetic aperture radar (SAR) images are commonly used due to their weather and time independence, their limited satellite availability limits the covered area (Zou and Shi, 2016; Eldhuset, 1996; Dragosevic and Vachon, 2008; Li and Chong, 2008). In contrast, optical images are becoming increasingly prevalent due to the recent surge in polar orbit, geostationary orbit, and clustered microsatellites. With improved spatial resolution and an expanding amount of data, the combined use of high-resolution optical satellite images can cover larger areas and increase observation frequency. However, false alarms due to clouds, waves, and ship wake can cause errors in optical ship detection (Yang et al., 2013). While ship wake can provide reference points for detecting the direction and speed of a ship and identifying small ships with low-resolution problems, it can also contribute to inaccuracies in ship size detection, which can impact maritime security and damage assessment in accidents. As such, removing the wake is essential for improving the accuracy of ship detection, as emphasized in various studies (Yang et al., 2011; Bouma et al., 2013; Buck et al., 2007).

Ship detection techniques that rely on thresholding to distinguish between high and low pixel values can be effective in detecting ships on smooth sea surfaces or when there is a high contrast between ship targets and the sea background. However, these methods may result in a high false alarm rate when images are cluttered. In recent years, machine learning has emerged as a popular ship detection technique due to its ability to handle large amounts of data and produce fast results. Machine learning involves training artificial neural networks to learn from the input data without explicitly defining object features (Mitchell, 1997). However, machine learning models require a large training set, and if not implemented carefully, they may misclassify new objects that differ from the training set (Kanjir et al., 2018). This study aims to accurately detect ships and their wakes in optical images using two machine learning-based models: random forest (RF) and convolutional neural network (CNN). The study will also compare the performance of the two models and combine their results to achieve the best possible ship detection accuracy. By detecting wakes, which are the primary cause of false alarms in optical images, this study seeks to improve the accuracy of ship detection in cluttered image environments.

2. Study Area and Data

2.1. Study Area

Fig. 1 is the location of ports located in Korea. Korea has a high rate of ship accidents due to its complicated coastline (Park et al., 2013). According to the National Port Entry and Exit Ship Statistics provided by the Ministry of Maritime Affairs and Fisheries, Busan has the largest number of ships coming in and out with 292,873 ships over the past two years. Therefore, we designated the Yeongdo (35.03806ºN–35.07056ºN and 129.0278ºN–129.0878ºN) of Busan as a study area and conducted ship detection (Fig. 1a).

Fig. 1. The location of ports located in Korea. (a) The study area of this thesis.

2.1.1. Sentinel-2A-2B/MSI Satellite Data

Sentinel-2A-2B Multispectral Instrument (MSI) images used in this study are Level 2 bottom of atmosphere (BOA) data from February 2019 to February 2021 (the 16 scenes between 2019 and 2020 are training data, the 4 scenes for 2021 are test data), and these images are used in the study of Blue (Band 2), Green (Band 3), Red (Band 4), and near-infrared (NIR) (Band 8) bands with cloudless spatial resolution of 10 m. Table 1 shows Sentinel-2A-2B/MSI channel data used in this study.

Table 1. Sentinel-2A-2B/MSI channel data used in this study

We used crop images of the study area from the collected Sentinel-2A-2B/MSI satellite images, aligned with the latitude and longitude of the study area. In addition, to eliminate the possibility of any potential cloud presence, we utilized Whiteness Test (Eq. 1, 2) and Hot Test (Eq. 3). The Whiteness Test index is a method of detecting clouds by utilizing points that appear white due to the characteristic of clouds showing high reflectance values in the visible spectrum, using the sum of the absolute differences between the Red, Green, Blue channels and the overall brightness (Zhu et al., 2012). The Hot Test index is a method of detecting clouds by utilizing the point where the reflectance difference between the Blue and Red channels occurs for haze and thin clouds (Zhu et al., 2012).

\(\begin{aligned}\text {Whitness Test}=\\\left(\frac{\mid \text { Red } \mid- \text { Mean }}{\text { Mean }}+\frac{\mid \text { Green } \mid- \text { Mean }}{\text { Mean }}+\frac{\mid \text { Blue } \mid- \text { Mean }}{\text { Mean }}\right)\\\end{aligned}\)       (1)

\(\begin{aligned}Mean=\frac{\text { Red }+ \text { Green }+ \text { Blue }}{3}\end{aligned}\)       (2)

Hot Test = Blue – (0.5 × Red) – 0.08 (3)

2.1.1.1. Ship Reference Data

In this paper, since the instance segmentation method will be used for ship detection, annotation work is required for the learning dataset. Therefore, Labelme, one of the Annotation tools used for data labeling, was used to build ship reference data. Labelme is an open annotation tool that can manually label the desired area through the polygon in the image. Fig. 2 shows the reference data constructed using Labelme in this study in RGB images. The area marked with a red border is a ship built with reference data.

Fig. 2. Ship reference data built with Labelme and enlarged images, red border is the area of the ship reference data.

3. Methods and Results

Fig. 3 shows a flow chart of the study. Perform data preprocessing and build references to the collected data. Divide the data into training and test data at a ratio of 8:2 and learn both RF and CNN models using the training data. Using the test data, ship detection for each model is performed, and the final ship detection result is obtained by comparing and analyzing the results and accuracy of each model in the 3.1.1.1. model evaluation section. RF+CNN classified the pixel as a ship when even one model was classified as a ship and proceeded the same for wake, to reduce the number of misdetected or undetected ship pixels when a ship detection is performed using one model.

Fig. 3. Flow Chart.

3.1. Random Forest

RF is a representative ensemble classifier based on several decision trees trained with randomly selected data subsets and feature sets (Breiman, 2001). RF is a representative ensemble technique for regression and classification and is mainly used in the field of object detection and calculation in remote sensing.

To estimate how well the model performed, the RF uses approximately 2/3 of the samples (referred to as in-bag samples) for tree learning and the remaining 1/3 (referred to as out-of-bag) for internal cross-validation (Breiman, 2001). This estimate is called out-of-bag (OOB) score, and the closer the OOB score is to 1, the better the model is learned. The number of pixels in the learning data in this study is 8,423 ship pixels, 639 wake pixels and 2,612,224 sea pixels. If the data is used as it is, learning biased toward sea characteristics can be performed due to the imbalance of the data, resulting in an overfitting model for the sea. Therefore, we adjusted the data imbalance by increasing the number of wake pixels and reducing the number of sea pixels based on the number of ship pixels. Finally, 8,670 ship pixels, 7,469 wake pixels and 18,698 sea pixels were used as RF input data sets.

First, a total of 14 input features - band reflectance and band ratio, ship detection index (SDI) (Park et al., 2018; Eq. 4), wake detection index (WDI) (Jeon et al., 2021; Eq. 5), normalized difference water index (NDWI) (Eq. 6), which is mainly used to analyze water bodies because ships exhibit spectral properties similar to land and normalized difference vegetation index (NDVI). Park et al. (2018) created SDI based on the spectral characteristics of ships showing higher reflectance values compared to the surrounding sea in multi-spectral channels, especially in the Red and NIR channels. Ship and wake have similar spectral characteristics, so it cannot be distinguished by SDI. Therefore, Jeon et al. (2021) developed WDI by utilizing the spectral characteristics of ship and wake to improve the accuracy of ship by distinguishing between ship and wake. First, the Blue channel reflectance value and the SDI value are similar, but the difference in reflectance is large in NIR. Second, the Red reflectance value of the ship is smaller than the NIR reflectance value, but the Red reflectance value of wake is larger than the NIR reflectance value.

\(\begin{aligned}S D I=\frac{\operatorname{Red}-\operatorname{Red}_{\min }}{\operatorname{Red}_{\max }-\operatorname{Red}_{\min }} \times \frac{N I R-N I R_{\min }}{N I R_{\max }-N I R_{\min }}\end{aligned}\)      (4)

\(\begin{aligned}W D I=\left(\frac{B l u e-N I R}{S D I+N I R}\right)+\left(\frac{\operatorname{Red}-N I R}{S D I}\right)-0.3\end{aligned}\)       (5)

\(\begin{aligned}N D W I=\frac{\text { Green }-N I R}{\text { Green }+N I R}\end{aligned}\)        (6)

We used a simple feature selection based on the relative feature importance provided by RF through iterative testing with different sets of input features. Finally, 7 parameters were selected and these were used for RF development, which simultaneously detects ships and wake and effectively removes false alarms. Fig. 4 shows the feature importance of the finally selected 7 features. The OOB score of the final RF model was 0.985.

Fig. 4. The feature importance of the finally selected 7 features.

3.1.1. Mask R-CNN (Detectron2)

As a state-of-the-art method for detecting objects, models using deep neural networks are being used and performing well. In this study, the Mask R-CNN approach to perform ship detection. Instead of developing the Mask R-CNN model from scratch, we use Detectron2 to shorten the development cycle. Detectron2 is a training/inference platform for Pytorch-based object detection and semantic segmentation created by Facebook artificial intelligence research (FAIR). Detectron2 has a structure that adds a classification branch that predicts the class of an object and a mask branch that predicts the segmentation mask parallel to the bbox regression branch that performs the bbox regression for region of interest (RoI) obtained from the region proposal network (RPN) of the Mask R-CNN. The Detectron2 architecture is shown in Fig. 5. The input data is in COCO JSON format. We tested and evaluated all the proposed related algorithms with the following settings: (i) different number of iterations (500 to 7,000), (ii) different number of images per batch (2–32), (iii) different batch size per image (8–512), (iv) different learning rates (0.00025–0.01). The total loss value for the final learning model is 0.5517.

Fig. 5. Detectron2 architecture based on Mask R-CNN.

3.1.1.1. Model Evaluation

The assessment criteria in Powers et al. (2020) are used here, and the definitions are as follows:

\(\begin{aligned}Precision=\frac{n_{t p}}{n_{t p}+n_{f p}}\end{aligned}\)        (7)

\(\begin{aligned}\operatorname{Recall}=\frac{n_{t p}}{n_{t p}+n_{f n}}\end{aligned}\)       (8)

\(\begin{aligned}F 1-score=2 \times \frac{\text { Precision } \times \text { Recall }}{\text { Precision }+ \text { Recall }}\end{aligned}\)       (9)

\(\begin{aligned} I o U & =\frac{\text { Overlapping Region }}{\text { Combined Region }} \\ & =\frac{\text { Reference and Prediction }}{\text { Reference or Prediction }}\end{aligned}\)      (10)

where n_tp is the number of true positive, n_fp represents the number of false positives, and n_fn denotes the number of false negatives. Precision and Recall are the precision rate and recall rate, respectively. F1 – score is the F1-measure.

Fig. 6 illustrates a ship detection result using test data and provides a visual comparison of the detection results obtained by RF and CNN models. The shape and size of the ship were detected differently by each model. For ships that have wake, the boundary between the ship and the wake was not clear, leading to more differences in the shape and size of the ship detected by each model compared to the case of ships without wake.

Fig. 6. Ship detection results of two models and ship detection results combining RF and CNN.

Table 2. Detection performance

RF detects ships based on the spectral characteristics of pixels, which may result in the phenomenon where one ship is disconnected. In contrast, CNN uses Anchor box, which is a bounding box with various aspect ratios, and RPN to perform candidate region extraction tasks more accurately, avoiding the issue of one ship being cut off. However, compared to the reference data, the size of the ship tended to be detected as small in the case of CNN.

It is encouraging to see that the combination of the RF and CNN models improved the ship detection results. The ship disconnection issue of the RF model was addressed by the more precise region proposal extraction of the CNN model, while the smaller ship detection issue of the CNN model was remedied by the spectral characteristics-based detection of the RF model. The RF-CNN fusion result produced the most accurate ship detection results, with the highest recall value of 0.97 and F1-score and intersection over union (IoU) values that were similar to the other two models. This indicates that the fusion result was able to maintain accuracy while improving on the weaknesses of the individual models. Overall, the study successfully demonstrated the effectiveness of machine learning-based ship detection using RF and CNN models, as well as the potential of combining different models to improve results.

4. Discussion and Conclusions

Table 3 is accuracy comparison with other studies. When comparing accuracy with other previous studies, it was difficult to compare accurately because the data used and study area were different, but the recall value of this study was the highest and the overall accuracy was similar. In object detection, the spatial resolution of the satellite image has a large effect. The lower the spatial resolution, the higher the accuracy of object detection. Ship detection was performed using satellite images with higher spatial resolution than Zhang (2020) and Wang (2021), but F1-score was similar. Therefore, if ship detection is performed using satellite images with better spatial resolution as the ship detection method of this study, higher accuracy can be expected.

Table 3. Accuracy comparison with other studies

Ship detection can be widely used in areas such as maritime security, maritime traffic, illegal fishing, and border control, and ship detection is important to respond quickly to ship accidents and minimize damage caused by increased sea traffic. In this study, ship detection was performed using Sentinel-2A-2B satellite images. In addition, in this study, the accuracy of ship detection was improved by removing the wake, which is a major element of False alarm. To detect ships and wake, RF of efficient and highly predictive method for big data and CNN of image-based object detection method were used. Both the RF and CNN models detected all ships well without undetected ships for the presence or absence of ships. However, in the case of RF, one ship was cut off because each pixel was distinguished whether it was a ship or not based on the spectral characteristics of each pixel, and CNN tended to detect ships smaller than reference data. Therefore, in this study, the ship disconnection phenomenon and the ship small detection phenomenon were improved by fusing the RF and CNN ship detection results. The quantitative verification of the final RF-CNN fusion results showed the accuracy of Precision 0.74, Recall 0.97, F1-score 0.84, and IoU 0.72.

This study performed ship detection using RF and CNN from optical images. It is significant in that it has increased accuracy by simultaneously removing wake when performing ship detection. In addition, it is significant in that the ship detection results were compared and analyzed by model, and the results were combined to supplement the limitations of each model while maintaining accuracy. The resolution of the optical image continues to improve. In the future, using optical satellite images with improved spatial resolution is expected to perform ship detection and wake detection with higher accuracy. Also, it is expected to be used for monitoring illegal China fishing boats and small North Korea’s ships.