Bridge collapses may deliver a huge impact on our society in a very negative way. Out of many reasons why bridges collapse, poor maintenance is becoming a main contributing factor to many recent collapses. Furthermore, the aging of bridges is able to make the situation much worse. In order to prevent this unwanted event, it is indispensable to conduct continuous bridge monitoring and timely maintenance. Visual inspection is the most widely used method, but it is heavily dependent on the experience of the inspectors. It is also time-consuming, labor-intensive, costly, disruptive, and even unsafe for the inspectors. In order to address its limitations, in recent years increasing interests have been paid to the use of unmanned aerial vehicles (UAVs), which is expected to make the inspection process safer, faster and more cost-effective. In addition, it can cover the area where it is too hard to reach by inspectors. However, this strategy is still in a primitive stage because there are many things to be addressed for real implementation. In this paper, a typical procedure of bridge inspection using UAVs consisting of three phases (i.e., pre-inspection, inspection, and post-inspection phases) and the detailed tasks by phase are described. Also, three major challenges, which are related to a UAV's flight, image data acquisition, and damage identification, respectively, are identified from a practical perspective (e.g., localization of a UAV under the bridge, high-quality image capture, etc.) and their possible solutions are discussed by examining recently developed or currently developing techniques such as the graph-based localization algorithm, and the image quality assessment and enhancement strategy. In particular, deep learning based algorithms such as R-CNN and Mask R-CNN for classifying, localizing and quantifying several damage types (e.g., cracks, corrosion, spalling, efflorescence, etc.) in an automatic manner are discussed. This strategy is based on a huge amount of image data obtained from unmanned inspection equipment consisting of the UAV and imaging devices (vision and IR cameras).