Browse > Article
http://dx.doi.org/10.4218/etrij.15.0113.1120

Fixed Homography-Based Real-Time SW/HW Image Stitching Engine for Motor Vehicles  

Suk, Jung-Hee (Information & Communications Core Technology Research Laboratory, ETRI)
Lyuh, Chun-Gi (Information & Communications Core Technology Research Laboratory, ETRI)
Yoon, Sanghoon (SoC Platform Research Center, KETI)
Roh, Tae Moon (Information & Communications Core Technology Research Laboratory, ETRI)
Publication Information
ETRI Journal / v.37, no.6, 2015 , pp. 1143-1153 More about this Journal
Abstract
In this paper, we propose an efficient architecture for a real-time image stitching engine for vision SoCs found in motor vehicles. To enlarge the obstacle-detection distance and area for safety, we adopt panoramic images from multiple telegraphic cameras. We propose a stitching method based on a fixed homography that is educed from the initial frame of a video sequence and is used to warp all input images without regeneration. Because the fixed homography is generated only once at the initial state, we can calculate it using SW to reduce HW costs. The proposed warping HW engine is based on a linear transform of the pixel positions of warped images and can reduce the computational complexity by 90% or more as compared to a conventional method. A dual-core SW/HW image stitching engine is applied to stitching input frames in parallel to improve the performance by 70% or more as compared to a single-core engine operation. In addition, a dual-core structure is used to detect a failure in state machines using rock-step logic to satisfy the ISO26262 standard. The dual-core SW/HW image stitching engine is fabricated in SoC with 254,968 gate counts using Global Foundry's 65 nm CMOS process. The single-core engine can make panoramic images from three YCbCr 4:2:0 formatted VGA images at 44 frames per second and frequency of 200 MHz without an LCD display.
Keywords
Image stitching; panoramic image; motor vehicles; real time; homography; vision SoC; ISO26262;
Citations & Related Records
Times Cited By KSCI : 3  (Citation Analysis)
연도 인용수 순위
1 R. Szeliski, "Image Alignment and Stitching: A Tutorial," Microsoft Research, Tech. Rep., MSR-TR-2004-92, Oct. 2004.
2 R. Szeliski, "Video Mosaics for Virtual Environments," IEEE Comput. Graph. Appl., vol. 16, no. 2, Mar. 1996, pp. 22-30.   DOI
3 B.S. Kim, S.H. Lee, and N.I. Cho, "Real-Time Panorama Canvas of Natural Images," IEEE Trans. Consum. Electron., vol. 57, no. 4, Nov. 2011, pp. 1961-1968.   DOI
4 S. Mann and R.W. Picard, "Virtual Bellows: Constructing High Quality Stills from Video," Proc. IEEE Int. Conf. Image Process., Austin, TX, USA, vol. 1, Nov. 13-16, 1994, pp. 363-367.
5 S. Chen, "Quicktime VR: An Image-Based Approach to Virtual Environment Navigation," Proc. SIGGRAPH, New York, USA, Aug. 1995, pp. 29-38.
6 Y. Xu, X. Li, and Y. Tian, "Automatic Panorama Mosaicing with High Distorted Fisheye Images," Proc. Int. Conf. Natural Comput., Yantai, China, Aug. 10-12, 2010, pp. 3286-3290.
7 S.K. Nayar, "Catadioptric Omnidirectional Camera," Proc. IEEE Conf. Comput. Vis. Pattern Recogn., San Juan, Puerto Rico, June 17-19, 1997, pp. 482-488.
8 T. Kawanishi et al., "Generation of High-Resolution Stereo Panoramic Images by Omnidirectional Sensor Using Hexagonal Pyramidal Mirrors," Proc. Int. Conf. Pattern Recogn., Brisbane, Australia, vol. 1, Aug. 16-20, 1998, pp. 485-489.
9 S. Peleg and M. Ben-Ezra, "Stereo Panorama with a Single Camera," IEEE Conf. Comput. Vis. Pattern Recogn., Fort Collins, CO, USA, vol. 1, June 23-25, 1999, pp. 395-401.
10 F. Huang et al., "Animated Panorama from a Panning Video Sequence," Int. Conf. Image Vis. Comput. New Zealand, Queenstown, New Zealand, Nov. 8-9, 2010, pp. 1-8.
11 S. Peleg, M. Ben-Ezra, and Y. Pritch, "Omnistereo: Panoramic Stereo Imaging," IEEE Trans. Pattern Anal. Mach. Intell., vol. 23, no. 3, Mar. 2001, pp. 279-290.   DOI
12 A. Ahmed et al., "Geometric Correction for Uneven Quadric Projection Surfaces Using Recursive Subdivision of Bezier Patches," ETRI J., vol. 35, no. 6, Dec. 2013, pp. 1115-1125.   DOI
13 S.J. Ha et al., "Panorama Mosaic Optimization for Mobile Camera Systems," IEEE Trans. Consum. Electron., vol. 53, no. 4, Nov. 2007, pp. 1217-1225.   DOI
14 M. Brown and D.G. Lowe, "Automatic Panoramic Image Stitching Using Invariant Features," Int. J. Comput. Vis., vol. 74, no. 1, Aug. 2007, pp. 59-73.   DOI
15 S.J. Ha et al., "Embedded Panoramic Mosaic System Using Auto-Shot Interface," IEEE Trans. Consum. Electron., vol. 54, no. 1, Feb. 2008, pp. 16-24.   DOI
16 J.M. Seok and Y. Lee, "Visual-Attention-Aware Progressive RoI Trick Mode Streaming in Interactive Panoramic Video Service," ETRI J., vol. 36, no. 2, Apr. 2014, pp. 253-263.   DOI
17 D. Wagner et al., "Real-Time Panoramic Mapping and Tracking on Mobile Phones," IEEE Conf. Virtual Reality, Waltham, MA, USA, Mar. 20-24, 2010, pp. 211-218.
18 D.G. Lowe, "Distinctive Image Features from Scale-Invariant Keypoints," Int. J. Comput. Vis., vol. 60, no. 2, 2004, pp. 91-110.   DOI
19 H. Bay et al., "SURF: Speeded-Up Robust Features," Comput. Vis. Image Understanding, vol. 110, no. 3, 2008, pp. 346-359.   DOI
20 A. Shashua et al., EyeQ, Mobileye, 2010. Accessed Aug. 15, 2013. http://www.mobileye.com/technology/processing-platforms/eyeq/
21 A. Shashua et al., EyeQ2, Mobileye, 2010. Accessed Aug. 15, 2013. http://www.mobileye.com/technology/processing-platforms/eyeq2/
22 G.P. Stein, Y. Gdalyahu, and A. Shashua, "Stereo-Assist: Top-Down Stereo for Driver Assistance Systems," IEEE. Conf. Intell. Vehicles Symp., San Diego, CA, USA, 2010, pp. 723-730.
23 J.-H. Suk et al., "An Efficient Architecture of Image Stitching Engine for Vis. SoC," Proc. Workshop Image Processing Image Understanding, Jeju, Rep. of Korea, Feb. 15-17, 2012, Index O-6.
24 K.H. Kwon et al., EISC, Advanced Digital Chips Inc., 2012. Accessed June 15. http://www.adc.co.kr/technology/eisc/eisc.php
25 Homography, Wikipedia, 2013. Accessed May 30, 2014. https://en.wikipedia.org/wiki/Homography
26 Image Stitching, Wikipedia, 2008. Accessed Aug. 30, 2013. https://en.wikipedia.org/wiki/Image_stitching
27 International Organization for Standardization (ISO), ISO 26262 Road Vehicle-Functional Safety Geneva, Switzerland: ISO, 2011.
28 H.S. Park et al., "In-Vehicle AR-HUD System to Provide Driving-Safety Information," ETRI J., vol. 35, no. 6, Dec. 2013, pp. 1038-1047.   DOI
29 M.A. Fischler and R.C. Bolles, "Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography," Commun. ACM, vol. 24, no. 6, June 1981, pp. 381-395.   DOI
30 M. Zuliani, "RANSAC for Dummies," Matlab draft, Jan. 2012.
31 K. Baker, "Singular Value Decomposition Tutorial," tutorial paper, Mar. 2005.
32 Y.Y. Boykov and M. Jolly, "Interactive Graph Cuts for Optimal Boundary and Region Segmentation of Objects in N-D Images," Proc. IEEE Int. Conf. Comput. Vis., Vancouver, Canada, vol. 1, July 7-14, 2001, pp. 105-112.
33 Alpha Compositing, Wikipedia, 2006. Accessed Aug. 30, 2013. https://en.wikipedia.org/wiki/Alpha_compositing#Alpha_blendin
34 S. Ali and M. Hussain, "Panoramic Image Construction Using Feature Based Registration Methods," Proc. Int. Multitopic Conf., Islamabad, Pakistan, Dec. 13-15, 2012, pp. 209-214.
35 Point Grey Research Inc. (2012, Dec. 7), Spherical Video System Ladybug2 and Ladybug3. Available: http://www.ptgrey.com
36 Y. Xu et al., "High-Speed Simultaneous Image Distortion Correction Transformations for a Multicamera Cylindrical Panorama Real-Time Video System Using FPGA," IEEE Trans. Circuits Syst. Video Technol., vol. 24, no. 6, June 2014, pp. 1061-1069.   DOI
37 O. Schreer et al., "Ultra-high-Resolution Panoramic Imaging for Format-Agnostic Video Production," Proc. IEEE, vol. 101, no. 1, Jan. 2013, pp. 99-114.   DOI
38 A. Akin et al., "Enhanced Omnidirectional Image Reconstruction Algorithm and its Real-Time Hardware," Euromicro Conf. Digital Syst. Des., Izmir, Turkey, Sept. 5-8, 2012, pp. 907-914.