• Title/Summary/Keyword: Dynamic Crawling

Search Result 24, Processing Time 0.024 seconds

Study on Dynamic Crawling of The Five-bar Planar Mechanism (5절 평면형 메커니즘의 동적 포복에 관한 연구)

  • Lee J.H.;Lim N.S.;Kim W.K.;Yi B.J.
    • Proceedings of the Korean Society of Precision Engineering Conference
    • /
    • 2005.06a
    • /
    • pp.1045-1049
    • /
    • 2005
  • In this paper, the dynamic crawling of a five-bar planar mechanism is investigated. One complete cycle of the crawling selected in this study consists of four different steps, i) sliding at one contact point between the mechanism and the ground, ii) changing its configuration without sliding at two contact points, iii) sliding at the other contact point, and iv) again changing its configuration without sliding at two contact points. In this type of crawling, the crawling mechanism maintains the shape of the parallel structure throughout a complete crawling cycle. The modeling algorithm for serial manipulators proposed by M. Thomas and et al.[1] is employed by introducing imaginary joints and links which represent the contact interfaces between the one end of the mechanism and the ground, while the other end of the mechanism is regarded as an end-effector of the imaginary serial manipulator which treats the reaction force and torque at the contact point as external forces. Then, a complete cycle of dynamic crawling of the mechanism is investigated through various computer simulations. The simulation result show that the stable crawling characteristics of the mechanism could be secured when the proper configurations depending on specified frictional constraints are met.

  • PDF

Comparison and Application of Dynamic and Static Crawling for Extracting Product Data from Web Pages (웹페이지에서의 상품 데이터 추출을 위한 동적, 정적 크롤링 비교 및 활용)

  • Sang-Hyuk Kim;Jeong-Hoon Kim;Seung-Dae Lee
    • The Journal of the Korea institute of electronic communication sciences
    • /
    • v.18 no.6
    • /
    • pp.1277-1284
    • /
    • 2023
  • In this paper, a web page that is easy for consumers to access event products in progress at convenience stores was created. In the production process, static crawling and dynamic crawling, two crawling methods for extracting data from event products, were compared and used. Static crawling is an extraction method of collecting static data from a homepage, and dynamic crawling is a method of collecting data from pages dynamically generated from a web page. Through the comparison of the two crawlings, we studied which crawl method is more effective in extracting event product data. Among them, a web page was created using effective static crawling, and 1+1 and 2+1 products were categorized and a search function was added to create a web page.

Development of a three-dimensional dynamic model for chemotaxis

  • Song, Jihwan;Kim, Dongchoul
    • Interaction and multiscale mechanics
    • /
    • v.4 no.2
    • /
    • pp.165-171
    • /
    • 2011
  • In this study, we proposed a three-dimensional dynamic model under the diffuse interface description for the single crawling cell. From the developed model, we described the clear evolution processes for crawling neutrophil and assessed the reliable quantitative chemotactic property, which confirmed the high possibility of adequate predictions. To establish the system considering of multiple mechanisms such as, diffusion, chemotaxis, and interaction with surface, a diffuse interface model is employed.

A Study on the Implementation of Crawling Robot using Q-Learning

  • Hyunki KIM;Kyung-A KIM;Myung-Ae CHUNG;Min-Soo KANG
    • Korean Journal of Artificial Intelligence
    • /
    • v.11 no.4
    • /
    • pp.15-20
    • /
    • 2023
  • Machine learning is comprised of supervised learning, unsupervised learning and reinforcement learning as the type of data and processing mechanism. In this paper, as input and output are unclear and it is difficult to apply the concrete modeling mathematically, reinforcement learning method are applied for crawling robot in this paper. Especially, Q-Learning is the most effective learning technique in model free reinforcement learning. This paper presents a method to implement a crawling robot that is operated by finding the most optimal crawling method through trial and error in a dynamic environment using a Q-learning algorithm. The goal is to perform reinforcement learning to find the optimal two motor angle for the best performance, and finally to maintain the most mature and stable motion about EV3 Crawling robot. In this paper, for the production of the crawling robot, it was produced using Lego Mindstorms with two motors, an ultrasonic sensor, a brick and switches, and EV3 Classroom SW are used for this implementation. By repeating 3 times learning, total 60 data are acquired, and two motor angles vs. crawling distance graph are plotted for the more understanding. Applying the Q-learning reinforcement learning algorithm, it was confirmed that the crawling robot found the optimal motor angle and operated with trained learning, and learn to know the direction for the future research.

Refresh Cycle Optimization for Web Crawlers (웹크롤러의 수집주기 최적화)

  • Cho, Wan-Sup;Lee, Jeong-Eun;Choi, Chi-Hwan
    • The Journal of the Korea Contents Association
    • /
    • v.13 no.6
    • /
    • pp.30-39
    • /
    • 2013
  • Web crawler should maintain fresh data with minimum server overhead for large amount of data in the web sites. The overhead in the server increases rapidly as the amount of data is exploding as in the big data era. The amount of web information is increasing rapidly with advanced wireless networks and emergence of diverse smart devices. Furthermore, the information is continuously being produced and updated in anywhere and anytime by means of easy web platforms, and smart devices. Now, it is becoming a hot issue how frequently updated web data has to be refreshed in data collection and integration. In this paper, we propose dynamic web-data crawling methods, which include sensitive checking of web site changes, and dynamic retrieving of web pages from target web sites based on historical update patterns. Furthermore, we implemented a Java-based web crawling application and compared efficiency between conventional static approaches and our dynamic one. Our experiment results showed 46.2% overhead benefits with more fresh data compared to the static crawling methods.

An Implementation and Performance Evaluation of Fast Web Crawler with Python

  • Kim, Cheong Ghil
    • Journal of the Semiconductor & Display Technology
    • /
    • v.18 no.3
    • /
    • pp.140-143
    • /
    • 2019
  • The Internet has been expanded constantly and greatly such that we are having vast number of web pages with dynamic changes. Especially, the fast development of wireless communication technology and the wide spread of various smart devices enable information being created at speed and changed anywhere, anytime. In this situation, web crawling, also known as web scraping, which is an organized, automated computer system for systematically navigating web pages residing on the web and for automatically searching and indexing information, has been inevitably used broadly in many fields today. This paper aims to implement a prototype web crawler with Python and to improve the execution speed using threads on multicore CPU. The results of the implementation confirmed the operation with crawling reference web sites and the performance improvement by evaluating the execution speed on the different thread configurations on multicore CPU.

The impact of inter-host links in crawling important pages early

  • Alam, Hijbul;Ha, Jong-Woo;Sim, Kyu-Sun;Lee, Sang-Keun
    • Proceedings of the Korean Information Science Society Conference
    • /
    • 2010.06c
    • /
    • pp.118-121
    • /
    • 2010
  • The dynamic nature and exponential growth of the World Wide Web remain crawling important pages early still challenging. State-of-the-art crawl scheduling algorithms require huge running time to prioritize web pages during crawling. In this research, we proposed crawl scheduling algorithms that are not only fast but also download important pages early. The algorithms give high importance to some specific pages those have good linkages such as inlinks from different domains or host. The proposed algorithms were experimented on publically available large datasets. The results of experiments showed that propagating more importance to the inter-host links improves the effectiveness of crawl scheduling than the current state-of-the-art crawl scheduling algorithms.

  • PDF

Development of Three-dimensional Chemotaxis Model for a Single Crawling Cell, Considering the Interaction between the Cell and Substrate (세포와 흡착면간의 영향을 고려한 흡착형 세포의 3 차원 동적 해석 모델 개발)

  • Song, Ji-Hwan;Kim, Dong-Choul
    • Transactions of the Korean Society of Mechanical Engineers A
    • /
    • v.35 no.11
    • /
    • pp.1355-1360
    • /
    • 2011
  • The interaction between the cell and the substrate is the most prominent feature affecting the migration of a crawling cell. This paper proposes a three-dimensional dynamic model using the diffuse interface description that reveals the effects of the interaction between a single crawling cell and the substrate during chemotactic migration. To illustrate the effects of interaction between the cell and the substrate, we consider the interfacial energy between the coexistent materials. Multiple mechanisms including the interface energy, chemotaxis effect, and diffusion, are addressed by employing a diffuse interface model.

Characterization of Dynamic Behavior of C. elegans in Different Physical Environments (PIV 및 TFM 측정 기법을 이용한 예쁜꼬마선충의 동적 패턴 가시화 연구)

  • Park, Jin-Sung;Yun, Byoung Hwan;Shin, Jennifer H.
    • Journal of the Korean Society of Visualization
    • /
    • v.12 no.2
    • /
    • pp.18-22
    • /
    • 2014
  • Caenorhabditis elegans (C. elegans) is an undulatory nematode which exhibits two distinct locomotion types of swimming and crawling. Although in its natural habitat C. elegans lives in a non-Newtonian fluidic environment, our current understanding has been limited to the behavior of C. elegans in a simple Newtonian fluid. Here, we present some experimental results on the penetrating behavior of C. elegans at the interface from liquid to solid environment. Once C. elegans, which otherwise swims freely in a liquid, makes a contact to the solid gel boundary, it begins to penetrate vertically to the surface by changing its stroke motion characterized by a stiffer body shape and a slow stroke frequency. The particle image velocimetry (PIV) analysis reveals the flow streamlines produced by the stroke of worm. For the worm that crawls on a solid surface, we utilize a technique of traction force microscopy (TFM) to find that the crawling nematode forms localized force islands along the body where makes direct contacts to the gel surface.

A Study on Minimizing Infection of Web-based Malware through Distributed & Dynamic Detection Method of Malicious Websites (악성코드 은닉사이트의 분산적, 동적 탐지를 통한 감염피해 최소화 방안 연구)

  • Shin, Hwa-Su;Moon, Jong-Sub
    • Journal of the Korea Institute of Information Security & Cryptology
    • /
    • v.21 no.3
    • /
    • pp.89-100
    • /
    • 2011
  • As the Internet usage with web browser is more increasing, the web-based malware which is distributed in websites is going to more serious problem than ever. The central type malicious website detection method based on crawling has the problem that the cost of detection is increasing geometrically if the crawling level is lowered more. In this paper, we proposed a security tool based on web browser which can detect the malicious web pages dynamically and support user's safe web browsing by stopping navigation to a certain malicious URL injected to those web pages. By applying these tools with many distributed web browser users, all those users get to participate in malicious website detection and feedback. As a result, we can detect the lower link level of websites distributed and dynamically.