An intelligent and distributed crawling algorithm using Map-Reduce

In this paper we have presented a Map-Reduce version of an effective crawler called FICA. The proposed approach is based on logarithmic distance and has reasonable time complexity. Here, we present the improved and distributed implementation of this crawler. Comprehensive test cases are designed and the result are analyzed. These experiments lead to identifying a major bottleneck in the distributed version of FICA and an improved version is presented. We achieved 3x speedup for total execution time in comparison with naïve Map-Reduce implementation.

کلیدواژه ها:

Crawling Algorithm ، Page Rank ، Map-Reduce ، Distributed ، FICA