A New Framework for Data Reduction in Large-scale Data Using Mapreduce

Storing and processing large volume datasets is one of the most critical problems in large-scale processing. Therefore, it is need to reduce their size before further processing. This paper is proposed a framework for data reduction in large-scale datasets. The proposed framework is based on MapReduce algorithm. It has three steps. Firstly, by reservoir sampling, some instances of a dataset are selected. In the second step, the features of these selected instances are weighted using ReliefF algorithm. Then, all weights are averaged for each feature and features with the highest weight values are selected. Finally, the selected features have been used in classification. Implementation results of the proposed framework show a good reduction of time. It also increases accuracy or maintains it when a large amount of data is removed by eliminating irrelevant features in classification algorithms.

کلیدواژه ها:

Large-scale data ، MapReduce ، Feature Selection ، Instance selection