A Novel Approach towards Big Data Challenges

Abstract

Big Data is a term for explosion of high quantity and diversity of high frequency digital data. In this era of “digital universe” we encounter data sets with high volume, variety, velocity and complexity. Data, information and knowledge are touching exponential powers of exabytes and zettabytes. The data generated is that immense that it cannot be efficiently processed by traditional database methods, tools and current technologies. The source of Big Data is sensor networks, call logs, retail transactions, user generated content that is producing structured and unstructured data. The problem does not only comprise these aspects, heterogeneity, privacy, data aggregation and transporting are also huge contributors. So today, all we require is to center on these challenges and revamping our approach towards the methodologies for Big Data representation, analysis and design. We require superior storage and management architectures that are fast, fault tolerant, flexible and scalable for analyzing data efficiently and extracting relevant data for decision-making. The focus of this paper is on giving a holistic view of Big Data, its challenges, how present technologies are dealing with these challenges and what is more to be explored as a solution to Big Data. Also to look over technologies like Hadoop, MapReduce, BigQuery and Apache Sparks.