Preprint
Machine Learning

Big data storage technologies: a survey

Aisha Siddiqa(University of Malaya), Ahmad Karim(Bahauddin Zakariya University), Abdullah Gani(University of Malaya)
August 1, 2017Frontiers of Information Technology & Electronic Engineering122 citations
PDFDOI

122

Citations

0

Influential Citations

Frontiers of Information Technology & Electronic Engineering

Venue

2017

Year

Abstract

There is a great thrust in industry toward the development of more feasible and viable tools for storing fast-growing volume, velocity, and diversity of data, termed ‘big data’. The structural shift of the storage mechanism from traditional data management systems to NoSQL technology is due to the intention of fulfilling big data storage requirements. However, the available big data storage technologies are inefficient to provide consistent, scalable, and available solutions for continuously growing heterogeneous data. Storage is the preliminary process of big data analytics for real-world applications such as scientific experiments, healthcare, social networks, and e-business. So far, Amazon, Google, and Apache are some of the industry standards in providing big data storage solutions, yet the literature does not report an in-depth survey of storage technologies available for big data, investigating the performance and magnitude gains of these technologies. The primary objective of this paper is to conduct a comprehensive investigation of state-of-the-art storage technologies available for big data. A well-defined taxonomy of big data storage technologies is presented to assist data analysts and researchers in understanding and selecting a storage mechanism that better fits their needs. To evaluate the performance of different storage architectures, we compare and analyze the existing approaches using Brewer’s CAP theorem. The significance and applications of storage technologies and support to other categories are discussed. Several future research challenges are highlighted with the intention to expedite the deployment of a reliable and scalable storage system.

Analysis

Why This Paper Matters

Big data has become a cornerstone of modern AI and analytics, with storage being the foundational layer that enables efficient processing and analysis. The exponential growth in data volume, velocity, and variety has rendered traditional relational database management systems inadequate for many big data applications. This survey addresses a critical gap in the literature by providing a comprehensive overview of storage technologies specifically designed for big data. It matters because choosing the right storage solution directly impacts the performance, scalability, and cost-effectiveness of big data analytics pipelines. By presenting a taxonomy and comparing technologies using the CAP theorem, the paper equips practitioners with a structured framework for decision-making.

Moreover, the paper highlights the shift from traditional data management systems to NoSQL technologies, which are better suited to handle the demands of big data. As industries increasingly rely on real-time analytics and AI-driven insights, the need for robust, scalable, and consistent storage solutions becomes paramount. This survey not only consolidates existing knowledge but also identifies future research challenges, thereby guiding both academic and industrial efforts toward developing more efficient storage systems.

Technical Contributions

  • Taxonomy of Big Data Storage Technologies: The paper introduces a well-defined taxonomy that categorizes storage technologies based on their characteristics, such as data model, consistency, and scalability. This taxonomy helps in understanding the landscape and selecting appropriate solutions.
  • CAP Theorem-Based Comparison: It systematically evaluates existing storage approaches using Brewer's CAP theorem, which states that a distributed system can only guarantee two out of three properties: consistency, availability, and partition tolerance. This analysis clarifies the trade-offs inherent in different storage systems.
  • Comprehensive Survey: The survey covers a wide range of storage technologies from industry leaders like Amazon (DynamoDB, S3), Google (Bigtable, Spanner), and Apache (HBase, Cassandra), providing insights into their architectures and use cases.
  • Future Research Directions: The paper identifies open challenges, such as achieving consistency and scalability simultaneously, and suggests areas for further research to enable reliable and scalable storage systems.

Results

As a survey paper, it does not present new experimental results. Instead, it synthesizes findings from existing literature and analyzes storage technologies based on the CAP theorem. The key result is the categorization of storage systems into CP (consistency and partition tolerance) and AP (availability and partition tolerance) systems, highlighting that no system can fully satisfy all three CAP properties. For example, systems like HBase and Bigtable are CP-oriented, while Cassandra and DynamoDB are AP-oriented. The paper also notes that current technologies are inefficient in providing consistent, scalable, and available solutions for continuously growing heterogeneous data, underscoring the need for further advancements.

Significance

The significance of this paper lies in its contribution to the big data community by providing a clear, structured overview of storage technologies. It serves as a valuable resource for data analysts, researchers, and practitioners who need to make informed decisions about storage infrastructure for big data analytics. By highlighting the trade-offs and limitations of existing systems, it motivates future research into novel storage architectures that can better balance consistency, availability, and partition tolerance. Furthermore, the paper's emphasis on the role of storage as a preliminary process for big data analytics reinforces its importance in enabling AI and machine learning applications that depend on large-scale data processing. Overall, this survey helps bridge the gap between theoretical concepts and practical implementation, fostering the development of more efficient and scalable big data storage solutions.

big-datastoragenosqlcap-theoremsurveydistributed-systemsdata-managementscalability

Related Papers