![]() ![]() MinIO maintains a helpful erasure code calculator to help determine requirements in your environment. Erasure coding can be configured for different ratios of data to parity blocks, resulting in a range of storage efficiency. For example, for 10 PB of data replication would require more than 30 PB of storage, whereas object storage would require 15-20 PB to securely store and protect the same data using erasure coding. Compared to the N-way replication shown above, with erasure coding configured for n = 5 and k = 3, a distributed storage system could tolerate the loss of 2 drives, while improving storage efficiency by 80%. If k indicates potential failure, then k < n, and with MDS codes the system can guarantee to tolerate n - k drive failures, meaning that k drives are sufficient to access any object.Ĭonsidering an object that is M bytes in size, the size of each coded object is M / k (ignoring the size of metadata). In object storage, the unit of data to be protected is an object. Erasure coding stripes data and parity across drives. There are many different erasure coding algorithms, and Maximum Distance Separable (MDS) codes such as Reed-Solomon achieve the greatest storage efficiency. ![]() If a drive or node fails or data becomes corrupted, the original data can be reconstructed from the blocks saved on other drives and nodes.Įrasure coding is able to tolerate the same number of drive failures as other technologies with much better efficiency by striping data across nodes and drives. Horizontally scalable distributed storage systems rely on erasure coding to provide data protection by saving encoded data across multiple drives and nodes. It splits data files into data and parity blocks and encodes it so that the primary data is recoverable even if part of the encoded data is not available. When a drive fails, the system will place itself in read-only mode at reduced performance while it fully copies an intact drive onto a new drive to replace the failed drive.Įrasure coding is applied to data protection for distributed storage because it is resilient and efficient. Not only is replication inefficient in terms of storage utilization, it is also operationally inefficient when it recovers from failure. Many distributed systems use 3-way replication for data protection, where the original data is written in full to 3 different drives and any one drive is able to repair or read the original data. RAID is a good solution for data protection on a single node, but fails to scale due to time consuming rebuild operations required to bring failed drives back online. More complex technologies such as RAID5 and RAID6 provide the same fault tolerance while reducing storage overhead. Mirroring and replication rely on one or more complete redundant copies of data - this is a costly way to consume storage. Traditionally, different types of RAID technologies or mirroring/replication were used to provide hardware fault tolerance. This blog post explains how erasure coding satisfies enterprise requirements for data protection and how it is implemented in MinIO.ĭata protection is essential in any enterprise environment because hardware failure, specifically drive failure, is common. ![]() Docs Blog Resources Partner Pricing DownloadĮrasure coding is a key data protection method for distributed storage systems. VMware Discover how MinIO integrates with VMware across the portfolio from the Persistent Data platform to TKGI and how we support their Kubernetes ambitions. HDFS Migration Modernize and simplify your big data storage infrastructure with high-performance, Kubernetes-native object storage from MinIO. Splunk Find out how MinIO is delivering performance at scale for Splunk SmartStores Veeam Learn how MinIO and Veeam have partnered to drive performance and scalability for a variety of backup use cases. No need to move the data, just query using SnowSQL. Snowflake Query and analyze multiple data sources, including streaming data, residing on MinIO with the Snowflake Data Cloud. Commvault Learn how Commvault and MinIO are partnered to deliver performance at scale for mission critical backup and restore workloads. Integrations Browse our vast portfolio of integrations SQL Server Discover how to pair SQL Server 2022 with MinIO to run queries on your data on any cloud - without having to move it. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |