Why data integrity take place nowadays?
In the digital era, small files I/O and requests have taken place for nowadays technologies, such as financial blockchains, artificial intelligence, internet of things and e-commerce. These happened everywhere in our daily lives, for example, paying grocery shopping via mobile payments, withdraw cash in ATM, activity sensors in offices, factories, and public transportation stations, and many more activities that you can’t imagine. The rise of mobile apps and innovative technology transformed nearly every aspect of our lives. The incredible convenience was created to serve people and hard to believe how it been working in the last decade. However, the more convenience we have experienced, the more data we have been created. However, have you ever imagined that once if the data has been incorrectly placed or recorded, what if the data was accidentally corrupted by itself, what if the wrong data has replaced the right one? With these ideas came out, data integrity has become the major challenge for IT for the devastating threats and the fundamental of the digital era.
In the principle of designing XCubeNAS, data integrity was at the top of the list making the development team to face multiple trade-offs between safety and performance. We have crafted a unique design for user-defined pools, which fulfills different application requirements; each of them serves a different purpose: Databases, General, and Media Streaming.
The database pool is designed to have the highest performance with smaller files. The general pool fulfills requirements of 80% of applications, like backup and file sharing. The media streaming pool provides high performance for larger files and is optimized for 10GbE, Thunderbolt 3.
How XCubeNAS ensure data integrity?
Data integrity and data safety are essential for databases. QSAN’s storage management software, QSM, offers two mechanisms to ensure an absence of data failures between servers and storage. First, we set the data to write through the cache to prevent data loss from accidental disasters, like flash damage or power outage. The second, we apply ZFS checksum. ZFS’ checksum was designed to ensure data integrity. In other file systems, like EXT4, checksums are used for integrity checks and only protect against bit rots. ZFS’ checksum, however, prevents issues such as phantom writes, misdirected reads and writes, direct memory access parity errors and accidental overwrites. The checksum is accomplished by storing it in the parent block pointer instead of in the block itself that contains a self-validating SHA-256 checksum.
Additionally, ZFS also supports self-healing data by using a mirrored setup similar to RAID0. Apart from this, other file systems are not addressable if a block is corrupt unless it’s a metadata block, which means other file systems may pass bad blocks to the application and it might overwrite the accurate data in other mirrors. ZFS is a transaction-based file system, which means the operations are atomic, if operation crash or power outage occurs, the file system can quickly recover from corruption.
Beyond Security, the performance
Higher security requires higher resources. As mentioned above, ZFS is using the memory-based checksum, which claims higher memory capacity installed in the system. Also, ZFS has used ARC, Adaptive Replacement Cache algorithm, which was implemented in ZFS to outperform the traditional RAM algorithm by providing a superior hit rate. Moreover, high performance is not the only goal for ZFS, to ensure each file and blocks are correct, ZFS also requires longer computing time and RAM capacity to complete the process, which may impact the latency and impact performance.
In this summary, ZFS has its capacity to prevent silent data corruption which secures the data during transmission and storage. In any application scenario, sending and receiving the uncorrupt data is essential. That’s how the XCubeNAS was designed. Built on ZFS, it is a reliable and high-performance enterprise storage solution.