Decentralization Storage: The Long Journey from Concept to Application

Storage was once one of the hottest sectors in the blockchain industry. Filecoin, as the leading project of the last bull market, once had a market cap exceeding $10 billion. During the same period, Arweave touted permanent storage as its selling point, reaching a peak market cap of $3.5 billion. However, as the practicality of cold data storage is being questioned, whether decentralization storage can truly be realized has been cast into doubt.

Recently, the emergence of Walrus has brought new attention to the long-silent storage track. The Shelby project launched by Aptos in collaboration with Jump Crypto aims to elevate decentralized storage to new heights in the hot data field. So, can decentralized storage rise again, or is it just another round of speculation? This article will analyze the evolution of decentralized storage from the development paths of four projects: Filecoin, Arweave, Walrus, and Shelby, and explore its future development prospects.

Filecoin: The Name of Storage, The Reality of Mining

Filecoin is one of the representative projects that emerged early on, with its development direction centered around Decentralization. This aligns with the common characteristics of early altcoins – seeking decentralized application scenarios in various traditional fields. Filecoin combines storage with Decentralization, focusing on addressing the trust issues of centralized storage services. However, certain compromises made to achieve Decentralization have instead become the pain points that later projects attempt to resolve.

To understand that Filecoin is essentially just a mining coin, one needs to be aware of the objective limitations of its underlying technology IPFS in the application of hot data.

IPFS: Transmission bottleneck of Decentralization architecture

IPFS( InterPlanetary File System) was introduced around 2015, aiming to replace the traditional HTTP protocol through content addressing. However, the biggest drawback of IPFS is its extremely slow retrieval speed. In today's world, where traditional data services can respond in milliseconds, retrieving a file via IPFS still takes several seconds, making it difficult to promote in practical applications and explaining why it is rarely adopted by traditional industries, except for a few blockchain projects.

The underlying P2P protocol of IPFS is mainly suitable for "cold data", which refers to static content that does not change often, such as videos, images, and documents. However, when it comes to handling hot data, such as dynamic web pages, online games, or AI applications, the P2P protocol does not have a significant advantage over traditional CDN.

Although IPFS itself is not a blockchain, its directed acyclic graph ( DAG ) design concept closely aligns with many public chains and Web3 protocols, making it inherently suitable as a foundational building framework for blockchain. Therefore, even in the absence of practical value, it is sufficient as a foundational framework for carrying blockchain narratives. Early projects only need a runnable framework to launch grand visions, but as Filecoin develops to a certain stage, the limitations brought by IPFS begin to hinder its progress.

The logic of mining coins under the storage cloak

The original intention of IPFS's design is to allow users to not only store data but also be a part of the storage network. However, without economic incentives, it is difficult for users to voluntarily use this system, let alone become active storage nodes. This means that most users will only store files on IPFS but will not contribute their own storage space or store others' files. It is against this backdrop that Filecoin was born.

In the token economic model of Filecoin, there are mainly three roles: users are responsible for paying fees to store data; storage miners receive token incentives for storing user data; and retrieval miners provide data when users need it and receive incentives.

This model has potential malicious space. Storage miners may fill garbage data after providing storage space to earn rewards. Since this garbage data will not be retrieved, even if lost, it will not trigger the penalty mechanism for storage miners. This allows storage miners to delete garbage data and repeat this process. Filecoin's proof of replication consensus can only ensure that user data has not been privately deleted, but cannot prevent miners from filling garbage data.

The operation of Filecoin largely relies on miners' continuous investment in the token economy, rather than on the real demand for distributed storage from end users. Although the project is still iterating, at this stage, the ecological construction of Filecoin aligns more with the "mining coin logic" rather than the "application-driven" definition of storage projects.

Arweave: A Double-Edged Sword of Long-Termism

If the design goal of Filecoin is to build an incentivized, provable Decentralization "data cloud" shell, then Arweave takes an extreme direction in storage: providing the capability for permanent storage of data. Arweave does not attempt to build a distributed computing platform; its entire system revolves around a core assumption - important data should be stored once and remain on the network forever. This extreme long-termism makes Arweave fundamentally different from Filecoin in terms of mechanism, incentive model, hardware requirements, and narrative perspective.

Arweave takes Bitcoin as a learning object, attempting to continuously optimize its permanent storage network over long cycles measured in years. Arweave does not care about marketing, nor does it care about competitors and market development trends. It is only constantly moving forward on the path of iterating its network architecture, not caring even if no one pays attention, because this is the essence of the Arweave development team: long-termism. Thanks to long-termism, Arweave was highly sought after during the last bull market; also because of long-termism, even when it hits rock bottom, Arweave may still survive several rounds of bull and bear markets. The only question is whether there will be a place for Arweave in the future of Decentralization storage. The value of permanent storage can only be proven over time.

Since version 1.5, Arweave's mainnet has been working hard to allow a broader range of miners to participate in the network at minimal cost and to incentivize miners to maximize data storage, thereby continuously enhancing the robustness of the entire network, despite losing market attention up to the recent version 2.9. Arweave is well aware that it does not align with market preferences; therefore, it has taken a conservative approach, not embracing the miner community, leading to a complete stagnation of the ecosystem. It upgrades the mainnet at minimal cost while continuously lowering hardware thresholds without compromising network security.

Review of the upgrade path from 1.5 to 2.9

The Arweave 1.5 version exposed a vulnerability that allowed miners to rely on GPU stacking instead of real storage to optimize block production chances. To curb this trend, version 1.7 introduced the RandomX algorithm, limiting the use of specialized computing power and instead requiring general-purpose CPUs to participate in mining, thereby weakening computing power centralization.

In version 2.0, Arweave adopts SPoA, transforming data proofs into a concise path of Merkle tree structure, and introduces format 2 transactions to reduce synchronization burdens. This architecture alleviates network bandwidth pressure, significantly enhancing the collaborative capabilities of nodes. However, some miners can still evade the responsibility of holding real data through centralized high-speed storage pool strategies.

To correct this bias, version 2.4 introduced the SPoRA mechanism, which incorporates global indexing and slow hash random access, requiring miners to genuinely hold data blocks to participate in effective block production, thereby weakening the effects of hash power stacking from a mechanism perspective. As a result, miners began to focus on storage access speed, driving the application of SSDs and high-speed read-write devices. Version 2.6 introduced hash chain control to regulate the block production rhythm, balancing the marginal benefits of high-performance devices and providing fair participation space for small and medium-sized miners.

Subsequent versions further strengthen network collaboration capabilities and storage diversity: 2.7 introduces collaborative mining and pool mechanisms to enhance the competitiveness of small miners; 2.8 launches a composite packaging mechanism, allowing large-capacity low-speed devices to participate flexibly; 2.9 introduces a new packaging process in replica_2_9 format, significantly improving efficiency and reducing computational dependencies, completing the closed loop of data-oriented mining models.

Overall, Arweave's upgrade path clearly presents its long-term strategy focused on storage: continuously resisting the trend of computing power centralization while lowering the participation threshold to ensure the possibility of the protocol's long-term operation.

Walrus: A New Attempt at Hot Data Storage

Walrus's design concept is completely different from Filecoin and Arweave. Filecoin's starting point is to create a decentralized verifiable storage system, at the cost of cold data storage; Arweave's starting point is to build an on-chain library of Alexandria that can permanently store data, at the cost of too few scenarios; while Walrus's starting point is to optimize the storage costs of hot data storage protocols.

Magic-modified error-correcting code: Cost innovation or new wine in an old bottle?

In terms of storage cost design, Walrus believes that the storage overhead of Filecoin and Arweave is unreasonable. Both of the latter adopt a fully replicated architecture, whose main advantage is that each node holds a complete copy, providing strong fault tolerance and independence among nodes. This type of architecture ensures that even if some nodes go offline, the network still has data availability. However, this also means that the system requires multiple copies of redundancy to maintain robustness, thus driving up storage costs. Especially in the design of Arweave, the consensus mechanism itself encourages nodes to redundantly store data to enhance data security. In contrast, Filecoin is more flexible in cost control, but at the price that some low-cost storage may carry a higher risk of data loss. Walrus attempts to find a balance between the two, with its mechanism controlling replication costs while enhancing availability through structured redundancy, thereby establishing a new compromise path between data availability and cost efficiency.

The Redstuff created by Walrus is a key technology for reducing node redundancy, originating from Reed-Solomon(RS) coding. RS coding is a very traditional erasure code algorithm, and erasure coding is a technique that allows for data sets to be doubled by adding redundant fragments(erasure code), which can be used to reconstruct the original data. From CD-ROMs to satellite communications to QR codes, it is frequently used in daily life.

Erasure coding allows users to take a block, for example, 1MB in size, and then "expand" it to 2MB, where the extra 1MB is special data known as erasure coding. If any byte in the block is lost, users can easily recover those bytes using the code. Even if up to 1MB of the block is lost, the entire block can still be recovered. The same technique enables computers to read all the data on a CD-ROM, even if it has been damaged.

The most commonly used is RS coding. The implementation method is to start from k information blocks, construct related polynomials, and evaluate them at different x coordinates to obtain the encoded blocks. Using RS erasure codes, the probability of randomly sampling large chunks of missing data is very low.

What is the biggest feature of Redstuff? By improving the erasure coding algorithm, Walrus can quickly and robustly encode unstructured data blocks into smaller shards, which are distributed and stored in a network of storage nodes. Even if up to two-thirds of the shards are lost, the original data block can be quickly reconstructed using partial shards. This is made possible while maintaining a replication factor of only 4 to 5 times.

Therefore, it is reasonable to define Walrus as a lightweight redundancy and recovery protocol redesigned around Decentralization scenarios. Compared to traditional erasure codes ( such as Reed-Solomon ), RedStuff no longer pursues strict mathematical consistency, but makes realistic trade-offs regarding data distribution, storage verification, and computational costs. This model abandons the immediate decoding mechanism required for centralized scheduling, instead adapting by verifying through on-chain Proof whether nodes hold specific data replicas, thereby accommodating a more dynamic and marginalized network structure.

The core design of RedStuff is to split data into two categories: primary slices and secondary slices. Primary slices are used to restore the original data, and their generation and distribution are subject to strict constraints, with a recovery threshold of f+1, requiring 2f+1 signatures as availability endorsement. Secondary slices are generated through simple operations like XOR combinations, serving the purpose of providing elastic fault tolerance and enhancing the overall system robustness. This structure essentially reduces the requirements for data consistency—allowing different nodes to temporarily store different versions of data, emphasizing a practice path of "eventual consistency." Although similar to the relaxed requirements for retroactive blocks in systems like Arweave, achieving some effect in reducing network burden, it simultaneously weakens the assurance of immediate data availability and integrity.

It cannot be ignored that RedStuff has achieved