Web3 is booming, and Arweave is becoming a popular infrastructure choice for developers. PermaDAO is a community where everyone can contribute to the Arweave ecosystem. It's a place to propose and tackle tasks related to Arweave, with the support and feedback of the entire community. Join PermaDAO and help shape Web3!

Proofreader: Orange @ Contributor of PermaDAO

Reviewer: Viya @ Contributor of PermaDAO

Yeah, so let's dive in. I’ll talk about how the Arweave mechanism works. So today, we're going to be talking about scaling Arweave and discussing the scaling of on-chain history from Bitcoin to Arweave. We'll delve into the history of storing human history on blockchains, unlock on-chain data, and explore where we go from here.

The history of storing human history on blockchains

Human history has always been part of blockchains. To see how, we have to go back to the first block of the first decentralized blockchain, which is Bitcoin's Genesis block. Bitcoin's Genesis block had a single transaction in it, known as a coinbase transaction. This special transaction in the protocol awarded the producer of the block with the block rewards. In this case, it awarded Satoshi Nakamoto 50 BTC. Another special property of the coin-based transaction was that it allowed the block producer to write 100 bytes or 100 characters of information to the blockchain. In this case, Satoshi took the opportunity to write the following quote: "The Times 3rd of January 2009 Chancellor on the brink of a second bailout for banks." In doing so, he inscribed human history into the very first block of the first blockchain.

Bitcoin launched in January 2009 and proceeded to get a number of updates and improvements. Even Satoshi himself stepped back from the protocol during this time, up until about 2015. During this period, a conflict started brewing within the Bitcoin community. Half of the community wanted to increase the block size to allow more transactions in Bitcoin, while the other half wanted to keep it small to preserve the decentralized properties of the network. This conflict stalled all development on Bitcoin for about two years, called the block size wars. It ended with the big blockers splitting off and creating a new chain called Bitcoin Cash, while the existing mainline of Bitcoin continued with a soft fork called SegWit.

Now, what did SegWit do? Well, you can see here on the left that before SegWit, Bitcoin blocks had their coinbase transactions, followed by a number of transactions moving tokens from one address to another, and their signatures. 

After SegWit, somehow they did this with a soft fork, which meant it was backwards compatible with existing mining software (some sort of magic there). SegWit moved all of the transactions to the top of the block and increased the overall block size while still limiting the transaction portion to 1 megabyte, preserving the original intent of the protocol. What it did was move the signature data and metadata into the witness data, increasing the amount of information that Bitcoin could store that wasn't purely transaction data. This ultimately led to some pretty interesting things down the road, like very compressed images being added to the witness data of the Bitcoin blockchain.

Going back to 2017, a Ph.D. student by the name of Sam Williams was reading some very depressing books like "Gulag Archipelago" and "1984". He worked himself up into a frenzy and decided to create a blockchain to permanently preserve the world's knowledge. He was reading these books about how authoritarian regimes manipulate their society by manipulating history. So Sam put together a team sometime in 2017, and a year later, Arweave Mainnet was launched. And if you thought I was kidding about the depressing books, it was launched on June 2018, which was the 39th anniversary of the publishing of George Orwell's "1984."

Now, Arweave had some impressive stats when it launched. It had 1,000 transactions per block and operated at 2-minute block times. Each transaction could be 5.8 megabytes in size. That means a single Arweave transaction could store an entire Bitcoin block - witness data and transaction data together. Just like Bitcoin, Arweave progressed with updates and improvements over the next couple of years until March 2020.

At that point, Arweave unlocked very large transactions. The maximum transaction size became 2 to the power of 256 minus 1, which is a theoretical limit, limited only by the number of atoms in the known universe. From a protocol standpoint, this was a significant scaling up. However, there's a practical limit due to the amount of data that miners provide to the network and the available bandwidth and the amount of bandwidth that exist does actually constrain the size. But from a protocol standpoint, it scaled to very large transactions.

So, how did they achieve this? Well, this is where it gets a bit tricky without the animations. To understand how Arweave did it, there are two different security mechanisms we need to look at. One is consensus, where all participants in the network, in the protocol, need to come to an agreement on the current state of the protocol. The other mechanism is mining, traditionally performed by individual miners. A work famously done in Bitcoin is proof of work. Just to be able to participate in the network.

So we can see here for Bitcoin, the consensus mechanism is longest chain, which is how the participants in the network come to an agreement on what the current state and witness data is for a Bitcoin block. And then, famously, the mining mechanism is proof of work, where they calculate a lot of SHA-256 hashes. So this is an example of the longest chain. Occasionally, because of the randomness in mining, two blocks can be produced at the same time, and at this point, the participants in the network need to decide which one to use. The way they decide that is by waiting for one of the chains to become longer than the other, and then using that as the official state of the network.

What's really interesting is this. So we have our Bitcoin blocks here; they have their little coin-based transactions. You can see the transaction data, the gray block on top, and then the witness data, the blue at the bottom, and they're all around 4 megabytes in size. 

What happens on Bitcoin is that you can kind of see it moving away here, but once the node, the miner, has used the witness data to verify the transactions, they can just throw it away. There's no reason for them to keep it beyond the very first time that they process a block, and that's by design. It's to keep the system requirements for operating a node low so that there can be a lot of nodes on Arweave or on Bitcoin. But Arweave is fixing this. For a protocol that wants to store the human, like, you know, all of the world's knowledge, it needs to have stronger incentives to store data, right?

So how does Arweave fix this? What changes? One of the key innovations that Arweave did was that it changed the mining mechanism. Instead of just blind hashes, Arweave uses something called SPoRa, which is "sync proof of random access." It's a cryptographic proof that the miners do to prove that they're storing small chunks of Arweave's data, 256 kilobytes at a time. What this did was allow the protocol to move the security from the data, from the consensus mechanism that required everyone in the network to have it, to come to an agreement, to the mining mechanism, which meant miners could all individually prove that they were storing data, greatly improving the system.

Here's an example of Arweave blocks, and you can see that the data portion of the blocks varies in size. Every block on Arweave can store different amounts of data. What the Arweave protocol does is every second, the protocol ticks; it gives a new range for miners to mine. So that's what these little blue rectangles are in the data portion. Every second, there's a new range of data for it to check for proofs. And occasionally, like we're seeing here, there's a little pink block that is a proof that allows the miner to produce a new block. And when the miner gets a winning proof, it makes a new block and earns block rewards.

So you can sort of see the issue there, or not the issue, but the interesting thing to note is that the network came to consensus about the transactions, but the data isn't there yet, which is a little bit surprising.

So you can see it's a gray box, and what happens over time is as mining continues, the data starts to stream in from different areas of the network, and with each step, with each new block that's produced, the miner trades with other miners to get that data and fill it all in.

You can see that the mining mechanism on Arweave starts to allow the miner to generate proofs from that new data as it comes in.

So as blocks are added, you can see where the third block is added now, and the miner is able to fully mine that data. And as the new blocks are added, you can see the data starting to stream in on them as well.

Unlocking on-chain data at any scale

What this does is it incentivizes miners to trade data with one another; they all want the new data because it allows them to perform more proofs and have a better chance of earning rewards. So what happens is the data added to Arweave gets replicated all around the network, hundreds of times, because of the mining mechanism. 

So this is how Arweave unlocks on-chain data at any scale. It also incentivizes the replication of data and makes Arweave data and transaction censorship-resistant. So you might say, "Look, DMac, I've used Arweave before; I just post the transaction, and it goes. I don't know what you're talking about—separating the transaction data and stuff like that."

  Well, the reality is that you probably posted your transaction to a gateway like arweave.net, and it did this under the covers as a convenience for you. It posted the transaction to a new block, and once it was mined, it sent the data along to all the miners in the network, which is great, but it also introduces the opportunity for censorship.  

Say the gateway didn't like the transaction; it saw something in your data and thought, "I don't want to post this.” It could just throw it out. If that was the whole story, Arweave would have a centralized gatekeeper to its protocol and wouldn't be completely decentralized. But the good news is Arweave is, and you can skip the gateway; you don't have to use it. You can just post your transactions directly to the network, and once they're included in a block, you can add the data, and it'll be replicated around the network.

Where we go from here

This is really important because it keeps gateways and services built on top of Arweave honest. So Arweave development continued, and a month after the large transactions were introduced, Arweave rolled out ANS-102 bundles (apologies for the layout here). What bundles did is allowed these very large transactions to include many other transactions. So Arweave started to scale in two ways: large transactions and large volumes of transactions. This was useful, but it wasn't until a year later when ANS-104 came out, optimizing the ANS-102 standard, improving the performance of bundles, their space efficiency, and the compute required to create them. This resulted in a huge unlock for the protocol.

ANS-104 bundles launched around the same time as NFT Summer on Solana, and people were uploading massive collections of 10,000 NFTs and using bundles to do it. You can see the growth of Arweave's dataset since then.

So that brings us up to today. We are maybe, I would say, weeks away from the 2.7 upgrade to Arweave. Again, it's going to include optimizations for bundles; they're going to be even less computationally expensive to produce. And it also introduces the same censorship-resistant mechanism that block transactions have to bundles as well. 

So yeah, I think the point that I want to make here is that the data model of Arweave has been stable for the last three years since the introduction of bundles and large transactions. The types of updates we've seen to Arweave have all been about improving performance, censorship resistance, and ultimately the decentralization of the protocol.

In review, recording human history has been part of blockchain since the beginning, but that frontier has really moved from Bitcoin to Arweave at this point with its scalable on-chain data. And then with bundles and large transactions since April of 2020, Arweave has been able to scale in multiple dimensions to be the ultimate location for on-chain data. Going forward, I expect to see more of the same with incremental updates to the performance and decentralization of Arweave. Thank you very much!

  🔗 More about PermaDAO ：Website | Twitter | Telegram | Discord | Medium | Youtube