When Vancouver-based venture studio Axiom Zen launched CryptoKitties in October of 2017, it was intended to be a playful experiment of early stage blockchain technology. CryptoKitties users breed and trade unique digital kittens, upvote and engage with the community, and revel in the still-novel joy of a fully-functional application built on a distributed network. Conceptually somewhere between Neopets and futures trading, the disarmingly simple CryptoKitties game captured imaginations with the emergent blockchain community and beyond. Before long, the CryptoKitties craze had spiraled beyond anyone’s expectations— for better and worse.
By early December, CryptoKitties had turned viral sensation, sending prices for some kittens well into the six figures, earning slightly bemused mentions from across mainstream news media, and clogging up the Ethereum network with so many transactions — a sixfold increase in total network requests in the first week of December alone — that many feared the entire Ethereum blockchain would grind to a halt before it ever even had a chance to achieve its promise.
In response to what was unfolding to be an industry-wide dilemma, an impromptu taskforce of Ethereum developers from projects like MetaMask, Infura, and Grid+ came together and joined the CryptoKitties team in formulating short-term optimizations and longer-term scaling solutions that not only alleviated the immediate danger, but laid out roadmaps towards a more functional future. Perhaps even more importantly, the saga highlights the remarkably collaborative and cooperative nature of the blockchain space and developer community.
In their own words, here is the inside story on the CryptoKitties Congestion Crisis of 2017…
Bryce Bladon, Co-Founder of CryptoKitties: “Although we wouldn’t have pursued the CryptoKitties project if we didn’t want it to be a success, we did not expect it to catch fire quite like it did. And I mean that in the metaphorical and, potentially, the literal sense. We soft launched an alpha at ETH Waterloo, the most basic version of the game possible on the Rinkeby Testnet. All signs were great. We thought we’d caught all the bugs and issues. We were expecting to have to deal with things like scaling at some point. We did not expect it to be in the first week…”
Dan Finlay, MetaMask: “When I went to ETH Waterloo, I met this scrappy team who gave me a business card printed on the back of a Pokemon card. I’ve always said about this space: it’s so young and immature that we should be trying small, stupid things first so we can learn how to do it well. Like when the DAO happened, for example. The stakes were so high that it made no sense. So I was into CryptoKitties, and tweeted about the project in exchange for naming a cat. They clearly had some social media savvy and were getting in touch with lots of people, pushing the idea.”
Bryce Bladon, CryptoKitties: “Less than a week later, we launched, and it was immediately clear that we were on an upward trajectory. By the end of the week, we knew that scaling was going to be our number one issue for the foreseeable future.”
Dan Finlay, MetaMask: “When CryptoKitties finally happened, they launched on a Friday, and we had scheduled a Consensys design thinking retreat for that week. We were all in New York, basically checked out, with no one on staff. While we were there, people kept talking about CryptoKitties, showing us their collections, and very quickly things escalated to the point that people we met were paying off chunks of their college debt by flipping cats. Myself, I was just trying to breed a funny looking cat! Right when that mania started happening, we began getting reports of major congestion, problems, nothing working.”
Michael Wuehler, Infura: “Initially, CryptoKitties made the decision to depend completely on MetaMask, and funnel everybody through the MetaMask browser to use the dApp, and all of that traffic goes through Infura nodes on the backend. As a result, we were supporting all of that traffic. The network started to become really clogged. All of the blockchain read activity caused our daily requests to explode from 2 Billion/day to over 4 Billion/day… overnight! In addition, the pending transaction queue kept rising, which means the network wasn’t processing fast enough to clear out new requests. When that happens, the miners reset the gas price to make transactions cost more, and then market forces will control the queue. So, all of a sudden, the fees to buy the kitty cost more than the asset itself. That was not good, and was the trigger that we had a problem. That threatened to kill CryptoKitties entirely, because when the fees are so inhibitive, it’s game over.”
Alex Miller, Grid+: “I heard about CryptoKitties before it launched. I was one of the first users on there. They made all the artwork, the website, wrote up the smart contracts. It was the first kind-of mainstream app that just appeared and was usable from day one. The scope of it was not huge, it’s pretty well executed. From the contract side, there are a few inefficiencies, but overall, it’s well done. It was a lot of fun, but it clogged the network pretty quickly when it went viral. It was basically unusable, as was the whole network at that point.”
Dan Finlay, Metamask: “The network hadn’t been used that heavily for a while. There were no crowdsales that totally broke everything, because a lot of them had gotten a lot more tactical about gas and congestion. We had seen what a bad backlog looked like before, and they always came in spikes. Our workaround for our users was always — just wait! A couple days at the most. This seemed like it could become a much bigger problem. There were headlines saying the Ethereum network had been destroyed by CryptoKitties. That was what gave us a wake-up moment. This wasn’t another couple day backlog, like the Bancor or Status sales. This may be the new normal for the network. It felt like a real crisis of the network for many people. There’s always someone who is going to try to pay 1Gwei for a kitty!”
Bryce Bladon, CryptoKitties: “The response was a mix of elation and panic. Blockchain is a technology we absolutely believe in. It can change the way many legacy structures in society function. But at the same time, there is this sense of achievement — and horror — that we might have broken one of the world’s most remarkable examples of blockchain. As much as we want to push the technology forward, we didn’t want to shove others out of the way to do it. Especially as community and cooperation being something we so strongly believe in. We see it like other people in this space are pushing it forward alongside us, so the last thing we wanted to do was step on their toes. That said, Ethereum, MetaMask, and our own servers were the three major structures that dictate how will this thing would work. It’s very scary to have two of those structures outside of our control — which is why cooperation was so important! Every interaction we had with the Ethereum Foundation has been nothing but positive. We had been working with MetaMask as a tool since the Alpha, so we had been in touch with them. We reached out to everyone that we could.”
Dan Finlay, Metamask: “The CryptoKitties team reached out with some degree of like, ‘well, what do we do now?’ We were lucky that MetaMask, Infura, and Kyokan, who work on MetaMask as well, were all in one place at the design retreat. There was an evening where everyone else was partying, playing drinking games, and there were five of us sitting in a corner breaking down the problem until past midnight. It was a fun moment. We were in touch with them constantly, through video chats, Slack, tweets. I looked at their code, and not all of it needed to be on-chain, you could cut the number of transactions in half that way. The main issue was that users were sending a transaction that was underpriced in gas, as the market had moved up without them, and were waiting for days for it to go through. But they didn’t know that, and it just seemed like the whole Ethereum Network was broken.”
Bryce Bladon, CryptoKitties: “We did some very basic things from the user side to communicate some challenges. Blockchain tech moves much slower than your typical internet, so a lot of people who were new to the space were very confused and concerned when things weren’t instantaneous. Education became a key short term priority. Our onboarding process had to anticipate the problems. We added a feature to the site that live-recorded the status of the Ethereum Network. We added a transactional queue. Rufflet, which we built for the ETH Waterloo hackathon, has proven an unbelievably useful analytics tool. A lot of it was related to the user experience and making sure people aren’t repeatedly pounding the network when it’s a bad time or using terrible gas fee calculations, communicating the network choke points and times…”
Dan Finlay, Metamask: “Short term, they added indicators to calm people down. That’s pretty much all they could do, they were blocked by us. We came up with a small solution on MetaMask: For transactions that were taking a long time, we showed a button that let people resubmit the transaction with higher gas prices. That little button basically solved the whole problem. It allowed people to participate in the gas auction. Long term, it’s important that we’re empowering users to not be passive passengers in the games of crypto-economics, but to be active players. We learned that you can’t simplify away the gas auction. At times, people need the power to bid higher.”
Alex Miller, Grid+: “They were just trying to get through the day with some optimizations that would reduce the overhead for users, but it had triggered an idea I had been playing about with some months earlier for a more long-term fix. If we have even a few successful applications all on the same network, it will quickly become untenable. I had been kicking around this idea for Trusted Relayers, but the situation increased the urgency of it. The idea is that you connect multiple sidechains to the Ethereum MainNet, and users can move between the chains. It’s very similar to Plasma, but there’s a difference that you’re only relaying headers and not state. CryptoKitties liked the idea of moving to a sidechain at some point. It’s a technique we’re very interested in implementing at Grid+ for our physical agent devices, and is just one of many promising scaling solutions being worked on as we speak.”
Bryce Bladon, CryptoKitties: “One of the big things we learned was that although a lot of the very interesting aspects of blockchain technology have to do with decentralization. At the same time, we believe that certain centralized features can be a way to alleviate decentralized demand, and having much of the transactional weight tied to a sidechained scaling solution is a very interesting option. While interacting with the blockchain is what makes this interesting, it isn’t necessarily everything. We found that there were a lot of users who buy and breed cats — stuff that requires directly transacting on the network — but there are also users who just wanna browse or upvote or engage with the community.”
Dan Finlay, MetaMask: “What we worked on that week was not like a scaling solution in the sense that people often think about scaling, which is the total throughput of the network. But this is this other kind of scaling, like the user experience of these solutions. I think about this all the time. We come up with these cryptoeconomic games that are theoretically perfect, but if people don’t know how to play them, then they’re useless. MetaMask is in a funny position in that we don’t work at the protocol layer, but we are totally responsible for how people perceive the protocol. And perception of the protocol often ends up defining it.”
Alex Miller, Grid+: “After that, the initial wave died down a bit. There’s some level of permanence to the CryptoKitties user base, but it’s much less than the period in which it went viral. Now, the network is in something of an equilibrium. Fees are high, but they’re not increasing drastically right now. All it will take is another viral app to push it over the barrier, but for now we’re okay. A lot of scaling solutions are coming together, so hopefully we see some improvements in the infrastructure before the network sees another wave of adoption.”
Bryce Bladon, CryptoKitties: “Some say these growing pains are necessary. Others might say they could have been less painful. And we’re not through all of it yet. We’re looking at launching in China in the next little while. We hope the network has matures enough, and we have our own solutions for optimization, but all we can do is be better equipped to put out fires, not necessarily stop the fires from ever existing.”
Michael Wuehler, Infura: “One of the things that makes the Ethereum community different: Everybody, even if they’re working on potentially competing projects, works together to figure stuff out. It goes against the capitalist, startup world where everyone’s trying to beat out everyone. I’ve never seen anything like this in any other crypto-community this large.”
Alex Miller, Grid+: “I think the collaborative nature is unique. It’s much more collaborative than most spaces, and much of that has to do with shared economic incentives. Everyone who owns Ether is incentivized to make the system better. When you put a bunch of smart people with technical skill sets into the same ecosystem with shared incentives, you can iterate very quickly and build cool things. I think that’s a big reason for why we’re seeing the adoption of Ethereum go up very quickly.”
Dan Finlay, MetaMask: “In the blockchain world, we all want to see it work. We’re sharing this blockchain, this source of truth, so we need to see it working smoothly. It creates a common space where we have incentive to work out issues together. If other people had known how to contribute, they would have been happy to do so.”
Bryce Bladon, CryptoKitties: “We designed CryptoKitties in such a way that other projects could flourish in the space as well. It’s a public API, people can look up the ERC-721 token on Github. We’ve seen all sorts of extensions and fan projects built on top of ours. The community drove the initial groundswell for this project and every major success and obstacle has been driven by the fervor of the community, so we’re trying to leverage that more effectively and formalize some of the tools and processes for that. The whole experience has just been phenomenal. Not just in terms of what people have built on CryptoKitties, but that all the people we’ve met, potential competitors even, are acting like friends. We’re all moving forward and it’s a really good feeling. Once it gets stagnant and there are no major problems to solve, that’s when things might start getting a little more cutthroat. But for now, developers gonna develop, and they’re gonna help each other along the way. And I couldn’t be happier about that.”