Elemental Chat Public Test: Successes & Learnings

A week and a half ago, we invited the wider public to try out a proof-of-concept app called Elemental Chat that was designed to stress-test the Holochain framework and the Holo hosting network built on it. We finished the test yesterday, and from the entire team’s perspective, it went wonderfully and also provided us with excellent learnings.

- 300 Host HoloPorts

- 5,500+ Peers (User Chains hosted)

- 1,100+ Peak Simultaneous Peers

- 33,000+ Chat Messages

- 150,000+ Chain Entries

- 400,000+ DHT ops gossiped

So what do these numbers mean? During the test, we saw more than 5,500 people sign up and send over 33,000 messages during the course of about nine days. For the technical minded, that resulted in an estimated 150,000 chain entries. At one point, over 1,100 people were online at once, chatting simultaneously. In order to make all that happen, 300 HoloPorts were hosting the data and providing the compute power for the app.

While we did see a definite slow down toward the end, performance exceeded our expectations. It showed us that Holo is already able to handle a reasonable amount of traffic for a typical app. And this is without introducing sharding, or the last few weeks’ worth of improvements to Holochain, or any of the future optimizations we’ve got planned.

Elemental Chat is a pretty bare-bones chat app, so if you tried it out, it might not have seemed special. It isn’t as fancy as Slack or Discord. But under the hood, there’s a lot happening to bridge the traditional web to the P2P world.

And as far as we know, our approach is unique. Holochain + Holo hosting can blend the offline-friendliness of self-run nodes with the portability of web access, all while maintaining users’ sovereignty and making their experience pleasant and familiar. You can read more about that in the next Dev Pulse. 

This is a work in progress, so there were some unknowns before this test about how long Elemental Chat could run or how many people could be supported in a real world context, because of the nature of all of the components that were being tested simultaneously. The fact that we were able to reach over 5,000 users is one metric—but a second fact is new users were able to keep signing up successfully and syncing with the entire database of messages throughout the test. This is an equally important demonstration of the stability of all these pieces working together coherently. Overall, it held up remarkably well.

Actually, sharding is built and currently being thoroughly tested. However, we haven’t enabled it on Holochain by default and we didn’t have it configured for this test of Element Chat. In technical terms, this meant that every node was holding 100% of the DHT and every HoloPort was thus gossiping 100% of the DHT. That’s a lot of network traffic. And yet it still worked quite well—and most importantly we saw that gossip was always eventually consistent, as it must be.

That’s a great indicator because we were able to remain solidly stable throughout the test and reach these numbers without the primary scaling component of Holochain. Future tests will be able to go far beyond this in supporting much greater scalability.

To Cloud: With some UI optimizations and time to create a production app we could provide a comparable experience to a centrally hosted chat. There's still some higher latency for us because we have extra round trips for cryptographic signing, and we're largely hosted on home networks instead of high speed backbones. But for chat, it feels fairly responsive and we wouldn’t be surprised if speeds could be increased further in more local contexts.

To Blockchain: Some people commented that there are blockchain apps with 600k users in a day. But notice those systems are trading centralization for performance and only running a dozen nodes to produce those speeds. Since Holochain is P2P every user has their own chain, and is a node. So to have 5,000 chains running on 300 Hosts on home networks, is quite different than 600k people sending data to a dozen supernodes in data centers. In fact, even at this early stage of development these home networks are, in fact, good enough for the vast majority of applications. 

TPS: Transactions per second is not a very useful frame to talk about on Holochain since we enable simultaneous changes which do not block each other at all. But there are still limitations of throughput for gossip and reaching consistency across nodes on the DHT, so when we actually turn on sharding on this version of Holochain for future tests, then we'll talk more specifically about what kind of throughput we're seeing.

So while we don’t know what the Elemental Chat network’s peak TPS was, we do know that messages were delivered quickly and reliably, even when 1,000 people were online and chatting at once, and no messages were lost. In other words, it worked like a chat app should. This may sound like an uninteresting finding—and that’s exactly the point. Holochain is capable of running everyday apps—not just DeFi apps with low throughput. And those apps are hosted on the Holo P2P network, which also makes them accessible to anyone with a web browser. That means Web 3.0 is possible throughout our digital lives, not just in a few highly specialized contexts. That changes the whole game. 

This was a test, so of course not everything was perfect. As mentioned, some users did start to see slowdowns—logins would sometimes take a couple minutes, and channels generally started loading more and more slowly. We identified a number of bugs that were causing these and other issues.

But again, that’s exactly what we wanted. It would’ve been hard to discover these issues with automated tests—the real world has a way of creating scenarios you could never dream of. We managed to collect a lot of insights, some of which have already resulted in bug fixes. 

In the context of an end-to-end test of a Holochain app fully running on the Holo infrastructure, here are a few of the most interesting details about what we learned.

- Holochain networking—which includes both gossip and signals—is highly scalable with no noticeable networking issues even with this number of source chains (5000+), concurrent users (1000+), or chain entries (~150,000).

- Holochain is resilient enough for alpha use. It demonstrated graceful recoveries when it encountered errors, which means there were not lasting negative impacts for users. 

- The average number of users per HoloPort was approximately 20, with some HoloPorts supporting as many as 50+, and we expect that this number will scale significantly as we continue to deploy more infrastructure improvements.

- It is important to note that we started this test with the service logger installed, active, and functioning well. However, we knew that it is not yet up to the task of supporting on the scale of hundreds or thousands of users we wanted to participate in this initial open test, for the sake of testing many other components. It produces a lot of data, which would be unsustainable without sharding and other planned enhancements.

- Beyond the lack of good user experience features that people expect from a chat app, there were also known architectural shortcomings of the Elemental Chat application. For instance, every user received every message in every channel, even if they weren’t currently participating in it. This produced millions of signals, which was actually quite useful in this test because it gave us a great baseline for estimating the performance of a much busier chat app. What’s more, the channel and message updating in the app were not optimised in the ways that production apps would be written, adding to potential slowness once many messages were in the system. What this allowed us to do, however, was to find underlying Holochain gossip inefficiencies and optimize them (which we’re already doing). The bottom line is that if we had designed Elemental Chat perfectly, we might have missed important issues that are better found now. 

Testing Elemental Chat in Public Alpha was an important milestone on our roadmap—and there are several more that we’re quite close to completing over the next few months, or sooner. These include:

- HoloFuel Transactions Pre-release

- HoloPort Registration Usability

- Publish hApps Pre-release

- Holochain Sharding Resilience and Scaling Framework

The recent public test of Elemental Chat was successful beyond our expectations and we learned so much. Neither the successes nor the learnings would have been possible without the thousands of people who participated so we want to finish by sharing appreciation and gratitude. An additional special thank you for the consistent presence of our many Turkish friends, who helped us reach the high numbers of peer participants and messages. Uzun vadeli desteğiniz ve testiniz için büyük bir teşekkür!

-Mary