Mastodon Stampede

Slashdot’ing is back!

Cityscape at night with overlay emoji and number bubbles
Robert Pickard

As people shift away from centralized internet platforms, old problems creep back. Decentralization has exacerbated service availability issues when something goes viral. Sites are being knocked offline by a stampede of requests generated by popularity. All of NETSCOUT’s Arbor distributed denial-of-service (DDoS) protection solutions can identify and regulate these traffic spikes to prevent unwanted network congestion, outages, or collateral damage.

The Problems with Fediverse

The fediverse is shorthand for a collection of protocols and data structures for creating decentralized social media relationships. In contrast with walled garden platforms such as Twitter and Facebook, it uses open standards to mediate user-to-user, user-to-server, and server-to-server communications. As long as you are using a common language, you can operate a system that is interoperable with other people’s systems. This allows users to maintain independence while continuing to create communities via dynamic linkages.

The increasing popularity of distributed social network systems based on the fediverse has created a phenomenon of “fan-in” traffic that is behaviorally similar to reflection/amplification DDoS-style attacks. As more users join the fediverse via software such as Mastodon, websites have been strained with increased load, some to the point of breaking. While not necessarily malicious, the outcome is the same as a DDoS attack. 

The popular adoption of social media via common platforms such as Facebook and Twitter had the side effect of creating a caching layer on the consumer side. The users of a platform were not all getting distinct copies of data external to the platform from individual requests to the external source, but instead were receiving a cached copy the platform itself provided—specifically, for this use case, the small “thumbnails” of external links that are included in posts. The platform (Twitter, for example) would make the request to the link, create a thumbnail, and then serve the cached thumbnail to other Twitter users.

In a federated social networking system such as Mastodon, there isn’t a shared cache for the users to draw from: Each user ends up making their own requests to external sites for generating thumbnails to display in posts.

The problem arises when a federated user with a high number of followers shares a post, or boosts (Mastodon’s term for sharing another user’s post) on their timeline. The core protocol for coordinating and sharing information among Mastodon instances is ActivityPub. When a user shares a post, it is published via ActivityPub to the Mastodon instances of all of that user’s followers. When an instance gets a new post that contains a link, it makes a request to that link to generate a preview. The process happens automatically.

This creates the fan-in problem. A user with 100,000 followers posting a link to coolthing.example.com has effectively acted as a command-and-control for 100,000 processes to hit coolthing.example.com.

Graph of requests per minute of when a popular Mastodon user boosts a post

Opening the Floodgates

And it doesn’t stop there because some percentage of those 100,000 followers will themselves boost the boost by sharing with their users, which creates another flurry of requests.

Boosting a post is not the only activity that causes a flood of requests. Links in a user bio, editing a post, and pretty much anything that has a URL in it will activate an entire cadre of followers to flood a site. Some software will also continue to try to make a request if the current one fails, creating a long and persistent tail of requests. 

This has already brought down some sites and greatly increased the outgoing data rates of others. These are not cases of malicious intent. The sites are being punished for being popular, very similar to when Slashdot would post an article and the rush would take the site down.

This problem with ActivityPub has been known for a while. However, it has been exacerbated by the huge increase in interest in decentralized platforms as centralized ones have started to wobble.

GitHub Mastodon issue showing DDOS known issue from 2027

Fortunately, the most-used fediverse software is a good net citizen and identifies itself unambiguously through its user agent string value. It is easy to identify programmatically.

Mastodon user-agent strings

How NETSCOUT Can Help

NETSCOUT DDoS Protection solutions such as Arbor Threat Mitigation System (TMS), Arbor Edge Defense (AED), and Arbor Cloud can easily filter requests based on a regular expression (regex) to shield a site that’s under the strain of sudden popularity.

Uniform Request Identifier (URI) request limiting also can be used to ease the load on the target server. In addition to curbing automated requests, URI request limiting addresses users following the link in their browser from the thumbnail, slowing down but not denying access to the URI.

Because the automated requests are well formed, countermeasures used to filter out explicitly bad actors via techniques such as Slowloris are not applicable. From a network perspective, the requests are well-behaved; there are just a lot of them.

Currently, the industry is on the rising end of a hockey stick curve of mass adoption of distributed social media. At some point, the curve will level out. But right now, not only is there an increase in the number of users, but there also is an exponential increase in the messages between those increasing numbers of users. Everyone is excited. There is a lot more sharing and not a lot of understanding of the effect that this ebullient chatter has on the object of their excitement. The stampedes are likely to increase, no matter how well-meaning they are.

The fediverse is an exciting and hopefully revitalizing shift in creative interactions via the internet. The Mastodon stampede is a threat to fediverse viability because of the negative effects on the web via organic DDoS, even if it is not malicious. Sites can be love-bombed offline and users can feel discouraged from posting out of a sense of not wanting to intentionally harm the thing they were trying to share. There are discussions about changing the ActivityPub protocol to make it less of a hazard to the wider web, but those are long-term changes. Meanwhile, server-side protections such as NETSCOUT Arbor allow for the growth of a more decentralized web while keeping your site online and available.

NETSCOUT is a leader in protecting the internet from malicious and unintended DDoS attacks—including those in the fediverse. Check out NETSCOUT’s Arbor DDoS mitigation solutions for more details.