What Happened​

• Our monitoring detected service degradation early Saturday, followed by Hetzner's notification that our server would be blocked due to "suspected port scan attacks."
• Rspamd's service responds to legitimate external requests on port 11335 as part of its normal operation—the traffic flagged was fully expected and non-malicious.
• Despite prior communications clarifying this issue (with Hetzner support involved on three separate occasions), our server was nevertheless blocked following a provider-side false positive.
• Although Hetzner's support eventually agreed that the block was in error and restored service, we received further claims that network activity from our side (including connections to resolvable, legitimate mail service IPs) was improper, citing "unrouted addresses" — a claim contradicted by publicly available DNS records and usage reports.

Why This Matters​

• This ongoing false positive detection and inflexible incident response directly impacted the reliability of a widely-used open source project, with real-world effects for large-scale email filtering, spam defense, and downstream email service providers.
• We have complied with all incident forms, engaged in dialogue multiple times, and transparently reported our findings. Unfortunately, misinterpretation of perfectly ordinary and legitimate network traffic resulted in a block that, while now lifted, has forced us to disable certain public-facing Rspamd services to avoid future risk.

Our Next Steps​

• We are reviewing our hosting provider relationship and will migrate to a more robust and responsive provider to avoid future interruptions.
• Public access to some Rspamd services will remain suspended until a solution that meets both our standards and availability requirements is fully in place.

✅ Positive Update​

Due to the generous support from many companies and individuals who are eager to support the project, we expect the public fuzzy service to be back online in approximately one week. We are grateful for the community's overwhelming response and commitment to keeping Rspamd services available.

Conclusion​

This episode highlights the critical importance of technical nuance and responsive incident handling in the infrastructure supporting public open source projects.

We regret the inconvenience to our user base and assure everyone that maintaining reliable, high-quality service for the global community remains our top priority.

Thank you for your understanding and support.

For questions or concerns, please reach out through our support channels.

Rspamd has always been oriented on the performance but it was always quite hard to measure how fast it was as normally it runs just fast enough.

However, I was recently offered to process Abusix Intelligence feeds using Rspamd. These feeds are used to improve Rspamd fuzzy storage quality, to feed URLs and Emails to the DNS black lists provided by Rspamd project and used in SURBL module.

Problem statement​

The amount of data that required to be processing is huge - it is about 100 millions of messages per day.

Here is an example to calculate connections count when processing these messages using Rspamd:

It means that over 10 seconds Rspamd has to process around 15 thousands of messages which gives us a rate of 1500 messages per second.

Rspamd setup​

The settings used to process this amount of messages are pretty similar to those that are provided by default.

There is also some significant amount of home-crafted scripts written in Lua to provide the following functionality:

• Provides deduplication to save time on processing of duplicates
• Performs conditional checks for url and emails blacklisting:
  • checks if an url is in whitelists (around 5 whitelists stored in Redis are used)
  • check if an url is already listed
  • check if it matches any suspicious patterns
• Checks if a message should be learned on fuzzy storage (various conditions)
• Stores messages in IMAP folders providing sorting, partitioning and sampling logic
• Doing various HTTP and Redis queries for servicing purposes

Hardware​

Now some words about hardware being used.

Previously we have set the same setup on a small instance of AX-60 and it was loaded for around 80%. We have decided to move to a more powerful server to have some margin for processing more emails and doing some experiments.

Hence, we now have an AX-160 AMD server rented in Hetzner. This is quite a powerful machine and the current load pictures look like this one:

Rspamd is also being fed via proxy worker that runs on another host and performs initial data collection and emitting messages via the Internet providing transport encryption using HTTPCrypt. However, its CPU usage is quite negligible - it uses only a single CPU core by around 40% in average.

Results analytics​

As you can see, this machine runs also Clickhouse, Redis, own recursive resolver (Unbound), and it still has ~80% idle processing these 1500 messages per second.

If we look at the performance counters by attaching to some of the worker processes, we would see the following picture:

The top consumers are Lua allocator and garbage collector. Since we are using [Rspamd experimental package]({{ site.baseurl }}/downloads.html) on Debian Buster, then it is built with bundled LuaJIT 2.1 beta3 and Jemalloc allocator, however, it seems that there is some issue with this allocator in Debian Buster, so I had to load it manually via the following command:

Followed by restarting of Rspamd.

It is interesting that this Rspamd setup accepts all connections encrypted using HTTPCrypt but chacha_blocks_avx2 takes less than 0.16% of CPU according to perf report.

This particular instance of Rspamd is slightly tuned to use more memory to save some CPU cycles:

# local.d/options.inc

lua_gc_step = 100;
lua_gc_pause = 400;
full_gc_iters = 10000;

These options tell Rspamd to preserve Lua objects in memory for longer time, at the same time in this mode, we can also observe GC stats on workers that performs full GC loop each 10k messages being scanned:

As you can see, full GC iter takes quite a significant time. However, it still keeps Lua memory usage sane. The ideas behind this GC mode have been taken from the generational GC idea in LuaJIT Wiki.

Resulting graphs​

Here are some UI captures taken from a previous machine:

As you can observe, there was some HAM portion increase over the recent days, however, it was caused by adding new sampling logic and duplicates filtering to save CPU resources (these messages are marked as ham and excepted from scan).

There is also a Clickhouse based dashboard that's created using Redash:

Since we have Clickhouse on board, we can do various analytics. Here is an average scan time for messages:

... and average size of messages:

Conclusions​

So with this load rate (1500 messages per second) and with the average size of messages around 2Kb, Rspamd processes each message in around 100ms in average. I hope these numbers could give one some impression about Rspamd performance in general.

I would like to give the main kudos to Abusix who are constantly supporting Rspamd project and have generously provided their amazing spam feeds to improve Rspamd quality!