Lack of HTTP request rate limiting has been a staple low severity finding in penetration test reports for as long as I can remember. OWASP called it API4:2019 Lack of Resources & Rate Limiting in the 2019 API Top 10 and refined it to API4:2023 Unrestricted Resource Consumption in 2023. Is this still a good idea, or are we more likely to shoot ourselves in the foot with it?

TL;DR:

• Global HTTP request rate limiting comes with the risk of legitimate users being blocked because of some change you don’t necessarily control, like being featured by a big link aggregation site, or maybe just a particularly effective marketing campaign.
• Rate limiting specific functionality to protect against known attacks can be a better option. This works when you know your threat model.
• Monitoring and alerting on unusual request rates is super valuable.
• Security should enable legitimate users to do what they need to do in a safe way. Blocking requests because too many users are on a single IP address doesn’t enable users or security effectively.

Classic request rate limiting (which we’re talking about here) is counting the number of HTTP requests made by an IP address, and if it reaches some limit block further requests for a little while. This sounds great as long as the limit is above what a normal set of users would do from their shared IP address. If an attacker comes along and starts guessing usernames before a password spraying attack, or exploits some broken access controls to pull large amounts of data down by incrementing an entity ID somewhere, then strict rate limiting will frustrate this attack right?

Well kind of! It will slow the attack down but not prevent it, and if the attacker is using a botnet and cycling through 2000 IP addresses (pretty common these days), then you’re out of luck entirely. Editors note: Other less sketchy options, such as using an AWS API Gateway for egress traffic or using something like TOR to cycle your source IP also exist.

During a pentest, we have limited time to get things done and if we’re slowed down by rate limiting it’s a big deal. Real attackers on the other hand, will often run slow attacks regardless of rate limiting to decrease the chances they’re discovered before they finish and are often happy if an attack takes a few days or weeks if they don’t have to baby sit it. This means rate limiting can end up frustrating the pentesters legitimately trying to secure your platform, while having a negligible impact on the actual bad-guys.

We’ve advised on real world versions of these attacks over the last couple years and rate limiting can be a useful short term band aid, but isn’t a real fix to the problem. How do you really prevent password spraying and data theft through missing access controls? You enforce some form of multi-factor authentication (even a “verify your login from this new device” link in an email makes a huge difference), and carefully design, implement, and audit your access controls. I’d call those the minimum, but properly mitigating those attacks is a whole set of other articles and doing it well depends on the specifics of your website or application.

Let’s talk about the “footgun” aspect of request rate limiting. Let’s say you set a limiting for 50 req/sec from a single IP address, and then your site becomes popular with a large workplace or university, or with users on a ISP that uses CGNAT where lots of users share an IP address (looking at you, mobile ISPs). Maybe you’ve just launched a shiny new marketing campaign. Suddenly some users will start getting blocked, and if you’re not monitoring/alerting on this, the first you’ll know is if someone starts complaining.

So, where does request rate limiting really come into play? To mitigate specific attacks, and when in monitor-only mode, as a possible indicator of attack. If you normally get 2000 request per sec, then suddenly are getting 20,000 something is up! Did some news aggregation site just link to you (yay), or is your customer data flying out the door because of broken access controls (not yay)? You probably have functionality that you expect to be not be used super often and will be a target for attack. Think password reset, user registration, newsletter signup, contact forms etc. These are certainly things you want to rate limit (per IP or per user account), or use some kind of anti-automation on (bot detection, CAPTCHA, etc), depending on what your threat model looks like. This will also depend on and what will be usable for your technology stack, functionality, and users. If you do implement rate limiting, you should have alerting that triggers before this happens and provide graceful fall-back or nice error messages for users that hit the rate limits.

Good security should enable legitimate users to do what they need to do in a safe way. Blocking requests because too many users are on a single IP address doesn’t enable users or security effectively. If we’re smart about it, we can avoid blocking legitimate users and enable growth while mitigating real-world attacks where rate limiting may or may not feature as a security control.