Nodejs and Express API rate limiting

What is rate limiting and why would you need it?

Let’s say you create a free API that returns memes, and let’s assume there are already some APIs that do this, but these are paid APIs. Naturally, users will start shifting to your API, which will not make your competitors happy. In this case, there is one thing your competitors can do. They can set up a few hundred bots that will keep on making requests much faster than any human could, which will slow down your services because now your server is serving the bots, which are trying to mimic the consumer’s behavior.

So, what can you do?

You can tell your server, “Hey, once you have responded to a client then don’t respond to them again before one second.” This will slow down the bots because now there is a one-second cool-off period. This is rate limiting.

Now that you are convinced that rate limiting is important for your services let’s see how to set it up.

Let’s be very clear that you can use third-party packages to implement rate limiting. But that will not teach you how it works, so let’s get our hands dirty.

Setting up project:

1. Create an empty folder.

2. Run npm init -y

3. Run npm i express ioredis to install express and Redis.

4. Create index.js to write our server-side code and a file called redis.js.

5. Run nodemon index.js so that the server keeps running.

Getting started:

Paste it in your index.js file

1
2
3
4
5
6
7
8
9
10
const express = require('express');
const app = express();

app.get('/route', (req, res) => {
  res.json('You have successfully hit route');
})

app.listen(3000, () => {
  console.log('Server started');
})

Here, we have created a get request and initialized a server. This is basic express boilerplate code.

In rate limiting, we check whether the same user makes requests frequently or not. And to do that we will need to check the user’s IP address, which we will use to identify the user.

Modify your API for route like this.

1
2
3
4
5
app.get('/route', (req, res) => {
  const ip = req.headers['x-forwarded-for'] || req.socket.remoteAddress;
  console.log(ip);
  res.json('You have successfully hit route');
})

Let’s break down the code.
The req.headers['x-forwarded-for'] will fetch the IP if the user is sitting in front of a remote proxy like NGINX, and the req.socket.remoteAddress will return the IP if the user is not using proxy and is directly sitting across the internet. Let’s run it.
Following is my Postman response, you can also use a browser to make requests.

"You have successfully hit route"

The following is what I saw in my console as IP.

It does not look like a typical IP address because it’s on localhost. In a real working application, you will see something like 192.158.1.38.

Paste the following in your redis.js file.

1
2
3
4
const Redis = require('ioredis');
const redis = new Redis();

module.exports = redis;

It’s just a Redis instance that we will use. Also, make sure that the Redis server is running on your machine. If you do not have the Redis server installed, go to the official website to download it for your machine.

In your index.js import the redis.js file and modify the API like this.

1
2
3
4
5
6
7
8
const redis = require('./redis');

app.get('/route', async (req, res) => {
  const ip = req.headers['x-forwarded-for'] || req.socket.remoteAddress;
  const requests = await redis.incr(ip);
  console.log(`Number of requests made so far {requests}`);
  res.json('You have successfully hit route');
})

Now, make a request to the route twice. This is what the console will display.

We can see how many requests a particular user made.
Now, let’s put up a limit. Modify the API like this.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
app.get('/route', async (req, res) => {
  const ip = req.headers['x-forwarded-for'] || req.socket.remoteAddress;
  const requests = await redis.incr(ip);
  console.log(`Number of requests made so far {requests}`);
  if (requests > 5) {
    res.status(503)
      .json({
        response: 'Error',
        callsMade: requests,
        msg: 'Too many calls made'
      });
  } else
    res.json('You have successfully hit route');
})

If the calls made are more than five, we will return an error message.

1
2
3
4
5
{
  "response": "Error",
  "callsMade": 6,
  "msg": "Too many calls made"
}

Our console will also display the message.

Now, our app is remembering the number of requests by a user and also showing the error message for more than five requests.

All we need to do is renew this every second so that after a second the count of the number of requests gets set back to zero. To do that modify the API as follows.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
app.get('/route', async (req, res) => {
  const ip = req.headers['x-forwarded-for'] || req.socket.remoteAddress;
  const requests = await redis.incr(ip);
  console.log(`Number of requests made so far {requests}`);
  if (requests === 1) {
    await redis.expire(ip, 60);
  }
  if (requests > 5) {
    res.status(503)
      .json({
        response: 'Error',
        callsMade: requests,
        msg: 'Too many calls made'
      });
  } else
    res.json('You have successfully hit route');
})

Here we have added the following code segment:

1
2
3
if (requests === 1) {
  await redis.expire(ip, 60);
}

This means that when it is the first request we will set the expiration time to sixty seconds for the key ip. Before moving forward we need to clear our redis storage. To do that run FLUSHALL in the redis-cli, which should also be installed on your system.

Now try to hit the API again. It should give a response for the first five requests but an error after that. After sixty seconds it will again set back.
As you can see the value of the key ip was reset automatically.

Now you know how rate limiting works, and you can implement it in your routes.

You will find the source code at this GitHub link.