When doing our final project of the Technigos web developer boot camp we are making an arcade game review app, where the users can browse different arcade machine games and rate them with stars and text.

Before we start
This will use the IGDB API for fetching the data but the concept can be used in many different ways, I'm sure. Please use this as a reference but you will need to modify it to work on your project. And also tis will mostly be taken out of its context but will give a general point of view on how to set it up.

Prerequisites

Basic understanding of Express
Knowledge of how to set up a MongoDB using Mongoose
Understanding of how APIs work under the hood

Setting up the project

First, we need to import our dependencies at the top of our Server.js file.
All the code written in this guide will be located in the server.js file, except the .env file storing the API keys.

Fetching the games

Time to take a look at how to populate the database; the good stuff of this guide.
Let's start with creating a function called fetchAndSaveGames, this is the function that will do most of the heavy lifting.
First off we need it to be async since we will fetch data over the internet, and then do a fetch to the API.

const fetchAndSaveGames = async () => {
  try {
    const response = await fetch('https://api.igdb.com/v4/games', {
      method: 'POST',
      headers: {
        'Authorization': process.env.IGDB_CLIENT_SECRET, //This is in .env
        'Client-ID': process.env.IGDB_CLIENT_ID, //This is in .env
        Accept: 'application/json'
      },
      body: `fields name, cover.url, first_release_date, platforms.name, 
            genres.name, summary, slug, involved_companies.company.name,
             screenshots.url; where platforms = 52;`
    });
    const games = await response.json().save();
    //console.log(games);

}

Lets break a few things down:

This is the fetch from the URL:
const response = await fetch('https://api.igdb.com/v4/games', {

In the header, we need to pass in Authorization and Client-ID, and also Accept so we will return the data in JSON format. You can get all this from the IGDB website, where you need to sign up and create a developers account. No worries, it's free.
headers: {
'Authorization': process.env.IGDB_CLIENT_SECRET, //This is in .env
'Client-ID': process.env.IGDB_CLIENT_ID, //This is in .env
Accept: 'application/json'
},

In the body, you specify what you would like to get back from the API, in this case a quite specific set of data. If you want to return all the data from the API from the Game endpoint, you could shorten it down to only "fields *".
body: `fields name, cover.url, first_release_date, platforms.name, genres.name, summary, slug, involved_companies.company.name, screenshots.url; where platforms = 52;`

In the end, we want to store the data somehow, in this case as a variable, and save it to the database:
const games = await response.json().save();

...One more thing

So now we have fetched some games, and we have also hit a wall: The API that we fetch from has a limit on how many requests one can do, as of the time writing this it is 4 requests per second. So how do we solve this?

We need first to add two arguments in the fetchAndSaveGames, offset and batchSize.
We will use them on the request body of the API call, as below:

// Populate database with games from IGDB API
const fetchAndSaveGames = async (offset, batchSize) => {
  try {
    const response = await fetch('https://api.igdb.com/v4/games', {
      method: 'POST',
      headers: {
        'Authorization': process.env.IGDB_CLIENT_SECRET,
        'Client-ID': process.env.IGDB_CLIENT_ID,
        Accept: 'application/json'
      },
      body: `fields name, cover.url, first_release_date, platforms.name, 
        genres.name, summary, slug, involved_companies.company.name,
        screenshots.url; where platforms = 52; 
        limit ${batchSize}; offset ${offset};`
    });
    const games = await response.json().save();
    //console.log(games);
}

To make this work we need another function, called fetchAllGames:

const fetchAllGames = async () => {
  const batchSize = 1; // Number of games to fetch in each batch
  const totalGames = 10000; // Total number of games to fetch

  const delay = 250; // Delay in milliseconds (4 requests per second)
  let offset = 0; // Initial offset

  try {
    let totalCount = 0;
    let fetchedCount = 0;

    while (totalCount < totalGames) {
      fetchedCount = await fetchAndSaveGames(offset, batchSize);
      totalCount += fetchedCount;
      offset += batchSize;

      // Delay between API calls
      if (totalCount < totalGames) {
        await new Promise((resolve) => setTimeout(resolve, delay));
      }
    }

    console.log(`Total games fetched and saved: ${totalCount}`);
  } catch (error) {
    console.error(error);
  }
};

The fetchAllGames is not too complicated. Let's break it down:
Since the API only can take 4 requests per second, the batchSize is set to 1, and the delay is set to 250 milliseconds.

the offset variable is used to keep track of the starting position or index of the batch of games to fetch from the larger collection of games.

totalGames is to only fetch 10 000 games from the database, so it won't fetch more than that in this example, one could of course change this to a smaller or bigger number depending on the use case.
The while loop condition totalCount < totalGames checks if the total count of fetched games (totalCount) is less than the total number of games to fetch (totalGames).

Inside the loop, the function fetchAndSaveGames is called to fetch and save games. It takes the current offset and batch size as parameters. The returned value, fetchedCount, represents the number of games fetched and saved in the current batch.

The fetchedCount is added to the totalCount, keeping track of the total count of fetched games.

The offset is increased by the batchSize to set the starting position for the next batch of games to fetch.

To control the rate of API calls, a delay is added using the setTimeout function and await new Promise. This delay ensures that there is a pause between consecutive API calls. The duration of the delay is determined by the delay value, which is given in milliseconds.

The loop continues until the totalCount reaches or exceeds the totalGames value, meaning that all the games have been fetched and saved.

Finally, the total count of fetched and saved games is logged to the console.

If any error occurs during the process, it is caught in the catch block, and the error is logged to the console.

There we go, now the database is populated with some awesome games!
Now it's your turn to join the conversation! I would love to hear your thoughts and experiences with JavaScript. Have you encountered any interesting challenges or built something amazing? Share your stories in the comments below and let's continue the discussion. Don't forget to spread the word by sharing this post on your favourite social media platforms.
Together, let's inspire and empower the JavaScript community!

Fetch and populate database from API

In this guide, I will show you how to fetch games from an open API to populate a database with a specific filter on it, using Express and MongoDB.

Table of contents

Prerequisites

Setting up the project

Fetching the games

...One more thing