BLOG
ARTICLE

Pagination in Next.Js with Markdown Files

4 min read

Recently I made pagination for this site's blog page because I feel now I have plenty of articles to paginate. With my photography site, pagination is as simple as calling the pagination API of WordPress. But with this site, I don't have pagination API since the articles are just a bunch of Markdown files stored locally.

So here's what I did.

The Start

There are 2 basic things to do to easily paginate in React.

  1. Create the component for pagination.
  2. Create the API or handler for the component in (1).

For no 1 the answer is pretty easy. When you type 'React pagination' in Google, chances are the first result to come out is this library: react-paginate. Many people have used this simple component and so I did the same. Just a few lines of CSS and the pagination matched the site's color scheme.

For no 2, I researched a bit. Since I already have a function that reads the .md files as you can see when you go to the blog index page, I just needed to extend that function and make it as Next.Js API route.

1
http://localhost:3000/api/pages

After a few modifications, the API now has queries that handle the current page and items per page.

1
http://localhost:3000/api/pages?page=1&perPage=10

The API would return values like the below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
{
    success: true,
    totalPosts: 12,
    pageCount: 6,
    pages: [
        {
            title: "My Shell Script Best Practices",
            date: "2021-07-15T00:00:00.000Z",
            slug: "20210715-my-shell-script-best-practices",
            excerpt: "A compilation of best practices when working with shell scripts.",
            readTime: "5",
            tags: "Shell Script,Bash,Linux"
        },
        {
            title: "Simple CI/CD with Jenkins",
            date: "2021-06-23T00:00:00.000Z",
            slug: "20210623-simple-cicd-with-jenkins",
            excerpt: "Here's how I implemented a simple continuous deployment on some of my projects at work with Jenkins.",
            readTime: "7",
            tags: "Jenkins,Docker,CI CD,AWS,Automation"
        }
    ],
    currentPage: 1,
    perPage: 2
}

Looking good! 🥰 I made blog page calling this API whenever a page is selected. It worked in localhost with no problem so I tried to deploy to Vercel to test one last time before going to production.

But it didn't go well... the page didn't change and showed the correct articles.

Instead, the API gives me this:

1
Internal Server Error

The Problem

What gives? What went wrong here? 🤔

Looking at the server log, I found the following error. Seemed to me the API route could not find the Markdown files for articles.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
2021-08-15T14:00:01.869Z	1060b9b9-ca50-4271-8b4b-83943aab5073	ERROR	Error: ENOENT: no such file or directory, scandir '/var/task/public/posts'
    at Object.readdirSync (fs.js:1043:3)
    at getSlugs (/var/task/.next/server/chunks/883.js:31:52)
    at getAllPostsSortDescending (/var/task/.next/server/chunks/883.js:62:17)
    at paginatePosts (/var/task/.next/server/chunks/883.js:117:23)
    at handler (/var/task/.next/server/pages/api/pages.js:24:87)
    at apiResolver (/var/task/node_modules/next/dist/next-server/server/api-utils.js:8:7)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at async Server.handleApiRequest (/var/task/node_modules/next/dist/next-server/server/next-server.js:66:462)
    at async Object.fn (/var/task/node_modules/next/dist/next-server/server/next-server.js:58:580)
    at async Router.execute (/var/task/node_modules/next/dist/next-server/server/router.js:25:67) {
  errno: -2,
  syscall: 'scandir',
  code: 'ENOENT',
  path: '/var/task/public/posts'
}

After digging around I found a blog article that addresses the very problem. This has something to do with the way Next.Js works. During the build, Next.Js will remove any file that is not inside pages or public folders. Since I have my article posts inside posts folder, the Markdown files are not included in the build that is uploaded to Vercel server.

In the above blog article, the author made a very good point of storing ALL of the contents of Markdown files into a JSON file and store under public folder. So I did just that.

In hindsight, I encountered the same problem a few months ago when I added a sitemap to the site. I even created an article for that! So uh yeah, I SHOULD HAVE READ my articles more often. 😇 Nevertheless I'm still happy to find an alternative solution.

The Solution

1. Create script for generating JSON file

I created a script, lib/cachePosts.js, which reads all Markdown files and store under public/cache folder. By running the script node lib\cachePosts.js file before the build, we ensure that the contents of articles are readily available for API.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
const fs = require("fs");
const path = require("path");
const matter = require("gray-matter");

const getAll = dir => {
    // Read files at {dir}
    const directory = path.join(process.cwd(), `${dir}`);
    const fileNames = fs.readdirSync(directory);
    // Get the content of the files as JSON
    const content = fileNames.map(fileName => {
        const slug = fileName.replace(/\.md$/, "");
        const fullPath = path.join(directory, fileName);
        const fileContents = fs.readFileSync(fullPath, "utf8");
        const { data } = matter(fileContents);
        // Return only slug and data (content not needed)
        return {
            slug,
            data
        };
    });
    // Return a big array of JSON
    return JSON.stringify(content);
};

const allPosts = getAll("posts");

const postFileContents = `${allPosts}`;

// Create the cache folder under public if it doesn't exist
try {
    fs.readdirSync("public/cache");
} catch (e) {
    fs.mkdirSync("public/cache");
}

// Create JSON file for posts
fs.writeFile("public/cache/posts.json", postFileContents, err => {
    if (err) return console.log(err);
    console.log("Posts successfully cached.");
});

2. Modified API route to read the JSON file

Now that we have the JSON file, we need to make API route to fetch the said file. One caveat here, API route only responds to absolute URL or else you will encounter the below error. This is a characteristic of fetch. There's a discussion about it here.

1
Error: only absolute urls are supported

I added these lines to find the host for both localhost and production, then the API route worked fine.

1
2
3
4
5
6
// Setup protocol & hostname
const host = req.headers.host;
const protocol = host.includes('localhost') ? 'http://' : 'https://';
// Fetch the JSON file containing all posts
const response = await fetch(`${protocol}${host}/cache/posts.json`);
const posts = await response.json();

3. Add the script to package.json

Having to run the script in no 1 every time we want to build is rather cumbersome. So I added the command to package.json so that Vercel would execute during build automatically.

1
2
"scripts": {
    "build": "node lib/cachePosts.js && next build",

The Result

OK, so now whenever I push to Vercel, I can invoke the API route successfully and it returns the needed pagination values.

https://jerfareza.me/api/pages?page=1&postsPerPage=5

Go ahead and try the API link above in your browser! You will see JSON values for pagination.

In Retrospect...

This solution is not perfect, I know. There's a delay between clicking the page and then scroll to the top as the selected page changes. But in my opinion it's pretty fast, enough for a personal website.

If I were to make improvements, probably I will implement paging by leveraging Next.Js dynamic route like pages/blog/[page] and make use of getStaticPath just like I did on my photography website.

Jerfareza Daviano

Hi, I'm Jerfareza
Daviano 👋🏼

Hi, I'm Jerfareza Daviano 👋🏼

I'm a Full Stack Developer from Indonesia currently based in Japan.

Passionate in software development, I write my thoughts and experiments into this personal blog.