Deep Dive into Django's Pagination

Pagination is a technique for splitting large chunks of data across multiple web pages. It makes it for better user experience; not loading all the information available upfront, and also helps you saving some resources since you only query the database for what you truly need at the moment.

When paginating with Django Paginators, the code is not just splitting the content on the UI, but it is truly querying the database in smaller chunks.

Setting Up The Base Project

To illustrate the process of paginating data, I'll go through an example project that you can follow by creating a new Django project, or using the example I created for this post: - django-pagination.

Starting with a Book model:

# pages/core/models.py


class Book(models.Model):
    title = models.CharField(max_length=255)
    author = models.CharField(max_length=255)
    published_at = models.DateField(blank=True)
    
    def __str__(self):
        return self.title
        

A function based view to list all the book instances:

# pages/core/views.py


def book_list(request):
    books = Book.objects.all()
    return render(request, "core/book_list.html", {"books": books})

And a book_list.html template to display all the book entries. I'll be using bootstrap to style the page. Installation instructions available here: Installing bootstrap.

book_list.html extends from the base.html template, that holds all bootstrap's CSS and JavaScript files. The template structure used can be found at augustogoulart/django-pagination.

Here's the snippet that really matters. It will create an empty table on the UI, readily available to render data:

<!-- pages/core/templates/core/book_list.html -->

{% extends 'base.html' %}

{% block content %}
<div class="row">
  <div class="col-8 offset-2">
    <table class="table">
    <thead>
    <tr>
      <th scope="col">#</th>
      <th scope="col">Author</th>
      <th scope="col">Date Published</th>
      <th scope="col">Title</th>
    </tr>
    </thead>
    <tbody>
    {% for book in books %}
      <tr>
        <th scope="row">{{ book.id }}</th>
        <td>{{ book.author }}</td>
        <td>{{ book.published_at }}</td>
        <td>{{ book.title }}</td>
      </tr>
    {% endfor %}
    </tbody>
  </table>
  </div>
</div>
{% endblock content %}

Populating The Database

Now we need to populate the database, so there is data to paginate.

Leveraging the django-extensions library, create a python package called scripts in the root directory of the project, and a python file alongside the __init__.py. I called mine fake_date.py

The fake_data.py script will depend on django-extensions and Faker, so those must be installed:

$ pip install Faker django-extensions

On fake_data.py, here's the code to generate fake book entries:

# scripts/fake_data.py

from faker import Faker

from pages.core.models import Book


def run():
    fake = Faker()
    Book.objects.bulk_create([Book(
        author=fake.name(),
        title=fake.sentence(),
        published_at=fake.date()
    ) for book in range(1000)])

Running fake_data.py will populate the project's database with a thousand records. It's necessary to run that script within the context of the Django project, so the models can load properly:

$ python manage.py runscript fake_data

Running the Django project at this point should output a page listing all the thousand records from the database:

Paginating Django

Displaying all that data on a single page is confusing and makes it unnecessarily lengthy. Let's split that huge list into multiple subsets, called Pages:

The Paginator

Adding a paginator to the book_list view:

# pages/core/views.py

from django.shortcuts import render
from django.core.paginator import Paginator

from pages.core.models import Book


def book_list(request):
    books = Book.objects.all()
    paginator = Paginator(books, 10) # new
    return render(request, "core/book_list.html", {"books": books})

Right below books, I'm instantiating a Paginator, and telling it to take the list of books and slice it in chunks of ten books. Each chunk will be a Page.

The Page

The Paginator alone will only orchestrate the pages, but the pages themselves carry the paginated book data.

Retrieving and outputting the first page only:

# pages/core/views.py

from django.shortcuts import render
from django.core.paginator import Paginator

from pages.core.models import Book


def book_list(request):
    books = Book.objects.all()
    paginator = Paginator(books, 10)
    page_obj = paginator.get_page(1)  # new 
    return render(request, "core/book_list.html", {"page_obj": page_obj})

I'll also make some changes to the template and rename the book list from books to page_obj, which is more semantically accurate, since we now have a page in the context. That will also help later on in the article when refactoring the view to a Class-Based view.

<!-- core/book_list.html -->

{% for book in page_obj %}
  <tr>
    <th scope="row">{{ book.id }}</th>
    <td>{{ book.author }}</td>
    <td>{{ book.published_at }}</td>
    <td>{{ book.title }}</td>
  </tr>
{% endfor %}

The page will now display only ten items:

Navigating Pages

Pagination is only useful if the pages can be navigated. Can you imagine a book where only the first page is available?

Page Number From Query String

Let's have the view to look for a page number parameter in the URL's query string:

def book_list(request):
    books = Book.objects.all()
    paginator = Paginator(books, 10)

    page_number = request.GET.get('page')  # new
    page_obj = paginator.get_page(page_number)  # changed
    
    return render(request, "core/book_list.html", {"page_obj": page_obj})

Now the pages can be surfed by passing the page number as an argument to the URL:

http://localhost:8000/?page=2

The second page will display items from the 11th to the 20th:

Switching Pages On The UI

For the pagination to be really useful, people accessing one page should be able to switch pages without manually editing the URL.

The methods used to navigate pages are from the page instance itself, which means that each page knows about the peers around it.

Starting with an HTML boilerplate for the pagination. I'll use bootstrap CSS to help with the styling:

<div class="col-4 offset-4">
  <nav aria-label="...">
    <ul class="pagination">

      <li class="page-item disabled">
          <a class="page-link">Previous</a>
      </li>

      <li class="page-item">
        <a class="page-link" href="#">1</a>
      </li>

      <li class="page-item disabled">
        <a class="page-link" href="#">Next</a>
      </li>

    </ul>
  </nav>
</div>

That page number indicator is hardcoded, however, the current page number can be accessed with {{ page_obj.number }}:

<div class="col-4 offset-4">
  <nav aria-label="...">
    <ul class="pagination">

      <li class="page-item disabled">
          <a class="page-link">Previous</a>
      </li>

      <li class="page-item">
        <a class="page-link" href="#">{{ page_obj.number }}</a>  # new
      </li>

      <li class="page-item disabled">
        <a class="page-link" href="#">Next</a>
      </li>

    </ul>
  </nav>
</div>

Accessing /?page=5 will now show data relative to page number five:

The Previous link can be enabled using the has_previous and previous_page_number methods from the page instance. To handle the first-page scenario, where has_previous is false, and calling previous_page_number would raise an EmptyPage error, some template logic will be required:

<div class="col-4 offset-4">
  <nav aria-label="...">
    <ul class="pagination">
      
      <!-- new -->
      {% if page_obj.has_previous%}
        <li class="page-item">
          <a href="?page={{ page_obj.previous_page_number }}"><span class="page-link">Previous</span></a>
        </li>
      {% else %}
        <li class="page-item disabled">
          <span class="page-link">Previous</span>
        </li>
      {% endif %}
      <!-- /new -->

      <li class="page-item">
        <a class="page-link" href="#">{{ page_obj.number }}</a>
      </li>

      <li class="page-item disabled">
        <a class="page-link" href="#">Next</a>
      </li>

    </ul>
  </nav>
</div>

Similarly, has_next and next_page_number will handle the next page button, and a similar template logic will handle the last page case:

<div class="col-4 offset-4">
  <nav aria-label="...">
    <ul class="pagination">

      {% if page_obj.has_previous%}
        <li class="page-item">
          <a href="?page={{ page_obj.previous_page_number }}" class="page-link">Previous</a>
        </li>
      {% else %}
        <li class="page-item disabled">
          <span class="page-link">Previous</span>
        </li>
      {% endif %}

      <li class="page-item">
        <a class="page-link" href="#">{{ page_obj.number }}</a>
      </li>

    <!-- new -->
    {% if page_obj.has_next %}
      <li class="page-item ">
        <a class="page-link" href="?page={{page_obj.next_page_number}}">Next</a>
      </li>
    {% else %}
      <li class="page-item disabled">
        <span class="page-link">Next</span>
      </li>
    {% endif %}
    <!--/new -->

    </ul>
  </nav>
</div>

Page /?page=99 will have both a Previous and a Next page:

And /?page=100 won't have a Next page:

Displaying More Pages

Outputting only the current page doesn't make it for good user experience. Some visual flair can help to provide a sense of location and navigability. To do that, it's possible to extend the Previous/Next sections of the template and add an active CSS class to the current page.

Below is the full code for the page switcher. You can find all the templates at augustogoulart/django-pagination

<div class="col-4 offset-4">
  <nav aria-label="...">
    <ul class="pagination">

      {% if page_obj.has_previous %}
        <li class="page-item">
          <a href="?page={{ page_obj.previous_page_number }}" class="page-link">Previous</a>
        </li>
        <li>
          <a class="page-link" href="?page={{ page_obj.previous_page_number }}">
            {{ page_obj.previous_page_number }}
          </a>
        </li>
      {% else %}
        <li class="page-item disabled">
          <span class="page-link">Previous</span>
        </li>
      {% endif %}

      <li class="page-item active">
        <span class="page-link">{{ page_obj.number }}</span>
      </li>

      {% if page_obj.has_next %}
        <li class="page-item">
          <a class="page-link" href="?page={{ page_obj.next_page_number }}">
            {{ page_obj.next_page_number }}
          </a>
        </li>
        <li class="page-item ">
          <a class="page-link" href="?page={{ page_obj.next_page_number }}">Next</a>
        </li>
      {% else %}
        <li class="page-item disabled">
          <span class="page-link">Next</span>
        </li>
      {% endif %}

    </ul>
  </nav>
</div>

Moving to Class-based Views

With our paginated page in place, one might prefer going with Class-based views instead of a function-based view. That can be achieved by simply substituting the book_list view function by the code below:

# pages/core/views.py

from django.views.generic import ListView

from pages.core.models import Book


class BookListView(ListView):
    paginate_by = 10
    model = Book


book_list = BookListView.as_view()

From CBV to FBV And Back Again

It is not unrealistic to think that a project may require paginating an existing function-based view (FBV) that is already too large. And although CBVs offers a brilliant simplicity, the project may not be ready to move from FBV to CBV yet, or the team simply may not have the intention to tackle that refactoring now.

One alternative could be delegating the pagination to a model manager. I won't detail model managers because that is out of scope for this article, but you can read more about it here: https://docs.djangoproject.com/en/3.1/topics/db/managers/

To paginate with model managers, start with a managers.py file on the app that will receive the pagination. The code for the new manager could be like this:

from django.core.paginator import Paginator
from django.db import models


class PageQuerySet(models.QuerySet):
    def per_page(self, request, per_page):
        paginator = Paginator(self, per_page)
        page_number = request.GET.get("page")
        return paginator.get_page(page_number)


PageManager = models.Manager.from_queryset(PageQuerySet)

Then, apply the PageManager to the target model:

class Book(models.Model):
    title = models.CharField(max_length=255)
    author = models.CharField(max_length=255)
    published_at = models.DateField()

    objects = PageManager()  # New

    def __str__(self):
        return self.title

And on the FBV, apply the per_page at the end of you query:

def book_list(request):

    page_obj = Book.objects.all().per_page(request, 10)
    
    return render(request, "core/book_list.html", {"page_obj": page_obj})

The result is a nice and clean FBV, with the drawback of passing the request to a model manager.

Behind The Scenes

Knowing how to set-up pagination in Django is great, but understanding what is happening behind the scenes is even better, so bear with me for a couple of paragraphs more.

Installing The Explorer's Toolkit

To explore the mechanics of Django's pagination, some tools will help:

• IPython
• sqlformatter*

Installing IPython with pip:

$ pip install ipython

*By the time of this writing, sqlformatter has a mismatch dependency with Django 3+. If you try installing it from PyPI, you may get an error like this:

django 3.1.1 requires sqlparse>=0.2.2, but you'll have sqlparse 0.1.11 which is incompatible.

I created a PR to fix that, but in the meantime, you can use my fork that updates the conflicting dependency:

$ pip install git+https://github.com/augustogoulart/sqlformatter

If your project's database is empty, use the fake_data.py again to generate some data:

$ python manage.py runscript fake_data

Since IPython is installed, starting the shell_plus from django-extensions will load all the project's models on an IPython session:

$ python manage.py shell_plus

Inspecting the data created:

In [1]: books = Book.objects.all()

In [2]: books
Out[2]: <QuerySet [<Book: Which reason data of.>, [...], <Book: Foot admit attorney political suggest building report.>]>

In [3]: books.count()
Out[3]: 1000

Enabling sqlformatter and inspecting the data again, but this time, exploring the SQL run by the ORM. sqlformatter will log to the terminal every time there's a database hit:

In [4]: from sqlformatter import logdb

In [5]: logdb()
Out[5]: True

In [6]: books.count()
SELECT COUNT(*) AS "__count"
FROM "core_book"

Paginating And Exploring The SQL

Paginating the books list and retrieving the first page. At this point, the ORM only queried the database for the COUNT(*) of objects, so the Paginator can set the first and last pages:

In [7]: from django.core.paginator import Paginator

In [8]: paginator = Paginator(books, 20)

In [9]: first_page = paginator.get_page(1)
SELECT COUNT(*) AS "__count"
FROM "core_book"

In [10]: first_page
Out[10]: <Page 1 of 50>

Retrieving the books on the first page:

In [11]: first_page.object_list.all()
Out[11]: SELECT "core_book"."id",
       "core_book"."title",
       "core_book"."author",
       "core_book"."published_at"
FROM "core_book"
LIMIT 20

Only when the application asks for the objects on the first page, the ORM retrieves that data.

That, however, is purely the lazy nature of QuerySets in Django, and the only participation of the Paginator is to tell the ORM how it should limit the results, which is accomplished by the last SQL line LIMIT 20, and comes from the Paginator instance defined previously:

paginator = Paginator(books, 20) -> LIMIT 20

Retrieving the second and third pages will make that pattern more clear:

In [12]: second_page = paginator.get_page(2)

In [13]: third_page = paginator.get_page(3)

In [14]: second_page
Out[14]: <Page 2 of 50>

In [15]: third_page
Out[15]: <Page 3 of 50>

In [16]: second_page.object_list.all()
Out[16]: SELECT "core_book"."id",
       "core_book"."title",
       "core_book"."author",
       "core_book"."published_at"
FROM "core_book"
LIMIT 20
OFFSET 20

<QuerySet [<Book: Minute social sometimes guess station impact.>,[...], <Book: Often safe bar between meet fund.>]>

In [17]: third_page.object_list.all()
Out[17]: SELECT "core_book"."id",
       "core_book"."title",
       "core_book"."author",
       "core_book"."published_at"
FROM "core_book"
LIMIT 20
OFFSET 40

<QuerySet [<Book: Animal feel stock southern.>, [...], <Book: Look point wear none six loss.>]>

• The first page will have books from 1st to 20th.
• The second page will have books from 21th to 40th.
• The third page will have books from 41th to the 60th.

The pattern used by the Paginator to se the LIMIT and OFFSET values goes like this:

1. The Paginator starts with the page number: 3
2. Then it calculates the bottom index:

bottom = (number - 1) * self.per_page

bottom = (3 - 1) * 20

bottom = 40

3 - Then it calculates the top index:

top = bottom + self.per_page

top = 40 + 20

top = 60

4 - It then slices the QuerySet using the bottom and top values as boundaries:

In [11]: books[40:60]
Out[11]: SELECT "core_book"."id",
       "core_book"."title",
       "core_book"."author",
       "core_book"."published_at"
FROM "core_book"
LIMIT 20
OFFSET 40

And that allow us to say that books[40:60] is equivalent to third_page.object_list.all()

In [11]: books[40:60]
Out[11]: SELECT "core_book"."id",
       "core_book"."title",
       "core_book"."author",
       "core_book"."published_at"
FROM "core_book"
LIMIT 20
OFFSET 40
<QuerySet [<Book: Animal feel stock southern.>, [...], <Book: Look point wear none six loss.>]>


In [12]: third_page.object_list.all()
Out[12]: SELECT "core_book"."id",
       "core_book"."title",
       "core_book"."author",
       "core_book"."published_at"
FROM "core_book"
LIMIT 20
OFFSET 40
<QuerySet [<Book: Animal feel stock southern.>, [...], <Book: Look point wear none six loss.>]>

The code behind that is available at django/core/paginator.py. And more information about SQL's LIMIT and OFFSET, can be found at Limiting QuerySets.

Any Sequence Is Paginatable

Last but not least, it's important to say that any sliceable sequence is paginatable.

Accordingly to the Django documentation on Paginators:

(A Paginator ) does all the heavy lifting of actually splitting a QuerySet into Page objects.

However, the Paginator class doesn't explicitly validate the received sequence is a QuerySet, and it will work and paginate any sliceable objects. The code below was extracted from the Paginator class, and it shows the page method where the slicing happens:

Code extracted from django/core/paginator.py

def page(self, number):
    """Return a Page object for the given 1-based page number."""
    [...]
	return self._get_page(self.object_list[bottom:top], number, self)

Slicing the Alphabet

To see that any sequence is paginatable, we can make a quick experiment with the Latin Alphabet.

First, create the alphabet as a list of characters:

In [1]: alphabet = list(map(chr, range(97, 123)))

In [2]: alphabet
Out[2]:
['a',
 'b',
 'c',
 'd',
 'e',
 'f',
 'g',
 'h',
 'i',
 'j',
 'k',
 'l',
 'm',
 'n',
 'o',
 'p',
 'q',
 'r',
 's',
 't',
 'u',
 'v',
 'w',
 'x',
 'y',
 'z']

In [3]: len(alphabet)
Out[3]: 26

Instantiate a Paginator passing the alphabet sequence. I'm going to use five letters per page:

In [4]: from django.core.paginator import Paginator

In [5]: paginator = Paginator(alphabet, 5)

With the Paginator created, there must be six pages, having the first page letters a, b, c, d and e:

In [6]: first_five_letters = paginator.get_page(1)

In [7]: first_five_letters
Out[7]: <Page 1 of 6>

In [8]: first_five_letters.object_list
Out[8]: ['a', 'b', 'c', 'd', 'e']

The alphabet was paginated just like a QuerySet, which means that pagination can be used in any sliceable sequence a project might have, not only model queries.

Last Words

Comments, question or feedback are welcome. Feel free to use the comments section below, and subscribe if you want to know when the next article is out.

Thank you!