Pagination is a technique for splitting large chunks of data across multiple web pages. It makes it for better user experience; not loading all the information available upfront, and also helps you saving some resources since you only query the database for what you truly need at the moment.
When paginating with Django Paginators, the code is not just splitting the content on the UI, but it is truly querying the database in smaller chunks.
Setting Up The Base Project
To illustrate the process of paginating data, I'll go through an example project that you can follow by creating a new Django project, or using the example I created for this post: - django-pagination.
Starting with a Book model:
# pages/core/models.py
class Book(models.Model):
title = models.CharField(max_length=255)
author = models.CharField(max_length=255)
published_at = models.DateField(blank=True)
def __str__(self):
return self.title
A function based view to list all the book instances:
# pages/core/views.py
def book_list(request):
books = Book.objects.all()
return render(request, "core/book_list.html", {"books": books})
And a book_list.html template to display all the book entries. I'll be using bootstrap to style the page. Installation instructions available here: Installing bootstrap.
book_list.html extends from the base.html template, that holds all bootstrap's CSS and JavaScript files. The template structure used can be found at augustogoulart/django-pagination.
Here's the snippet that really matters. It will create an empty table on the UI, readily available to render data:
<!-- pages/core/templates/core/book_list.html -->
{% extends 'base.html' %}
{% block content %}
<div class="row">
<div class="col-8 offset-2">
<table class="table">
<thead>
<tr>
<th scope="col">#</th>
<th scope="col">Author</th>
<th scope="col">Date Published</th>
<th scope="col">Title</th>
</tr>
</thead>
<tbody>
{% for book in books %}
<tr>
<th scope="row">{{ book.id }}</th>
<td>{{ book.author }}</td>
<td>{{ book.published_at }}</td>
<td>{{ book.title }}</td>
</tr>
{% endfor %}
</tbody>
</table>
</div>
</div>
{% endblock content %}
Populating The Database
Now we need to populate the database, so there is data to paginate.
Leveraging the django-extensions library, create a python package called scripts in the root directory of the project, and a python file alongside the __init__.py. I called mine fake_date.py
The fake_data.py script will depend on django-extensions and Faker, so those must be installed:
$ pip install Faker django-extensions
On fake_data.py, here's the code to generate fake book entries:
# scripts/fake_data.py
from faker import Faker
from pages.core.models import Book
def run():
fake = Faker()
Book.objects.bulk_create([Book(
author=fake.name(),
title=fake.sentence(),
published_at=fake.date()
) for book in range(1000)])
Running fake_data.py will populate the project's database with a thousand records. It's necessary to run that script within the context of the Django project, so the models can load properly:
$ python manage.py runscript fake_data
Running the Django project at this point should output a page listing all the thousand records from the database:
Paginating Django
Displaying all that data on a single page is confusing and makes it unnecessarily lengthy. Let's split that huge list into multiple subsets, called Pages:
The Paginator
Adding a paginator to the book_list view:
# pages/core/views.py
from django.shortcuts import render
from django.core.paginator import Paginator
from pages.core.models import Book
def book_list(request):
books = Book.objects.all()
paginator = Paginator(books, 10) # new
return render(request, "core/book_list.html", {"books": books})
Right below books, I'm instantiating a Paginator, and telling it to take the list of books and slice it in chunks of ten books. Each chunk will be a Page.
The Page
The Paginator alone will only orchestrate the pages, but the pages themselves carry the paginated book data.
Retrieving and outputting the first page only:
# pages/core/views.py
from django.shortcuts import render
from django.core.paginator import Paginator
from pages.core.models import Book
def book_list(request):
books = Book.objects.all()
paginator = Paginator(books, 10)
page_obj = paginator.get_page(1) # new
return render(request, "core/book_list.html", {"page_obj": page_obj})
I'll also make some changes to the template and rename the book list from books to page_obj, which is more semantically accurate, since we now have a page in the context. That will also help later on in the article when refactoring the view to a Class-Based view.
<!-- core/book_list.html -->
{% for book in page_obj %}
<tr>
<th scope="row">{{ book.id }}</th>
<td>{{ book.author }}</td>
<td>{{ book.published_at }}</td>
<td>{{ book.title }}</td>
</tr>
{% endfor %}
The page will now display only ten items:
Navigating Pages
Pagination is only useful if the pages can be navigated. Can you imagine a book where only the first page is available?
Page Number From Query String
Let's have the view to look for a page number parameter in the URL's query string:
def book_list(request):
books = Book.objects.all()
paginator = Paginator(books, 10)
page_number = request.GET.get('page') # new
page_obj = paginator.get_page(page_number) # changed
return render(request, "core/book_list.html", {"page_obj": page_obj})
Now the pages can be surfed by passing the page number as an argument to the URL:
http://localhost:8000/?page=2
The second page will display items from the 11th to the 20th:
Switching Pages On The UI
For the pagination to be really useful, people accessing one page should be able to switch pages without manually editing the URL.
The methods used to navigate pages are from the page instance itself, which means that each page knows about the peers around it.
Starting with an HTML boilerplate for the pagination. I'll use bootstrap CSS to help with the styling:
<div class="col-4 offset-4">
<nav aria-label="...">
<ul class="pagination">
<li class="page-item disabled">
<a class="page-link">Previous</a>
</li>
<li class="page-item">
<a class="page-link" href="#">1</a>
</li>
<li class="page-item disabled">
<a class="page-link" href="#">Next</a>
</li>
</ul>
</nav>
</div>
That page number indicator is hardcoded, however, the current page number can be accessed with {{ page_obj.number }}:
<div class="col-4 offset-4">
<nav aria-label="...">
<ul class="pagination">
<li class="page-item disabled">
<a class="page-link">Previous</a>
</li>
<li class="page-item">
<a class="page-link" href="#">{{ page_obj.number }}</a> # new
</li>
<li class="page-item disabled">
<a class="page-link" href="#">Next</a>
</li>
</ul>
</nav>
</div>
Accessing /?page=5 will now show data relative to page number five:
The Previous link can be enabled using the has_previous and previous_page_number methods from the page instance. To handle the first-page scenario, where has_previous is false, and calling previous_page_number would raise an EmptyPage error, some template logic will be required:
<div class="col-4 offset-4">
<nav aria-label="...">
<ul class="pagination">
<!-- new -->
{% if page_obj.has_previous%}
<li class="page-item">
<a href="?page={{ page_obj.previous_page_number }}"><span class="page-link">Previous</span></a>
</li>
{% else %}
<li class="page-item disabled">
<span class="page-link">Previous</span>
</li>
{% endif %}
<!-- /new -->
<li class="page-item">
<a class="page-link" href="#">{{ page_obj.number }}</a>
</li>
<li class="page-item disabled">
<a class="page-link" href="#">Next</a>
</li>
</ul>
</nav>
</div>
Similarly, has_next and next_page_number will handle the next page button, and a similar template logic will handle the last page case:
<div class="col-4 offset-4">
<nav aria-label="...">
<ul class="pagination">
{% if page_obj.has_previous%}
<li class="page-item">
<a href="?page={{ page_obj.previous_page_number }}" class="page-link">Previous</a>
</li>
{% else %}
<li class="page-item disabled">
<span class="page-link">Previous</span>
</li>
{% endif %}
<li class="page-item">
<a class="page-link" href="#">{{ page_obj.number }}</a>
</li>
<!-- new -->
{% if page_obj.has_next %}
<li class="page-item ">
<a class="page-link" href="?page={{page_obj.next_page_number}}">Next</a>
</li>
{% else %}
<li class="page-item disabled">
<span class="page-link">Next</span>
</li>
{% endif %}
<!--/new -->
</ul>
</nav>
</div>
Page /?page=99 will have both a Previous and a Next page:
And /?page=100 won't have a Next page:
Displaying More Pages
Outputting only the current page doesn't make it for good user experience. Some visual flair can help to provide a sense of location and navigability. To do that, it's possible to extend the Previous/Next sections of the template and add an active CSS class to the current page.
Below is the full code for the page switcher. You can find all the templates at augustogoulart/django-pagination
<div class="col-4 offset-4">
<nav aria-label="...">
<ul class="pagination">
{% if page_obj.has_previous %}
<li class="page-item">
<a href="?page={{ page_obj.previous_page_number }}" class="page-link">Previous</a>
</li>
<li>
<a class="page-link" href="?page={{ page_obj.previous_page_number }}">
{{ page_obj.previous_page_number }}
</a>
</li>
{% else %}
<li class="page-item disabled">
<span class="page-link">Previous</span>
</li>
{% endif %}
<li class="page-item active">
<span class="page-link">{{ page_obj.number }}</span>
</li>
{% if page_obj.has_next %}
<li class="page-item">
<a class="page-link" href="?page={{ page_obj.next_page_number }}">
{{ page_obj.next_page_number }}
</a>
</li>
<li class="page-item ">
<a class="page-link" href="?page={{ page_obj.next_page_number }}">Next</a>
</li>
{% else %}
<li class="page-item disabled">
<span class="page-link">Next</span>
</li>
{% endif %}
</ul>
</nav>
</div>
Moving to Class-based Views
With our paginated page in place, one might prefer going with Class-based views instead of a function-based view. That can be achieved by simply substituting the book_list view function by the code below:
# pages/core/views.py
from django.views.generic import ListView
from pages.core.models import Book
class BookListView(ListView):
paginate_by = 10
model = Book
book_list = BookListView.as_view()
From CBV to FBV And Back Again
It is not unrealistic to think that a project may require paginating an existing function-based view (FBV) that is already too large. And although CBVs offers a brilliant simplicity, the project may not be ready to move from FBV to CBV yet, or the team simply may not have the intention to tackle that refactoring now.
One alternative could be delegating the pagination to a model manager. I won't detail model managers because that is out of scope for this article, but you can read more about it here: https://docs.djangoproject.com/en/3.1/topics/db/managers/
To paginate with model managers, start with a managers.py file on the app that will receive the pagination. The code for the new manager could be like this:
from django.core.paginator import Paginator
from django.db import models
class PageQuerySet(models.QuerySet):
def per_page(self, request, per_page):
paginator = Paginator(self, per_page)
page_number = request.GET.get("page")
return paginator.get_page(page_number)
PageManager = models.Manager.from_queryset(PageQuerySet)
Then, apply the PageManager to the target model:
class Book(models.Model):
title = models.CharField(max_length=255)
author = models.CharField(max_length=255)
published_at = models.DateField()
objects = PageManager() # New
def __str__(self):
return self.title
And on the FBV, apply the per_page at the end of you query:
def book_list(request):
page_obj = Book.objects.all().per_page(request, 10)
return render(request, "core/book_list.html", {"page_obj": page_obj})
The result is a nice and clean FBV, with the drawback of passing the request to a model manager.
Behind The Scenes
Knowing how to set-up pagination in Django is great, but understanding what is happening behind the scenes is even better, so bear with me for a couple of paragraphs more.
Installing The Explorer's Toolkit
To explore the mechanics of Django's pagination, some tools will help:
- IPython
- sqlformatter*
Installing IPython with pip:
$ pip install ipython
*By the time of this writing, sqlformatter has a mismatch dependency with Django 3+. If you try installing it from PyPI, you may get an error like this:
django 3.1.1 requires sqlparse>=0.2.2, but you'll have sqlparse 0.1.11 which is incompatible.
I created a PR to fix that, but in the meantime, you can use my fork that updates the conflicting dependency:
$ pip install git+https://github.com/augustogoulart/sqlformatter
If your project's database is empty, use the fake_data.py again to generate some data:
$ python manage.py runscript fake_data
Since IPython is installed, starting the shell_plus from django-extensions will load all the project's models on an IPython session:
$ python manage.py shell_plus
Inspecting the data created:
In [1]: books = Book.objects.all()
In [2]: books
Out[2]: <QuerySet [<Book: Which reason data of.>, [...], <Book: Foot admit attorney political suggest building report.>]>
In [3]: books.count()
Out[3]: 1000Enabling sqlformatter and inspecting the data again, but this time, exploring the SQL run by the ORM. sqlformatter will log to the terminal every time there's a database hit:
In [4]: from sqlformatter import logdb
In [5]: logdb()
Out[5]: True
In [6]: books.count()
SELECT COUNT(*) AS "__count"
FROM "core_book"
Paginating And Exploring The SQL
Paginating the books list and retrieving the first page. At this point, the ORM only queried the database for the COUNT(*) of objects, so the Paginator can set the first and last pages:
In [7]: from django.core.paginator import Paginator
In [8]: paginator = Paginator(books, 20)
In [9]: first_page = paginator.get_page(1)
SELECT COUNT(*) AS "__count"
FROM "core_book"
In [10]: first_page
Out[10]: <Page 1 of 50>
Retrieving the books on the first page:
In [11]: first_page.object_list.all()
Out[11]: SELECT "core_book"."id",
"core_book"."title",
"core_book"."author",
"core_book"."published_at"
FROM "core_book"
LIMIT 20
Only when the application asks for the objects on the first page, the ORM retrieves that data.
That, however, is purely the lazy nature of QuerySets in Django, and the only participation of the Paginator is to tell the ORM how it should limit the results, which is accomplished by the last SQL line LIMIT 20, and comes from the Paginator instance defined previously:
paginator = Paginator(books, 20) -> LIMIT 20
Retrieving the second and third pages will make that pattern more clear:
In [12]: second_page = paginator.get_page(2)
In [13]: third_page = paginator.get_page(3)
In [14]: second_page
Out[14]: <Page 2 of 50>
In [15]: third_page
Out[15]: <Page 3 of 50>
In [16]: second_page.object_list.all()
Out[16]: SELECT "core_book"."id",
"core_book"."title",
"core_book"."author",
"core_book"."published_at"
FROM "core_book"
LIMIT 20
OFFSET 20
<QuerySet [<Book: Minute social sometimes guess station impact.>,[...], <Book: Often safe bar between meet fund.>]>
In [17]: third_page.object_list.all()
Out[17]: SELECT "core_book"."id",
"core_book"."title",
"core_book"."author",
"core_book"."published_at"
FROM "core_book"
LIMIT 20
OFFSET 40
<QuerySet [<Book: Animal feel stock southern.>, [...], <Book: Look point wear none six loss.>]>
- The first page will have books from 1st to 20th.
- The second page will have books from 21th to 40th.
- The third page will have books from 41th to the 60th.
The pattern used by the Paginator to se the LIMIT and OFFSET values goes like this:
- The Paginator starts with the page number:
3 - Then it calculates the bottom index:
bottom = (number - 1) * self.per_page
bottom = (3 - 1) * 20
bottom = 40
3 - Then it calculates the top index:
top = bottom + self.per_page
top = 40 + 20
top = 604 - It then slices the QuerySet using the bottom and top values as boundaries:
In [11]: books[40:60]
Out[11]: SELECT "core_book"."id",
"core_book"."title",
"core_book"."author",
"core_book"."published_at"
FROM "core_book"
LIMIT 20
OFFSET 40And that allow us to say that books[40:60] is equivalent to third_page.object_list.all()
In [11]: books[40:60]
Out[11]: SELECT "core_book"."id",
"core_book"."title",
"core_book"."author",
"core_book"."published_at"
FROM "core_book"
LIMIT 20
OFFSET 40
<QuerySet [<Book: Animal feel stock southern.>, [...], <Book: Look point wear none six loss.>]>
In [12]: third_page.object_list.all()
Out[12]: SELECT "core_book"."id",
"core_book"."title",
"core_book"."author",
"core_book"."published_at"
FROM "core_book"
LIMIT 20
OFFSET 40
<QuerySet [<Book: Animal feel stock southern.>, [...], <Book: Look point wear none six loss.>]>The code behind that is available at django/core/paginator.py. And more information about SQL's LIMIT and OFFSET, can be found at Limiting QuerySets.
Any Sequence Is Paginatable
Last but not least, it's important to say that any sliceable sequence is paginatable.
Accordingly to the Django documentation on Paginators:
(A Paginator ) does all the heavy lifting of actually splitting a QuerySet into Page objects.
However, the Paginator class doesn't explicitly validate the received sequence is a QuerySet, and it will work and paginate any sliceable objects. The code below was extracted from the Paginator class, and it shows the page method where the slicing happens:
Code extracted from django/core/paginator.py
def page(self, number):
"""Return a Page object for the given 1-based page number."""
[...]
return self._get_page(self.object_list[bottom:top], number, self)Slicing the Alphabet
To see that any sequence is paginatable, we can make a quick experiment with the Latin Alphabet.
First, create the alphabet as a list of characters:
In [1]: alphabet = list(map(chr, range(97, 123)))
In [2]: alphabet
Out[2]:
['a',
'b',
'c',
'd',
'e',
'f',
'g',
'h',
'i',
'j',
'k',
'l',
'm',
'n',
'o',
'p',
'q',
'r',
's',
't',
'u',
'v',
'w',
'x',
'y',
'z']
In [3]: len(alphabet)
Out[3]: 26Instantiate a Paginator passing the alphabet sequence. I'm going to use five letters per page:
In [4]: from django.core.paginator import Paginator
In [5]: paginator = Paginator(alphabet, 5)
With the Paginator created, there must be six pages, having the first page letters a, b, c, d and e:
In [6]: first_five_letters = paginator.get_page(1)
In [7]: first_five_letters
Out[7]: <Page 1 of 6>
In [8]: first_five_letters.object_list
Out[8]: ['a', 'b', 'c', 'd', 'e']The alphabet was paginated just like a QuerySet, which means that pagination can be used in any sliceable sequence a project might have, not only model queries.
Last Words
Comments, question or feedback are welcome. Feel free to use the comments section below, and subscribe if you want to know when the next article is out.
Thank you!
