Streamline Your Django Settings With Type Hints: A Pydantic Tutorial

Django projects used to frustrate me because I lacked a robust and scalable way to add new environments. By bringing together pydantic and Python 3.5 type hints, I built the powerful foundation I needed.

As described in PEP 484, type hints support static analysis, but these same annotations are also available at runtime. Third-party packages like pydantic offer runtime type checking that uses this additional metadata. Pydantic uses Python type hints to help manage settings metadata and perform runtime data validation.

This pydantic tutorial will show the far-reaching, positive effects of using pydantic settings management with Django.

Our configuration adheres to the best practices described on the Twelve-Factor App website:

1. Define nonconstant and secret configurations as environment variables.
2. In development environments, define environment variables in a .env file and add the .env to .gitignore.
3. Use the cloud provider’s mechanisms to define (secret) environment variables for the QA, staging, and production environments.
4. Use a single settings.py file that configures itself from the environment variables.
5. Use pydantic to read, check, validate, and typecast environment variables onto Python variables that define the Django configurations.

Alternatively, some developers create multiple settings files, like settings_dev.py and settings_prod.py. Unfortunately, this approach does not scale well. It leads to code duplication, confusion, hard-to-find bugs, and higher maintenance efforts.

Using the aforementioned best practices, adding any number of environments is easy, well-defined, and error-proof. Although we could explore a more complicated environment configuration, we will focus on two for clarity: development and production.

What does this look like in practice?

Pydantic Settings Management and Environment Variables

We now focus on an example in both development and production. We show how each environment configures its settings differently and how pydantic supports each.

Our example application requires a Django-supported database, so we need to store the database connection string. We move the database connection configuration information into an environment variable, DATABASE_URL, using the Python package dj-database-url. Please note that this variable is of type str and is formatted as follows:

postgres://{user}:{password}@{hostname}:{port}/{database-name}
mysql://{user}:{password}@{hostname}:{port}/{database-name}
oracle://{user}:{password}@{hostname}:{port}/{database-name}
sqlite:///PATH

In our development environment, we can use a Docker-contained PostgreSQL instance for ease of use, while in our production environment, we will point to a provisioned database service.

Another variable we want to define is a boolean, DEBUG. The DEBUG flag in Django must never be turned on in a production deployment. It is intended to get additional feedback during development. For example, in debug mode, Django will display detailed error pages when an exception occurs.

Different values for development and production could be defined as follows:

We use the pydantic settings management module to manage these different sets of environment variable values depending on the environment.

Preparatory Steps

To put this into practice, we start configuring our development environment by creating our single .env file with this content:

DATABASE_URL=postgres://postgres:mypw@localhost:5432/mydb
DEBUG=True

Next, we add the .env file to the project’s .gitignore file. The .gitignore file avoids saving potentially sensitive information in source control.

Whereas this approach works well in our development environment, our production environment specification uses a different mechanism. Our best practices dictate that production environment variables use environment secrets. For example, on Heroku, these secrets are called Config Vars and are configured through the Heroku Dashboard. They are made available to the deployed application as environment variables:

After that, we need to adjust the application’s configuration to read these values from either environment automatically.

Configuring Django’s settings.py

Let’s start with a new Django project to provide the essential structure for our example. We scaffold a new Django project with the following terminal command:

$ django-admin startproject mysite

We now have a basic project structure for the mysite project. The project’s file structure is as follows:

mysite/
    manage.py
    mysite/
        __init__.py
        settings.py
        urls.py
        asgi.py
        wsgi.py

The settings.py file contains boilerplate code that allows us to manage the application configuration. It has many predefined default settings that we must adjust to the relevant environment.

To manage these application settings using environment variables and pydantic, add this code to the top of the settings.py file:

import os
from pathlib import Path
from pydantic import (
    BaseSettings,
    PostgresDsn,
    EmailStr,
    HttpUrl,
)
import dj_database_url

# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve().parent.parent

class SettingsFromEnvironment(BaseSettings):
    """Defines environment variables with their types and optional defaults"""

    DATABASE_URL: PostgresDsn
    DEBUG: bool = False

    class Config:
        """Defines configuration for pydantic environment loading"""

        env_file = str(BASE_DIR / ".env")
        case_sensitive = True

config = SettingsFromEnvironment()

os.environ["DATABASE_URL"] = config.DATABASE_URL
DATABASES = {
    "default": dj_database_url.config(conn_max_age=600, ssl_require=True)
}
DEBUG = config.DEBUG

This code does the following:

• Defines a class SettingsFromEnvironment, inheriting from pydantic’s BaseSettings class.
• Defines DATABASE_URL and DEBUG, setting their type and optional default using Python type hints.
• Defines a class Config telling pydantic to look for the variables in a .env file if not present in the system’s environment variables.
• Instantiates the Config class into the object config; the desired variables become available as config.DATABASE_URL and config.DEBUG.
• Defines the regular Django variables DATABASES and DEBUG from these config members.

The same code runs in all environments, and pydantic takes care of the following:

• It looks for the environment variables DATABASE_URL and DEBUG.
  • If defined as environment variables, like in production, it will use those.
  • Otherwise, it pulls those values from the .env file.
  • If it doesn’t find a value, it will do the following:
    • For DATABASE_URL, it throws an error.
    • For DEBUG, it assigns a default value of False.
• If it finds an environment variable, it will check field types and give an error if either of them are wrong:
  • For DATABASE_URL, it verifies that its field type is a PostgresDsn-style URL.
  • For DEBUG, it verifies that its field type is a valid, nonstrict pydantic boolean.

Please note the explicit setting of the operating system’s environment variable from the configuration value for DATABASE_URL. It may seem redundant to set os.environ["DATABASE_URL"] = config.DATABASE_URL because DATABASE_URL is already defined as an external environment variable. However, this allows pydantic to parse, check, and validate this variable. If the environment variable DATABASE_URL is missing or formatted incorrectly, pydantic will give a clear error message. These error checks are invaluable as the application moves from development to subsequent environments.

If a variable is not defined, either a default will be assigned or an error will prompt for it to be defined. Any generated prompts also detail the desired variable type. A side benefit of these checks is that new team members and DevOps engineers more easily discover which variables need to be defined. This avoids the hard-to-find issues that result when the application runs without all the variables defined.

That’s it. The application now has a maintainable, settings-management implementation using a single version of settings.py. The beauty of this approach is that it allows us to specify the correct environment variables in a .env file or any other desired means available through our hosting environment.

The Scalable Path

I have been using Django settings management with pydantic runtime typing in all of my Django projects. I have found it leads to smaller, more maintainable codebases. It also provides a well-structured, self-documenting, and scalable approach to adding new environments.

The next article in this series is a step-by-step tutorial on building a Django application from scratch, with pydantic settings management, and deploying it to Heroku.

The Toptal Engineering Blog extends its gratitude to Stephen Davidson for reviewing the code samples presented in this article.