Patrick's Software Blog

Learning Python Web Application Development with Flask

How to use Docker and Docker Compose to Create a Flask Application

Introduction

In one of my previous blog posts, I explained WHY I was switching from a traditional deployment approach to using Docker. In this blog post, I’ll dive into the details of HOW to use Docker and Docker Compose to create a Flask web application.

My experience with using Docker has been filled with ups and downs, as it is part exhilarating and part frustrating. I ran into a few roadblocks when configuring my application to work with Docker that were quite frustrating to resolve (which I’ll detail in this blog post). However, the sense of accomplishment when you get a working application running in Docker, and knowing that it will be an exact match to your production environment, is AMAZING!

This blog post is not intended to be an introduction to Docker. If you’re new to Docker, I’d highly recommend ‘Docker for Developers’ by Chris Tankersley ($19.99 recommended price). If you want to gain a solid understanding of all the pieces to Docker and some of the history leading up to Docker, read this book!

Architecture

One of the (good) side-effects of using Docker is needing to think about the overall architecture of your application early in the development process. While it’s definitely beneficial to use a development server (such as the built-in development server that comes with the Flask framework), it’s also beneficial to be able to switch to a production environment using Docker for testing early in the development cycle.

The work that I did with Docker involved using the Flask web application that I’ve been documenting on this site (Flask Tutorial). Here is a diagram that illustrates the structure of the application and how Docker fits into it:

docker-application-architecture

There are four Docker containers used in this architecture:

  1. Web application – Flask web application with the Gunicorn WSGI server
  2. Web server – NGINX
  3. Relational database – PostgreSQL server
  4. Data volume – persistent data storage for Postgres database

The first three components are all created from Docker images that expand on the respective official images from Docker Hub. Each of these images are built using separate Dockerfiles. Docker Compose is then used to create all four containers and connect them correctly into a unified application.

Directory Structure

For a typical Flask application, your directory structure will typically look similar to:

$ tree
.
├── README.md
├── instance
│   ├── db_create.py
│   ├── flask.cfg
├── project
│   ├── __init__.py
│   ├── models.py
│   ├── recipes
│   │   ├── __init__.py
│   │   ├── forms.py
│   │   └── views.py
│   ├── static
│   ├── templates
│   ├── tests
│   └── users
│       ├── __init__.py
│       ├── forms.py
│       └── views.py
├── requirements.txt
└── run.py

By adding the use of Docker to your application, I’d recommend changing the directory structure of your application to:

$ tree
.
├── README.md
├── docker-compose.yml
├── nginx
│   ├── Dockerfile
│   ├── family_recipes.conf
│   └── nginx.conf
├── postgresql
│   └── Dockerfile  * Not included in git repository
└── web
    ├── Dockerfile
    ├── create_postgres_dockerfile.py
    ├── instance
    │   ├── db_create.py
    │   ├── flask.cfg
    ├── project
    │   ├── __init__.py
    │   ├── models.py
    │   ├── recipes
    │   │   ├── __init__.py
    │   │   ├── forms.py
    │   │   └── views.py
    │   ├── static
    │   ├── templates
    │   ├── tests
    │   └── users
    │       ├── __init__.py
    │       ├── forms.py
    │       └── views.py
    ├── requirements.txt
    └── run.py

At first glance, this may seem more complicated, but it’s actually a great change. Your repository is now storing your source code AND the configuration of your application environment. You have the configuration of your web server (NGINX) included, the configuration of your database (Postgres) included, and a way to tie all the pieces together (docker_compose.yml).

As I was preparing to write this blog post, I wanted to check on the NGINX configuration that I had created for a previous web application. It required logging in to my remote server, finding the configuration files, and there was no version history of these files. Being able to store the configuration of your application environment in your repository is so powerful.

Docker Image #1 – NGINX

In order to use NGINX for this web application, we’re going to take the official NGINX image from Docker Hub and then add layers on top of it to configure it for a Flask web application. Let’s take a look at the directory for NGINX in the new directory structure:

$ pwd
.../flask_recipe_app/nginx
$ tree
.
├── Dockerfile
├── family_recipes.conf
└── nginx.conf

This folder contains a Dockerfile and then two configuration files (family_recipes.conf and nginx.conf). The Dockerfile is used to specify how the new image should be created:

FROM nginx:1.11.3
RUN rm /etc/nginx/nginx.conf
COPY nginx.conf /etc/nginx/
RUN rm /etc/nginx/conf.d/default.conf
COPY family_recipes.conf /etc/nginx/conf.d/

This file starts by taking the ‘nginx:1.11.13’ image (either stored locally already on your computer or downloaded from Docker Hub), removing the default NGINX configuration files, and then copying the new NGINX configuration files from this directory to their appropriate locations in the NGINX image.

The configuration of NGINX is a very interesting topic, which I covered in detail in a previous blog post: How to Configure NGINX for a Flask Web Application. To summarize, we’re configuring NGINX to serve static content (CSS, JavaScript, Images, etc.) and to reverse proxy to our WSGI server (Gunicorn) for our Flask application to process requests.

Docker Image #2 – Web Application (including Gunicorn)

The Web Application image stores our Flask web application, the Gunicorn web server, and all of the dependent modules (Flask, SQLAlchemy, etc.). This image is built on top of the official python 3.4.5 Docker image from Docker Hub. I picked this version of python3, as I started developing this application with python 3.4.x. Remember, it’s always best to explicitly state a version number when selecting a base Docker image instead of selecting “xxx:latest”, as this will result in the version changing constantly as new versions of a particular Docker image are added to Docker Hub.

The Dockerfile that defines the Web container is just a single line(!!!):

FROM python:3.4.5-onbuild

Within the vast array of the official Docker images for python, there are a number of images that end with “-onbuild”. These are special Docker images that include a set pattern for creating a standalone python application. To better understand what these images do, let’s look at the source code for the Dockerfile for the python:3.4-onbuild:

FROM python:3.4

RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app

ONBUILD COPY requirements.txt /usr/src/app/
ONBUILD RUN pip install --no-cache-dir -r requirements.txt

ONBUILD COPY . /usr/src/app

This image uses the official python:3.4 image as the base. It is assumed that you are in the top-level of your python application when using this image.

The first steps create a new directory in the Docker image for storing the python application source code (/usr/src/app) and set this directory as the working directory, which is from where all commands will be executed from.

Next, the requirements.txt file is copied into the working directory and all of the dependent modules defined in the requirements.txt file are installed. For our application, we’re installing the Flask web framework, SQLAlchemy, WTForms, etc. Included in this list is also Gunicorn, which is the web server that we’ll be using to run our Flask web application. More details on how to configure Gunicorn are upcoming when we define the docker-compose.yml file for having all of the different containers of our web application run together.

FInally, the source code for our python application is copied to the working directory (/usr/src/app). Our web application image is now complete and ready to use.

Docker Image #3 – Postgres

The official Postgres image from the Docker Hub is a great starting point. This image will create a default user (‘postgres’) and database (‘postgres’) for you, which is convenient for development. However, it is advisable to create a separate user/password for accessing a specific database. This is accomplished by setting the following environment variables within the Postgres Docker image:

  • POSTGRES_PASSWORD
  • POSTGRES_USER
  • POSTGRES_DB

Within my Flask application, I’ve found that creating an ‘instance’ directory to store the sensitive information and then keeping that directory out of my git repository to be a convenient method for attempting to protect the sensitive parameters of the application. Within the ‘instance’ directory, there is a file called ‘flask.cfg’ which defines the Secret Key for my application, the parameters for the Postgres database, and other sensitive parameters that I don’t want others to be able to access. Therefore, storing a Dockerfile for creating the Postgres image in my git repository with all of this sensitive information like username/password is not a good idea.

Luckily, we’re using python so there is an incredible flexibility to solve problems. My solution? Create a script to automatically generate the Postgres Dockerfile by reading the sensitive parameters associated with the Postgres database from the ‘instance’ directory.

In order to get the access to the correct python modules correct, it’s easiest to store this script in the ‘…/flask_recipe_app/web/’ directory:

import os
from project import app


# Postgres Initialization Files
docker_file = 'Dockerfile'
source_dir = os.path.abspath(os.curdir)
destination_dir = os.path.join(source_dir, '../postgresql')

# Before creating files, check that the destination directory exists
if not os.path.isdir(destination_dir):
   os.makedirs(destination_dir)

# Create the 'Dockerfile' for initializing the Postgres Docker image
with open(os.path.join(destination_dir, docker_file), 'w') as postgres_dockerfile:
   postgres_dockerfile.write('FROM postgres:9.6')
   postgres_dockerfile.write('\n')
   postgres_dockerfile.write('\n# Set environment variables')
   postgres_dockerfile.write('\nENV POSTGRES_USER {}'.format(app.config['POSTGRES_USER']))
   postgres_dockerfile.write('\nENV POSTGRES_PASSWORD {}'.format(app.config['POSTGRES_PASSWORD']))
   postgres_dockerfile.write('\nENV POSTGRES_DB {}'.format(app.config['POSTGRES_DB']))
   postgres_dockerfile.write('\n')

This script will create the destination directory (‘…/flask_recipe_app/postgres/’) if it does not exist. Then, the sensitive Postgres parameters are read from the configuration parameters and the Dockerfile is written. Let’s check:

$ pwd
.../flask_recipe_app/postgresql
$ tree
.
└── Dockerfile

Nice! The Postgres image is ready to be used. Be sure to add ‘/postgresql/Dockerfile’ to your .gitignore file to make sure this Dockerfile isn’t included in your git repository.

Docker Compose

Docker Compose allows you to define the structure of your application by utilizing multiple containers. Docker Compose handles so much for you and all you have to do is define a simple docker_compose.yml file.

Docker Compose reads the docker_compose.yml file and builds the applicable ‘docker run’ commands (in the correct order!) to create the multi-container application. While it is possible to create a series of ‘docker run’ commands to build up the multi-container application, Docker Compose simplifies this process significantly. Plus, Docker Compose does a lot of stuff in the background that you don’t even have to worry about, like automatically creating a network for the containers to talk to each other on.

Here is the docker_compose.yml file contents:

version: '2'

services:
 web:
   restart: always
   build: ./web
   expose:
     - "8000"
   volumes:
     - /usr/src/app/project/static
   command: /usr/local/bin/gunicorn -w 2 -b :8000 project:app
   depends_on:
     - postgres

 nginx:
   restart: always
   build: ./nginx
   ports:
     - "80:80"
   volumes:
     - /www/static
   volumes_from:
     - web
   depends_on:
     - web

 data:
   image: postgres:9.6
   volumes:
     - /var/lib/postgresql
   command: "true"

 postgres:
   restart: always
   build: ./postgresql
   volumes_from:
     - data
   ports:
     - "5432:5432"

The first line defines the version of Docker Compose file format. I recommend using version 2, which is the latest.

The next line (‘services:’) starts the definition of each service (think of this as the containers and data volumes) for the application.

The first container that is defined is the web application. The ‘restart’ command should be set to ‘always’ to make sure that the container is always restarted regardless of the exit code. Remember, we’ll be using this same configuration in production, so we want our containers to always restart. The ‘build’ command defines the location of the Dockerfile to build the image with. The ‘expose’ command specifies the port that should be exposed to the other containers on the network, but does not pushed this port to the outside world. The ‘volumes’ command specifies the persistent data to maintain in this container, even on restarts. The ‘command’ command specifies the override of the default command for an image. Since this we’re using Gunicorn as a WSGI server to generate the dynamic content of our application, we want to start the Gunicorn server using ‘/usr/local/bin/gunicorn -w 2 -b :8000 project:app‘ once the container starts running. Finally, the ‘depends_on’ command specifies which service(s) this service depends on. This is important to making sure the containers are started in a proper order.

The second container that is defined is the NGINX container. There are similar commands used for creating this container, but there is a new command: ports. This command exposes the specified port of the container to the specified port of the host (ie. to the outside world). Since NGINX is our web server, we’ll allowing the standard HTTP port (80) to be exposed to the outside world to allow access to our application.

The third container that is defined is the persistent data volume which stores the Postgres database. The use of the ‘command‘ command is used to override the default command for the image. Since this volume is just intended to store persistent data, there is no need to run the full Postgres initialization, so ‘true’ just allows you to skip the standard installation process.

The fourth (and last) container that is defined is the Postgres container. The Dockerfile at ./postgresql is used to build this container. This container exposes the 5432 port (standard Postgres port) to the other containers in the network to allow access to the Postgres database. Due to the sensitive data that can be stored in this file, it should not be stored in your git repository. Additionally, it is automatically generated using the script created above in the Postgres Image section.

To test out the application, build the application components and then run the application in the foreground (ie. not as a daemon) by executing the following in your top-level project directory:

$ docker-compose build
$ docker-compose up

By just running ‘docker-compose up’ without any options, you are running the Docker multi-container application in the foreground. You’ll be able to see all the log information that the containers output. This can be convenient for checking the configuration of a docker_compose.yml file, but it’s preferable to run the application as a daemon (background process):

$ docker-compose up -d

One interesting thing to note is the order in which the containers are started, which we defined using the ‘depends_on’ parameters in the docker_compose.yml file:

$ docker-compose up -d
Creating flaskrecipeapp_data_1
Creating flaskrecipeapp_postgres_1
Creating flaskrecipeapp_web_1
Creating flaskrecipeapp_nginx_1

If you want to see the logs from the different containers:

$ docker-compose logs

You can also see that the individual containers are running:

$ docker ps -a

You can even see the network that Docker Compose automatically created for the application (flaskrecipeapp_default in this case):

$ docker network ls
NETWORK ID          NAME                     DRIVER              SCOPE
714ecc20febe        bridge                   bridge              local               
412dc758466a        flaskrecipeapp_default   bridge              local               
f5b6025063dd        host                     host                local               
6d1d581fe9e3        none                     null                local      

Before we can access our application via a web browser, we need to create the tables in our Postgres database. There is a script in the …/web/instances folder (outside of the git repository) that automatically creates the tables and populates some initial data. This script can be run in the context of Docker:

$ docker-compose run --rm web python ./instance/db_create.py

This script contains some text output to indicate that it was successful.

Let’s test that the application is running… find the IP address of the Docker Machine that you are running:

$ docker-machine ls

Go to your favorite web browser and enter either the IP address. You should see the main page for the Flask application:

screen-shot-2016-10-12-at-10-09-02-pm

Issues Encountered

Coming up with the proper configuration for getting this Flask web application running with Docker did have its challenges. Here are some of the issues that I encountered:

Issue #1 – Issue Encountered While Creating docker-compose.yml

The biggest issue that I encountered with getting my Flask application running in Docker was trying to initialize the Postgres database via a python script. I utilize the …/instance/ folder to store the configuration parameters for my Flask application (flask.cfg) and the script to initialize the Postgres database (db_create.py). Since these files contain a lot of sensitive information (secret key, postgres credentials, admin credentials, etc.), I do not include this folder in my git repository.

During my development work running with the Flask development server on my laptop, I’ve always been able to navigate to the top-level directory of my project and run:

> python instance/db_create.py

Honestly, I took it for granted that this just worked. Here are the sanitized contents of the ‘db_create.py’ file:

from project import db
from project.models import Recipe, User


# Drop all of the existing database tables
db.drop_all()

# Create the database and the database table
db.create_all()

# Insert user data
user1 = User(email=‘***’, plaintext_password='***', role='user')
user2 = User(email='***', plaintext_password='***', role='user')
user3 = User(email='***', plaintext_password='***', role='user')
admin_user = User(email='***', plaintext_password='***', role='admin')
db.session.add(user1)
db.session.add(user2)
db.session.add(user3)
db.session.add(admin_user)

# Commit the changes for the users
db.session.commit()

# Insert recipe data
recipe1 = Recipe('Slow-Cooker Tacos', 'Delicious ground beef that has been simmering in taco seasoning and sauce.  Perfect with hard-shelled tortillas!', admin_user.id, False)
recipe2 = Recipe('Hamburgers', 'Classic dish elevated with pretzel buns.', admin_user.id, True)
recipe3 = Recipe('Mediterranean Chicken', 'Grilled chicken served with pitas, hummus, and sauted vegetables.', user1.id, True)
db.session.add(recipe1)
db.session.add(recipe2)
db.session.add(recipe3)

# Commit the changes for the recipes
db.session.commit()

Here’s what happened… I created and started up the Docker containers using the docker-compose.yml file:

> docker-compose build
> docker-compose up -d

Before being able to actually use the Flask application, the Postgres database must be configured, which is why I run the …/instance/db_create.py file:

> docker-compose run —rm web python instance/db_create.py
Traceback (most recent call last):
  File "./instance/db_create.py", line 6, in <module>
    from project import db
ImportError: No module named 'project'

What!??! I was not expecting to see an error when I ran this script. This error message is saying that the ‘project’ module cannot be found. I had always been trying to avoid this type of error, which is why I was running the script from the ./web/ directory:

> python instance/db_create.py

As opposed to running within the ./web/instance/ directory:

> cd instance
> python db_create.py

By running in the top-level directory, I was expecting the python interpreter to recognize the …/project/ folder as a python module since it included a __init__.py file.

So why is this not the case when running in a Docker container? I’m not exactly clear why this is implemented in this fashion, but running the following command causes the /usr/src/app/instance/ folder to be added to the list of paths for the python interpreter to search through (defined in the PYTHONPATH environment variable). I was really surprised to see this when I added some debug statements to the …/instance/db_create.py script:

root@f132890a7897:/usr/src/app# python ./instance/db_create.py 
sys.path: ['/usr/src/app/instance', '/usr/local/lib/python34.zip', '/usr/local/lib/python3.4', '/usr/local/lib/python3.4/plat-linux', '/usr/local/lib/python3.4/lib-dynload', '/usr/local/lib/python3.4/site-packages']

I really expected the first folder listed to be /usr/src/app instead of /usr/src/app/instance.

Luckily, python provides the tools to solve almost any problem, so I updated …/instance/db_create.py to check if the top-level directory (/usr/src/app) is in PYTHONPATH and to add it if it is not included:

# Check the PYTHONPATH environment variable before beginning to ensure that the
# top-level directory is included.  If not, append the top-level.  This allows
# the modules within the .../project/ directory to be discovered.
import sys
import os

if os.path.abspath(os.curdir) not in sys.path:
    print('...missing directory in PYTHONPATH... added!')
    sys.path.append(os.path.abspath(os.curdir))

<.. rest of file …>

Problem solved, though this took a lot more trial-and-error than I expected would be needed with Docker. Lesson learned (again!)… no technology is perfect for every situation.

Issue #2 – Passing environment variables to a Docker container or image

I honestly don’t know why this was so challenging, but I found so many different syntax examples online for setting environment variables either in docker_compose.yml or Dockerfile. After sorting through the noise, I found that setting environment variables in a Dockerfile using the following format was the most straight-forward and worked successfully:

ENV POSTGRES_DB flask_family_recipes_db

Conclusion

Docker is awesome! The frustrations encountered while trying to configure my Flask application to work with Docker were just part of the learning curve. The excitement in creating a fully functioning web application that is running locally on your laptop is amazing. This isn’t just a development version of your application, this is the production version.

In my next blog post, I’ll discuss how to take this application and deploy it to a production server on DigitalOcean.

The source code for this application can be found on GitLab.

Reference – Key Commands when Running Docker and Docker Compose

Build all of the images in preparation for running your application:
$ docker-compose build

Using Docker Compose to run the multi-container application (in daemon mode):
$ docker-compose up -d

View the logs from the different running containers:
$ docker-compose logs

Stop all of the containers that were started by Docker Compose:
$ docker-compose stop

Run a command in a specific container:
$ docker-compose run –rm web python ./instance/db_create.py
$ docker-compose run web bash

Check the containers that are running:
$ docker ps

Stop all running containers:
$ docker stop $(docker ps -a -q)

Delete all running containers:
$ docker rm $(docker ps -a -q)

Delete all untagged Docker images
$ docker rmi $(docker images | grep “^” | awk ‘{print $3}’)

References

Dockerizing Flask With Compose and Machine – From Localhost to the Cloud (from Real Python Blog):
https://realpython.com/blog/python/dockerizing-flask-with-compose-and-machine-from-localhost-to-the-cloud/

Docker Documentation:
https://docs.docker.com

Docker Compose Documentation:
https://docs.docker.com/compose/gettingstarted/
https://docs.docker.com/compose/compose-file/

Docker for Beginners:
https://prakhar.me/docker-curriculum/

Docker for Developers (eBook) by Chris Tankersley:
http://leanpub.com/dockerfordevs ($19.99)

10 Comments

  1. Thanks, great post. One question on the docker_compose.yml. It looks like you’re exposing Posgres to the host, using the ‘port’ directive, rather than just to other containers like you did with the Flask app and the ‘expose’ directive. Did I miss some detail somewhere? Possibly you plan on running Postgres on another host and that would require the port to be exposed?

    • patkennedy79@gmail.com

      June 23, 2017 at 5:40 am

      Thank you for the comment! This is a great catch… I think I left the use of ‘port’ by mistake in the Postgres service in the docker-compose.yml file so that I could access the Postgres shell. Totally correct that this needs to be ‘expose’ instead of ‘port!

  2. Hi,

    Thank you for this tutorial.
    Your GitLab link is not working, where can I get the code?

    Thanks you in advance,

    Clement

    • patkennedy79@gmail.com

      June 23, 2017 at 5:33 am

      Sorry, I had the access level set for this project to require people to be logged in to GitLab. I’ve updated the settings, so you should be able to view the project without a GitLab account.

  3. Hi,

    The script create_postgres_dockerfile.py needs the app object and app needs flask. But there is no flask outside of the web container… I don’t understand how should it work.
    Please help!

    Thanks

    • patkennedy79@gmail.com

      June 23, 2017 at 5:30 am

      Thanks for the comment! I run the create_postgres_dockerfile.py script prior to running any of the Docker Compose commands. Within the …/flask_recipe_app/web/ directory, you should run ‘python create_postgres_dockerfile.py’. This script will read the applicable configuration parameters related to the Postgres database and write the Dockerfile to the correct folder (…/flask_recipe_app/postgres/). Hope this helps!

  4. Hey,
    I just stumbled across this blog searching for “docker nginx gunicorn flask” This is the best result I found so far. Thanks for the strong write up. I’m currently struggling to incorporate Docker into my Flask development process. I really like the approach of adding the conf files to versioning. I can see this helping me better organize deployments. Thank you!

  5. Thanks for a great tutorial. I am very new to docker and you have explained things in a very simple efficient manner. Found this tutorial really helpful.

    • One question: How should I create the postgres image given the create_postgres_dockerfile.py file. I tried with the following command and its not working. Please help.

      sonal@ubuntuVM:~/code/flask_recipe_app$ docker-compose run –rm web python ./create_postgres_dockerfile.py
      ERROR: build path /home/sonal/code/flask_recipe_app/postgresql either does not exist, is not accessible, or is not a valid URL.

      • patkennedy79@gmail.com

        June 23, 2017 at 5:27 am

        Thanks for the comment! Actually, creating the Dockerfile for Postgres should be done prior to running any of the Docker Compose commands. Within the …/flask_recipe_app/web/ directory, you should run ‘python create_postgres_dockerfile.py’. This script will read the applicable configuration parameters related to the Postgres database and write the Dockerfile to the correct folder (…/flask_recipe_app/postgres/). Hope this helps!

Leave a Reply

Your email address will not be published.

*