Patrick's Software Blog

Learning Python Web Application Development with Flask

Using Docker for Flask Application Development (not just Production!)

Introduction

I’ve been using Docker for my staging and production environments, but I’ve recently figured out how to make Docker work for my development environment as well.

When I work on my personal web applications, I have three environments:

  • Production – the actual application that serves the users
  • Staging – a replica of the production environment on my laptop
  • Development – the environment where I write source code, unit/integration test, debug, integrate, etc.

While having a development environment that is significantly different (ie. not using Docker) from the staging/production environments is not an issue, I’ve really enjoyed the switch to using Docker for development.

The key aspects that were important to me when deciding to switch to Docker for my development environment were:

  1. Utilize the Flask development server instead of a production web server (Gunicorn)
  2. Allow easy access to my database (Postgres)
  3. Maintain my unit/integration testing capability

This blog post shows how to configure Docker and Docker Compose for creating a development environment that you can easily use on a day-to-day basis for developing a Flask application.

For reference, my Flask project that is the basis for this blog post can be found on GitLab.

Architecture

The architecture for this Flask application is illustrated in the following diagram:

Docker Application Architecture

Each key component has its own sub-directory in the repository:

$ tree
.
├── docker-compose.yml
├── nginx
│   ├── Dockerfile
├── postgresql
│   └── Dockerfile  * Not included in git repository
└── web
    ├── Dockerfile
    ├── create_postgres_dockerfile.py
    ├── instance
    ├── project
    ├── requirements.txt
    └── run.py

Configuration of Dockerfiles and Docker Compose for Production

The setup for my application utilizes separate Dockerfiles for the web application, Nginx, and Postgres; these services are integrated together using Docker Compose.

Web Application Service

Originally, I had been using the python-*:onbuild image for my web application image, as this seemed like a convenient and reasonable option (it provided the standard configurations for a python project). However, in reading the notes in the python page on Docker Hub, the use of the python-*:onbuild images are not recommended anymore.

Therefore, I created a Dockerfile that I use for my web application that creates a non-root user (flask) for the container to run as:

FROM python:3.6.1
MAINTAINER Patrick Kennedy <patkennedy79@gmail.com>

# Create the group and user to be used in this container
RUN groupadd flaskgroup && useradd -m -g flaskgroup -s /bin/bash flask

# Create the working directory (and set it as the working directory)
RUN mkdir -p /home/flask/app/web
WORKDIR /home/flask/app/web

# Install the package dependencies (this step is separated
# from copying all the source code to avoid having to
# re-install all python packages defined in requirements.txt
# whenever any source code change is made)
COPY requirements.txt /home/flask/app/web
RUN pip install --no-cache-dir -r requirements.txt

# Copy the source code into the container
COPY . /home/flask/app/web

RUN chown -R flask:flaskgroup /home/flask

USER flask

It may seem odd or out of sequence to copy the requirements.txt file from the local system into the container separately from the entire repository, but this is intentional. If you copy over the entire repository and then ‘pip install’ all the packages in requirements.txt, any change in the repository will cause all the packages to be re-installed (this can take a long time and is unnecessary) when you build this container. A better approach is to first just copy over the requirements.txt file and then run ‘pip install’. If changes are made to the repository (not to requirements.txt), then the cached intermediate container (or layer in your service) will be utilized. This is a big time saver, especially during development. Of course, if you make a change to requirements.txt, this will be detected during the next build and all the python packages will be re-installed in the intermediate container.

Nginx Service

Here is the Dockerfile that I use for my Nginx service:

FROM nginx:1.11.3
RUN rm /etc/nginx/nginx.conf
COPY nginx.conf /etc/nginx/
RUN rm /etc/nginx/conf.d/default.conf
COPY family_recipes.conf /etc/nginx/conf.d/

There is a lot of complexity when it comes to configuring Nginx, so please refer to my blog post entitled ‘How to Configure Nginx for a Flask Web Application‘.

Postgres Service

The Dockerfile for the postgres service is very simple, but I actually use a python script (create_postgres_dockerfile.py) to auto-generate it based on the credentials of my postgres database. The structure of the Dockerfile is:

FROM postgres:9.6

# Set environment variables
ENV POSTGRES_USER <postgres_user>
ENV POSTGRES_PASSWORD <postgres_password>
ENV POSTGRES_DB <postgres_database>

Docker Compose

Docker Compose is a great tool for connecting different services (ie. containers) to create a fully functioning application. The configuration of the application is defined in the docker-compose.yml file:

version: '2'

services:
  web:
    restart: always
    build: ./web
    expose:
      - "8000"
    volumes:
      - /usr/src/app/web/project/static
    command: /usr/local/bin/gunicorn -w 2 -b :8000 project:app
    depends_on:
      - postgres

  nginx:
    restart: always
    build: ./nginx
    ports:
      - "80:80"
    volumes:
      - /www/static
    volumes_from:
      - web
    depends_on:
      - web

  data:
    image: postgres:9.6
    volumes:
      - /var/lib/postgresql
    command: "true"

  postgres:
    restart: always
    build: ./postgresql
    volumes_from:
      - data
    expose:
      - "5432"

The following commands need to be run to build and then start these containers:

  docker-compose build
  docker-compose -f docker-compose.yml up -d

Additionally, I utilize a script to re-initialize the database, which is frequently used in the staging environment:

  docker-compose run --rm web python ./instance/db_create.py

To see the application, utilize your favorite web browser and navigate to http://ip_of_docker_machine/ to access the application; this will often be http://192.168.99.100/. The command ‘docker-machine ip’ will tell you the IP address to use.

Changes Needed for Development Environment

The easiest way to make the necessary changes for the development environment is to create the changes in the docker-compose.override.yml file.

Docker Compose automatically checks for docker-compose.yml and docker-compose.override.yml when the ‘up’ command is used. Therefore, in development use ‘docker-compose up -d’ and in production or staging use ‘docker-compose -f docker-compose.yml up -d’ to prevent the loading of docker-compose.override.yml.

Here are the contents of the docker-compose.override.yml file:

version: '2'

services:
  web:
    build: ./web
    ports:
      - "5000:5000"
    environment:
      - FLASK_APP=run.py
      - FLASK_DEBUG=1
    volumes:
      - ./web/:/usr/src/app/web
    command: flask run --host=0.0.0.0

  postgres:
    ports:
      - "5432:5432"

Each line in the docker-compose.override.yml overrides the applicable setting from docker-compose.ml.

Web Application Service

For the web application container, the web server is being switched from Gunicorn (used in production) to the Flask development server. The Flask development server allows auto-reloading of the application whenever a change is made and has debugging capability right in the browser when exceptions occurs. These are create features to have during development. Additionally, port 5000 is now accessible from the web application container. This allows the developer to gain access to the Flask web server by navigating to http://ip_of_docker_machine:5000.

Postgres Service

For the postgres container, the only change that is made is to allow access to port 5432 by the host machine instead of just other services. For reference, here is a good explanation of the use of ‘ports’ vs. ‘expose’ from Stack Overflow.

This change allows direct access to the postgres database using the psql shell. When accessing the postgres database, I prefer specifying the URI:

psql postgresql://<username>:<password>@192.168.99.100:5432/<postgres_database>

This allows you access to the postgres database, which will come in really handy at some point during development (almost a guarantee).

Nginx Service

While there are no override commands for the Nginx service, this service will be basically ignored during development, as the web application is accessed directly through the Flask web server by navigating to http://ip_of_docker_machine:5000/. I have not found a clear way to disable a service, so the Nginx service is left untouched.

Alternate Solution with Nginx Service

While there is a nice simplicity to just using the Flask development server (web service) with the Postgres database (postgres service), it is also possible to utilize the Nginx service in development. This approach has the advantage of being more similar to the production environment.

In order to implement this configuration, the docker-compose.override.yml file needs to be changed by updating the highlighted lines:

version: '2'

services:
  web:
    build: ./web
    ports:
      - "8000:8000"
    environment:
      - FLASK_APP=run.py
      - FLASK_DEBUG=1
    volumes:
      - ./web/:/usr/src/app/web
    command: flask run --host=0.0.0.0 --port 8000

  postgres:
    ports:
      - "5432:5432"

With this configuration, you will just navigate to http://ip_of_docker_machine/ to access the application (for example, http://192.168.99.100/). The command ‘docker-machine ip’ will tell you the IP address to use.

Running the Development Application

The following commands should be run to build and run the containers:

docker-compose stop    # If there are existing containers running, stop them
docker-compose build
docker-compose up -d

Since you are running in a development environment with the Flask development server, you will need to navigate to http://ip_of_docker_machine:5000/ to access the application (for example, http://192.168.99.100:5000/). The command ‘docker-machine ip’ will tell you the IP address to use.

Another helpful command that allows quick access to the logs of a specific container is:

docker-compose logs <service>

For example, to see the logs of the web application, run ‘docker-compose logs web’. In the development environment, you should see something similar to:

$ docker-compose logs web
Attaching to flaskrecipeapp_web_1
web_1 | * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
web_1 | * Restarting with stat
web_1 | * Debugger is active!
web_1 | * Debugger pin code: ***-***-***

Conclusion

Docker is an amazing product that I have really come to enjoy using for my development environment. I really feel that using Docker makes you think about your entire architecture, as Docker provides such an easy way to start integrating complex services, like web services, databases, etc.

Using Docker for a development environment does require a good deal of setup, but once you have the configuration working, it’s a great way for quickly developing your application while still having that one foot towards the production environment.

References

Docker Compose File (version 2) Reference
https://docs.docker.com/compose/compose-file/compose-file-v2/

Dockerizing Flask With Compose and Machine – From Localhost to the Cloud
NOTE: This was the blog post that got me really excited to learn about Docker!
https://realpython.com/blog/python/dockerizing-flask-with-compose-and-machine-from-localhost-to-the-cloud/

Docker Compose for Development and Production – GitHub – Antonis Kalipetis
https://github.com/akalipetis/docker-compose-dev-prod
Also, check out Antonis’ talk from DockerCon17 on YouTube.

Overview of Docker Compose CLI
https://docs.docker.com/compose/reference/overview/

Guidance for Docker Image Authors
http://www.projectatomic.io/docs/docker-image-author-guidance/

Docker Command Reference

Start or Re-start Docker Machine:
$ docker-machine start default
$ eval $(docker-machine env default)

Build all of the images in preparation for running your application:
$ docker-compose build

Using Docker Compose to run the multi-container application (in daemon mode):
$ docker-compose up -d
$ docker-compose -f docker-compose.yml up -d

View the logs from the different running containers:
$ docker-compose logs
$ docker-compose logs web # or whatever service you want

Stop all of the containers that were started by Docker Compose:
$ docker-compose stop

Run a command in a specific container:
$ docker-compose run –rm web python ./instance/db_create.py
$ docker-compose run web bash

Check the containers that are running:
$ docker ps

Stop all running containers:
$ docker stop $(docker ps -a -q)

Delete all running containers:
$ docker rm $(docker ps -a -q)

Delete all untagged Docker images
$ docker rmi $(docker images | grep “^” | awk ‘{print $3}’)

9 Comments

  1. Thanks for this, it’s really helpful. I have some questions before I go this route:

    In terms of things like reloading when you make a change to your source code, what is the overhead (in terms of any manual intervention and time) compared to not using Docker?

    I sometimes do development and debugging by dropping an IPython prompt directly into a bit of Python code (`import IPython; IPython.embed()
    `) then triggering the view. This requires having the development server work running in a terminal with both input and output connected. Would this workflow work with the web application being run in Docker?

    You talk about having to re-initialise the DB – do you have to do this in development mode at all? Are their gotchas around postgresql and permanent storage etc.?

    Thanks!

    • patkennedy79@gmail.com

      July 1, 2017 at 4:41 pm

      Thanks for the comment! The turning point for me with using Docker for development was being able to use the Flask development server with the auto-reload capability while using Docker Compose. I don’t notice any difference in terms of time when using Docker, as the auto-reload happens the same way (plus no manual steps are needed once you’re up and running!).

      I am not familiar with IPython, so I don’t know how this configuration would work with trying to drop an Python prompt into the code.

      The re-initialization of the database is something that I like to do during development so that I start out with a fresh configuration. This script does a drop_all(), create_all(), and then sets some pre-canned data into the database. Unfortunately, this script is needed to configure the Postgres database, so it will need to be done when you start off with new containers (if you’ve deleted all of the running containers).

      Hope that helps!

  2. Thank you for sharing your experience Patrick! Looking at your Dockerfile for Web service, do I correctly understand that you run your Flask app as a root inside a container? Wouldn’t it be safer to run it as a “flask” user for example?

    • patkennedy79@gmail.com

      July 2, 2017 at 5:10 am

      Thanks for the comment! After looking into the use of root or non-root users to run the container, it does seem like running as root can be potentially dangerous. Based on your recommendation, I updated the Dockerfile for the web service to create a new group and user to run the container as. Thanks so much for this suggestion!

  3. Why not just use the nginx reverse proxy while developing? Then your environment would totally match that of prod?

    • patkennedy79@gmail.com

      July 1, 2017 at 4:40 am

      Thanks for your comment! I’m so familiar with running with the Flask development server and a database during my development, that I was trying to re-create the configuration. I do like the flexibility (and the closer similarity to the production environment) by actually using the Nginx container instead of ignoring it. I added a sub-section to the ‘Nginx Service’ description for the development configuration to explain how to actually utilize the Nginx container. Really appreciate the recommendation!

  4. One recommendation if you also develop/contribute to other Python libraries could be to use a local virtualenv on your host machine and then mount that inside the docker containers at this path:

    “`
    volumes:
    – “./path/to/virtualenv/lib/python3.5/site-packages:/usr/local/lib/python3.5/site-packages”
    “`

    This also avoids having to rebuild your Docker image whenever you add a pip dependency to requirements.txt (in fact you can leave that build step out of your development Dockerfile together). This can save a lot of time if you would otherwise have to compile certain modules inside your container like lxml — and repeatedly so whenever you add to your reqs.txt.

    Then, to work on a Python module, you just `git clone …` and then `pip install -e .` inside of the git project. This installs the Python module into your virtualenv — which is now also mounted inside your Flask docker container.

  5. Good description, thanks. I use it the same way.

  6. I have been working on a Flask application hosted in Docker the last few weeks and this post fit the bill! I had been wanting to streamline my container use a bit.

    Great tip on the requirements.txt optimization.

Leave a Reply

Your email address will not be published.

*