Patrick's Software Blog

Learning Python Web Application Development with Flask

Unit Testing a Flask Application

Introduction

This blog post provides an introduction to unit testing a Flask application.  I’ve been on a big unit testing kick at work recently which is spilling over into updating the unit tests for my personal projects.  Therefore, I thought it would be a good time to document the basics of unit testing a Flask application.
 
There seems to be a lot of opinions about unit testing amongst software developers, but I believe the following:

  • 100% code coverage using unit tests does not mean your code works, but it’s a good indicator that the code has been developed well
  • No unit tests mean that the code does not work and cannot be expanded upon

 
The second belief might be a bit extreme, but if I inherit a software module that does not have unit tests, I’m already assuming that it does not run properly and has not been maintained.  Simply put, the lack of unit tests leads to a negative impression of the software module.

I believe that you should create an environment for developing unit test that is both easy to use and fun to use. These may sound like very fluffy words, but they have meaning for me:

  • create helper methods to allow quick generation of unit tests
  • provide quick feedback on progress of unit testing
  • create an environment that allows for tests to be added quickly
  • make sure that the unit tests can be executed with a single command
  • provide visual feedback of path coverage

As with so many things in software development, if it’s easy to use, it will be used. If it’s tough to use, it won’t be used as much as it should be.

Unit Test Frameworks

The two unit test frameworks for python that I’ve used are:

  • unittest – built-in unit test framework that is based on the xUnit framework
  • py.test – module for building unit tests

Both work really well, but I tend to prefer unittest since it’s built-in to the python library (there is a wealth of incredible modules built-in to python!).  I’ll be using unittest throughout this blog post.
 
If you want to see just how many options there are for tools to help with unit testing in python, check out this listing of python testing tools.  An incredible number of tools available!

Where to Store Your Unit Tests

Based on the flexibility that using unit test runners gives you, you could probably store your unit test files in any location in your Flask application. However, I find it best to store the unit tests (in this case located in the ‘tests’ directory) at the same level as the files that you are going to be testing:

$ tree
├── instance
├── migrations
├── project
│   ├── __init__.py
│   ├── models.py
│   ├── recipes
│   ├── static
│   ├── templates
│   ├── tests
│   │   ├── test_basic.py
│   │   ├── test_recipes.py
│   │   └── test_users.py
│   └── users
├── requirements.txt
└── run.py

In this structure, the unit tests are stored in the ‘tests’ directory and the unit tests are going to be focused on testing the functionality in the ‘recipes’ and ‘users’ modules, which are at the same level.

Creating a Basic Unit Test File

There are a lot of important principles to follow when writing unit tests, but I really believe it’s important to create an environment that makes writing unit tests easy. I’m a fan of writing unit tests, but I know a lot of software developers that just don’t like it. Therefore, I think it’s important to develop a unit testing structure that has a lot ‘helper’ functions to facilitate writing unit tests.

With that in mind, let’s create a simple unit test file that is not testing our Flask application, but is simply showing the structure of a unit test:

# project/test_basic.py


import os
import unittest

from project import app, db, mail


TEST_DB = 'test.db'


class BasicTests(unittest.TestCase):

    ############################
    #### setup and teardown ####
    ############################

    # executed prior to each test
    def setUp(self):
        app.config['TESTING'] = True
        app.config['WTF_CSRF_ENABLED'] = False
        app.config['DEBUG'] = False
        app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///' + \
            os.path.join(app.config['BASEDIR'], TEST_DB)
        self.app = app.test_client()
        db.drop_all()
        db.create_all()

        # Disable sending emails during unit testing
        mail.init_app(app)
        self.assertEqual(app.debug, False)

    # executed after each test
    def tearDown(self):
        pass


###############
#### tests ####
###############

    def test_main_page(self):
        response = self.app.get('/', follow_redirects=True)
        self.assertEqual(response.status_code, 200)


if __name__ == "__main__":
    unittest.main()

This unit test file creates a class, BasicTests, based on the unittest.TestCase class. Within this class, the setUp() and tearDown() methods are defined. This is really critical, as the setUp() method is called prior to each unit test executing and the tearDown() method is called after each unit test finishes executing.

Since we want to test out the functionality of the web application, this unit test uses a SQLite database for storing data instead of the typical Postgres database that we’ve been using. Why I’ve seen lots of discussions about whether or not this is a good idea, I feel like it really simplifies the development of the unit tests, so I’m in favor of it.

Running the Unit Tests

If you want to run the unit test that was just created, the easiest way is to just execute that file:

$ python project/tests/test_basic.py 
.
----------------------------------------------------------------------
Ran 1 test in 0.122s

OK

Please take note of the directory that the unit test was run from (top-level folder of the Flask application). Why can’t we just run from the directory where the unit test file is located? We would have this option available for simple unit test, but since we’ll importing the ‘app’, ‘db’, and ‘mail’ objects, we need to be at a location where those are discoverable. Therefore, running at the top-level directory of the Flask application allows the python interpreter to find these objects within the ‘project’ directory.

Taking it one step further, I’d highly recommend using Nose2 as the unit test runner. A unit test runner provides the ability to easily detect the unit tests in your project and then execute them.

The easiest way to run Nose2 is simply to call the executable from the top-level directory:

$ nose2

This command will find all of the unit tests (as long as the files start with test_*.py) and execute them. I’m a bit preferential to using the verbose mode:

$ nose2 -v

If you just want to run a single unit test file, you can still use Nose2:

$ nose2 -v project.tests.test_basic
test_main_page2 (project.tests.test_basic.BasicTests) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.130s

OK

I’ve found Nose2 to be so easy to use and I highly recommend using it for running your unit tests.

Adding Helper Methods to Facilitate Developing More Unit Tests

For the unit testing in this blog post, we’re going to be focused on testing some of the user management aspects of the Flask application. As such, we’re likely going to be doing a lot of logging in, logging out, and registering for a new account. Let’s simplify the development of future unit tests by creating some helper methods to perform these steps:

    ########################
    #### helper methods ####
    ########################

    def register(self, email, password, confirm):
        return self.app.post(
            '/register',
            data=dict(email=email, password=password, confirm=confirm),
            follow_redirects=True
        )

    def login(self, email, password):
        return self.app.post(
            '/login',
            data=dict(email=email, password=password),
            follow_redirects=True
        )

    def logout(self):
        return self.app.get(
            '/logout',
            follow_redirects=True
        )

The ‘register’ method registers a new user by sending a POST command to the ‘/register’ URL of the Flask application. The ‘login’ method logs a user in by sending a POST command to the ‘/login’ URL of the Flask application. The ‘logout’ method logs a user out of the application by sending a GET command to the ‘/logout’ URL of the Flask application. These will come in handy when we…

…add a test to make sure we can register a new user:

    def test_valid_user_registration(self):
        response = self.register('patkennedy79@gmail.com', 'FlaskIsAwesome', 'FlaskIsAwesome')
        self.assertEqual(response.status_code, 200)
        self.assertIn(b'Thanks for registering!', response.data)

This unit test checks that we can successfully register a new user and receive a positive confirmation. We’re testing a nominal situation here, but it’s also good to check what happens when off-nominal data is provided to the application. Let’s try registering a new user where the confirmation password does not match the original password:

    def test_invalid_user_registration_different_passwords(self):
        response = self.register('patkennedy79@gmail.com', 'FlaskIsAwesome', 'FlaskIsNotAwesome')
        self.assertIn(b'Field must be equal to password.', response.data)

Try running the unit tests and you should see that all three unit tests pass:

$ nose2 -v project.tests.test_basic
test_invalid_user_registration_different_passwords (project.tests.test_basic.BasicTests) ... ok
test_main_page (project.tests.test_basic.BasicTests) ... ok
test_valid_user_registration (project.tests.test_basic.BasicTests) ... ok

----------------------------------------------------------------------
Ran 3 tests in 3.544s

OK

One of the important checks that is in the application is to prevent duplicate emails being used for registration. Let’s check that this functionality works:

   def test_invalid_user_registration_duplicate_email(self):
        response = self.register('patkennedy79@gmail.com', 'FlaskIsAwesome', 'FlaskIsAwesome')
        self.assertEqual(response.status_code, 200)
        response = self.register('patkennedy79@gmail.com', 'FlaskIsReallyAwesome', 'FlaskIsReallyAwesome')
        self.assertIn(b'ERROR! Email (patkennedy79@gmail.com) already exists.', response.data)

Running the unit tests:

$ nose2 -v project.tests.test_basic
test_invalid_user_registration_different_passwords (project.tests.test_basic.BasicTests) ... ok
test_invalid_user_registration_duplicate_email (project.tests.test_basic.BasicTests) ... ok
test_main_page (project.tests.test_basic.BasicTests) ... ok
test_valid_user_registration (project.tests.test_basic.BasicTests) ... ok

----------------------------------------------------------------------
Ran 4 tests in 10.442s

OK

Excellent!

Please take note that the order that the unit tests are listed in test_*.py does not determine the order in which the unit tests are run. This is really important to understand, as each unit test needs to be self-contained. Don’t count on unit test #1 registering a new user and unit test #2 logging that user in.

Code Coverage

In order to check the code coverage of your unit tests, I’d recommend using the Coverage module. This module works great with unit tests that have been written using the unittest model.

Start by installing the Coverage module and updating your list of modules for your project:

$ pip install coverage
$ pip freeze > requirements.txt

The order of commands to execute when using the coverage module is:

  1. coverage run …
  2. coverage report …

The ‘run’ command runs the unit tests and collects the data for determining the code coverage. The ‘report’ command shows a basic text output of the code coverage.

For example, I’ve written some unit tests for the application and the here is how I use the Coverage module:

$ coverage run project/tests/test_basic.py 
$ coverage report project/users/*.py
Name                        Stmts   Miss  Cover
-----------------------------------------------
project/users/__init__.py       0      0   100%
project/users/forms.py         14      0   100%
project/users/views.py        177    118    33%
-----------------------------------------------
TOTAL                         191    118    38%

If you add the -m flag, you can see which lines are not being tested:

$ coverage report -m project/users/*.py
Name                        Stmts   Miss  Cover   Missing
---------------------------------------------------------
project/users/__init__.py       0      0   100%
project/users/forms.py         14      0   100%
project/users/views.py        177    118    33%   33-35, 69-80, 109-124, 130-136, 141-159, 164-179, 184-206, 212, 218-238, 244-254, 260-266, 272-277
---------------------------------------------------------
TOTAL                         191    118    38%

This text-based output is nice to get a summary of the code coverage, but just seeing line numbers that are not being testing in a file is just not that beneficial. Luckily, the Coverage module is able to generate HTML to provide better insight into which lines are being tested and which are not. You can generate the HTML output by running:

$ coverage html project/users/*.py 

Now you can go to navigate to your project’s folder, open the newly created ‘htmlcov’ directory, and open the index.html file. In your web browser, you should see a summary of the path coverage for this directory:

screen-shot-2016-11-21-at-8-52-40-pm

By clicking on views.py, you get a detailed view of the file contents including a color-coded line-by-line view of which lines are tested by the unit tests and which are not:

screen-shot-2016-11-21-at-8-52-48-pm

Conclusion

This blog post was intended to show you how to develop unit tests for a Flask application with a focus on using the right tools to make the process of writing and running unit tests as easy as possible. The recommended modules for unit testing a Flask application are:
– unittest – built-in python module for developing unit tests
– nose2 – runner for identifying and running the unit tests
– Coverage – seeing the code coverage of your unit tests

To recap, I think that the following aspects are needed to have a functioning setup for writing and running unit tests:
– create helper methods to allow quick generation of unit tests
– provide quick feedback on progress of unit testing
– create an environment that allows for tests to be added quickly
– make sure that the unit tests can be executed with a single command
– provide visual feedback of path coverage

I’ve found that unit tests are critical to testing out any software module that I write. Writing unit tests should be an efficient process that can be greatly simplified by setting up a proper infrastructure.

For the source code utilized in this blog post, please see my GitLab repository.

6 Comments

  1. Hi, I was actually looking forward to seeing how you test Flask applications but unfortunately your intro has very little in terms of Flask. I may be wrong but I think you probably wrote it as a general overview rather than a proper look at unit-testing in anger.

    Just a thought!

  2. Nice article,

    But I find it weird you are mentioning unittests and py.test as framework, and then recommend using nose2.

    nose2 is also a framework, and a test runner. py.test can be a damn good test runner, and it could run both unittests an py.test style tests (maybe even nose style, but I’m not sure, haven’t used nose for years)

    py.test is really powerful, but some of its properties might be not that clearly documented, and some of the concepts are a big hard to grasp.

    But it can help you simplify a look how your tests looks and behaves.

  3. “Ran 4 tests in 10.442s”?

    This is not “unit testing” this is “integration/functional testing”.

    Real unit tests would not need a DB and would run in a matter of milliseconds…

    Good post otherwise 🙂

  4. Nice article!

    Just wanted to mention that you can run standard unittests using pytest as well, as opposed to nose. If you need to install a third-party library to run the tests, might just install pytest and get rid of the “self.assert*” methods using standard “assert” statements instead. 🙂

    • patkennedy79@gmail.com

      November 25, 2016 at 9:55 pm

      Thanks for the comment! Yes, definitely agree with being able to use pytest as a substitute for either unittest and/or nose2. I’ve had more experience with unittest, so that was the main reason that I went in that direction for the blog post.

Leave a Reply

Your email address will not be published.

*