As a core contributor to OpenFaaS, you’ll find me in the OpenFaaS Slack hanging out and ready to help new users and contributors. You can join the community here. This was post inspired by a question from a new user who had just joined the community. He asked:

“How can I write a function in Python that uses multiple source files so that the main handler.py is able to import helper functions from separate source files.”

Using multiple files to create sub-modules helps keep the code organized and makes reusing code between projects/functions much easier. Functions and variables defined within a module importable into other modules and allows you to scope your function and variable names without worrying about conflicts. From the Python docs:

Each module has its own private symbol table, which is used as the global symbol table by all functions defined in the module. Thus, the author of a module can use global variables in the module without worrying about accidental clashes with a user’s global variables.

Today, I will demonstrate how to do this in OpenFaaS by creating a Python 3 word count function that is split across multiple files.

You should have a basic knowledge of OpenFaaS and Python, but if you’re new you can get up to speed using the workshop. I want to focus on how to take a “one-liner” function and turn it into a larger multi-module project.

Start the function

Next, to start a new Python 3 function, I like to use the flask template because I am going to load a static list of “stop words” into memory when the function starts. Using the flask templates allows the function to do this only once instead of on each invocation.

  • Let’s create a new folder to work in
mkdir -p ~/dev/multi-module
cd ~/dev/multi-module

  • Type in docker login

  • Change --prefix to your Docker Hub account or your private registry

  • Now pull in the python3-flask template and create a new function named wordcount

$ faas-cli template store pull python3-flask
$ faas-cli new wordcount --lang python3-flask --prefix=USERNAME

The project should now look like

.
├── stack.yml
├── template
│   ├── python27-flask
│   │   ├ ...
│   ├── python3-flask
│   │   ├ ...
│   └── python3-flask-armhf
│       ├ ...
└── wordcount
    ├── __init__.py
    ├── handler.py
    └── requirements.txt

Relative imports

Python allows you to reuse code from other files by importing it. For functions, relative imports allow us to specify those resources relative to the file doing the import. This provides maximum flexibility because the import is then agnostic to how/where the package is installed, which is perfect for OpenFaaS functions.

We’re going to start by creating two more files:

touch wordcount/stopwords
touch wordcount/wordcount.py

The project should now look like this

.
├── stack.yml
├── template
│   ├── ...
└── wordcount
    ├── __init__.py
    ├── handler.py
    ├── requirements.txt
    ├── stopwords
    └── wordcount.py

The stopwords is a plain text file of words that will be excluded in the wordcount. These are short common words that you would not want to include in a wordcount visualization such as:

a
an
him
her

This list of words will depend on your use case and local, you should add more to match your needs, for example the 100 most common English words or the 100 most common French words.

All of the code for processing text and generating the counts is found in wordcount.py:

# modified wordcount.py
import unicodedata
import os
from typing import Dict, List

from operator import itemgetter
from collections import defaultdict

FILE = os.path.dirname(__file__)
STOPWORDS = set(
    map(str.lower,
        map(str.strip,
            open(os.path.join(FILE, 'stopwords'), 'r').readlines()
            )
        )
)


def process_text(text: bytes) -> Dict[str, int]:
    """Splits a long text into words a count of interesting words in the text.

    process_text will eliminate any of the stopwords, punctuation, and normalize
    the text to merge cases and plurals into a single value.
    """
    words = text.decode("utf-8").split()
    # remove stopwords
    # remove 's
    words = [
        word[:-2] if word.lower().endswith(("'s",))
        else word
        for word in words
    ]
    # remove numbers
    words = [word for word in words if not word.isdigit()]
    words = [strip_punctuation(word) for word in words]
    words = [word for word in words if word.lower() not in STOPWORDS]

    return process_tokens(words)

def process_tokens(words: List[str]) -> Dict[str, int]:
    """Normalize cases and remove plurals.
    """
    # ...

def strip_punctuation(text: str) -> str:
    # ...

Note that processing the STOPWORDS at the start of the file means it will be loaded into memory once the package is imported rather than on every request, which would create additional latency and I/O overhead.

Using a relative import, we can very easily use the process_text method to create a very simple handler.py:

# handler.py
import json
from .wordcount import process_text


def handle(req):
    """handle a request to the function
    Args:
        req (str): request body
    """

    return json.dumps(process_text(req))

This style of relative imports will work for any file or sub-package you include inside of your function folder. Additionally, your IDE and linter will be able to resolve the imported code correctly!

Deploy and test the function

You should have OpenFaaS deployed and have run faas-cli login already.

$  faas-cli up -f wordcount.yml
# Docker build output ...
Deploying: wordcount.

Deployed. 202 Accepted.
URL: http://127.0.0.1:31112/function/wordcount

$ echo \
  'This is some example text that we want to see a frequency response for.  It has text like apple, apples, apple tree, etc' \
  | faas-cli -f wordcount.yml invoke wordcount

{"example": 1, "text": 2, "want": 1, "see": 1, "frequency": 1, "response": 1, "for": 1, "apple": 3, "tree": 1, "etc": 1}

A note on Python 2 usage

Using the __future__ package you can get the same behavior in your Python 2 functions. Add from __future__ import absolute_import as the first import in handler.py and wordcount.py to ensure that the relative imports are resolved correctly.

Note, Python 2 is End Of Life this year and will not receive any bugfix releases after 2020. If you are still transitioning to Python 3, use the __future__ package to help smooth the transition.

Wrapping up

When we use relative imports then we can easily split our code over several files for better organisation. We could take this further and import from sub-folders or sub-folders of sub-folders. This has the added benefit that the code is valid in both your local environment and the final docker container. Try the completed code example in this repo.

Checkout the OpenFaas Workshop for a step-by-step guide of writing and deploying a Python function detailing the other features of OpenFaas: asynchronous functions, timeouts, auto-scaling, and managing secret values.

For questions, comments and suggestions follow us on Twitter @openfaas and join the Slack community.

Lucas Roesler

Core Team @openfaas. Senior Engineer and Team Lead @contiamo