Jul 9, 2021 7 min read Foundations

In-Depth Guide to Backend Route Design

As with all posts, we try to address issues that appear in codebases from a framework-agnostic and language-agnostic point of view. An issue that does not seem to have enough coverage (in dynamic language frameworks mostly) is the design of backend routes. This issue is not limited to Python and its web frameworks however, it could appear in any framework and any language. We'll use multiple Python frameworks to demonstrate how this issue transcends any specific framework.

Route Design Goals

In general, when designing backend routes, there are some things we'd like to have:

Documentation: document path, query and body parameters to API clients
Validation: validate path, query, and body parameters for each API call
Reusable business logic: what the route actually does on the backend
Error handling: return meaningful and helpful errors to clients

Designing a solution that addresses each of these points is what differentiates a good backend from a great one.

Before expanding on these points, here's a quick refresher on path, query, and body parameters:

Path parameters

We can parametrize routes when it makes sense to do so, for example looking up a user by their ID. This is useful when trying to create easy-to-remember routes (ex. Twitter profiles, LinkedIn profiles, etc.)

api.company.com/users/{user_id}  # user_id is the parameter here

Query parameters

We can add keyword arguments to a route by using query parameters. For example, two query parameters are passed here. Note that this is passed as part of the URL.

api.company.com/search?name=Rami&age=23

This is nice to have when we want to create shareable links, for example sharing a search with certain filters like the path above. An example of this would be a google search results page. Here, query parameters are {"q": "why do we sleep"}.

https://www.google.com/search?q=why+do+we+sleep

Body parameters

Here, things are a little different, as body parameters are part of the request body (separate from request header). These are usually JSON objects, and are typically used with POST/DELETE requests. An example create user POST request to api.company.com/user/create might have the following body:

{
	"name": "Rami",
	"age": "23",
	"email": "rami@softgrade.org"
}

Getting Started

It is now time to introduce Kevin, a backend developer responsible for designing the backend routes for a new service. Previous services at Kevin's company were lacking; clients did not get meaningful errors, validation was sparse, documentation was also sparse and out of date, and the routes logic was messy. Recognizing these issues, Kevin decided to take some time to experiment and improve on the current implementation.

Flask

from utils import is_valid_email

@app.route('/user', methods=['POST', 'GET'])
def user():
    error = None
    if request.method == "POST":
        name = request.form.get("name")
        age = int(request.form.get("age"))
        email = request.form.get("email")
        
        # Validation
        if not name or not age or not email:
        	abort(401)

        if age < 0 or age > 120:
        	abort(401)
        
        if not is_valid_email(email):
        	abort(401)

 	    # Save user object to db
        db.user.save({
        	"name": name,
            "age": age,
            "email": email
        })
 		return {"response": "ok"}
	
    elif request.method == "GET":
    	try:
            user = db.user.get(name=request.args["name"])
            
            # Do not want to return user email
            del user["email"]
            return user
       	except Exception:
        	abort(404)

Validation takes up majority of route, and is flimsy. Next person changing the logic in this route will probably forget to change the validation or just skip it for the sake of time.
During validation, in case errors are encountered, error response that is returned to the user does not indicate what the validation issue was, making it impossible to debug without looking at the backend code or talking to the author.
No documentation, and even if there were more comments, the next update to this route will probably miss updating them.
Business logic here for returning a user could be reused, for example for returning a group of users.

A Model-Based Approach

Three out of four of these issues can be solved with model-based validation. With model-based validation, validation and documentation issues are automatically solved. We can use Pydantic for this purpose, and write the above route with FastAPI instead of Flask.

Note that you can use Pydantic with any Python framework, as it is a standalone library for model validation. It is easier to use with FastAPI since it is integrated by design. I have also written a Pydantic decorator for Sanic routes.

FastAPI

Writing models for each request/response serves as documentation, as the expected inputs are specified exactly, and the validation lives inside the model. These are the model definitions:

# models/user.py

from utils import is_valid_email
from pydantic import BaseModel, validator


class UserIn(BaseModel):
	name: str
    age: str
    email: str

	@validator("age")
    def valid_age(cls, v):
        if v < 0 or v > 120:
            raise ValueError("Must be a valid age")
        return v
        
    @validator("email")
    def valid_email(cls, v):
    	if not is_valid_email(v):
        	raise ValueError("Must be a valid email")
		return v


class UserOut(BaseModel):
	name: str
    age: str

This not only cleans up our routes, but also allows us to reuse the same models along with their validation in different places! Here is what the routes file looks like:

# routes/user.py

from .models.user import UserIn, UserOut


@app.get("/user")
async def get_user(name: str):
	user = db.user.get(name=name)

	# Return UserOut instance from user dict
    return UserOut(**user)
    

@app.post("/user")
async def create_user(user: UserIn):
	# Save user to db
	db.user.save(user.dict())
    return {"response": "ok"}

Using Pydantic, validation errors are very verbose, and returning those would clarify what went wrong to the API user. Storing the validation inside each model makes the code a lot more elegant, because request validation could potentially take hundreds of lines of code from what I've seen.

Whenever a validation error occurs, Pydantic would raise the verbose error, which FastAPI then captures and forwards to the client. This, on top of good documentation and reusability, is what makes model-based validation that much better!

Django

Django, another Python web framework uses models for query/body parameters, and has some validation built into the REST plugin. The only downside is that it's a little complicated to implement when compared to Pydantic, and is not even as flexible. This can be seen by comparing Django REST framework's validators to Pydantic's validators.

Validators - Django REST framework

Django, API, REST, Validators

Django REST frameworkTom Christie

Validators - pydantic

Data validation and settings management using python 3.6 type hinting

logo

Here is their quickstart guide to building a REST API.

Sanic, Flask

Using Pydantic with Sanic or Flask is pretty simple as well, and decorators could be written to wrap the route as a whole. An example of this is this Sanic route decorator that allows for Path, Query, and Body models, and injects them as function arguments.

In the example below, we provide a path model (University) and a query model (StudentIn). This is useful to have in cases where the path model requires some specific validation that's reused in multiple routes, and in cases where the query model accepts many arguments (ex. pagination with count, page, offset, search string, filters, etc.).

# Route example: Get student from a specific university
# /<university>/student?first_name=X&last_name=Y

from pydantic import BaseModel, validator


class University(BaseModel):
    university: str

    @validator("university")
    def is_valid_university(cls, v):
    	if not db.university.get(v):
	    	raise ValueError("Invalid university")
	    return v

class StudentIn(BaseModel):
    first_name: str
    last_name: str


@validate_unwrap(path_model=University, query_model=StudentIn)
def get_uni_student(request, uni: University, student: StudentIn):
	return db.student.get(university=uni.university, first_name=student.first_name, last_name=student.last_name)

Others

In statically typed languages and their web frameworks (Go, C#), we can see that frameworks such as Gin Gonic and .NET have a model-based approach out of the box. Here is a link to "binding models" to routes using Gin-Gonic for example:

In general, we can see how using models to validate and structure requests prevents several issues when designing backend routes. The structure doubles as documentation, the model-specific validation is more likely to stay updated and run wherever the model is used, and errors are potentially more specific and useful (depending on which framework you use to build those models).

What to do with business logic?

The only thing left to do is separate the business logic from the routes. This is a refactoring that would allow us to reuse business logic across different routes, although it is not apparently useful in this simple example. This is usually done by creating "services" on top of having routes and models.

A typical project file structure would look like:

/app
	/routes
    	user.py
    /models
    	user.py
    /services
    	user.py

And a typical service could just be a collection of functions (not recommended) or a class housing several functions (better since isolated in their own namespace, helps avoid name collisions ).

By making the functions below python class methods, we could call them without instantiating the class for example. This makes sense when we need "static" functions that don't depend on a class instance in Python.

# services/user.py

from app.models.user import UserIn, UserOut


class UserService:
	@classmethod
    def create_user(cls, user: UserIn) -> bool:
    	...
    
    @classmethod
    def lookup_user(cls, name: str) -> UserOut:
    	...

Conclusion

Using models in your backend routes provides better validation, documentation, and meaningful errors to clients.

This is a common problem across most web frameworks in dynamic languages due to the unstructured nature of objects by default. This makes good design harder, so introducing a bit of structure helps improve things.

As for business logic in routes, the most elegant approach is to take out any logic completely. Routers can grow big, and having logic inside them would quickly prove unscalable. Having services in parallel to your routers and models solves this problem.

You can find me on Twitter every day @iamramiawar. For more content like this, be sure to subscribe below!

Rami Awar

Delft, Netherlands