What's the whole deal with static files?

Static files, as opposed to dynamic files, are files that don't require any processing before being sent to the browser requesting them. Whereas in a Django app, most HTML pages are dynamic (e.g. showing content from the database), static files are seen on the client side the same way they are stored on the server. Static files include all the images, icons and videos you embed on your web pages, your css files and your javascript files. 

In this post, I won't talk about media files, that as far as the client is concerned are also static files, but as far as Django is concerned are handled in an entire different way, usually as files belonging to models (ImageField and FileField are good examples). Your static files are part of your code repository and committed along with your code, unlike media files that are uploaded by users or staff while using your website.

Static files are, as their name says, static, so spawning a full django process when a request comes in just to return a file that's stored on disk is a huge overkill. That's why in a production environment we need to make sure we serve them differently than on our development machine. 

To understand how to achieve this, I'll take you step by step from the simplest of production setups to more complex and also more robust setups.

Part 1: The simplest setup. Static files served directly by your web server

I'll make the assumption that you've got Django running on a production server behind an nginx webserver, with your WSGI scripts (Django) run by gunicorn. This makes it easier to name the various components, but the same principles would apply with Apache/mod_wsgi or with uwsgi instead of gunicorn.

The simplest setup is very similar to your local environment:

• In your code repository, you static files reside inside myapp/static/. If you have multiple apps, each app may contain a /static/ directory. Django looks into these folders for static files by default.
  If you want to add static files elsewhere, use the STATICFILES_DIRS setting to list any path that contains your static files (other than the default /static/ folder in each installed app).
• We will want to move all those static files to one location on your server. This is your STATIC_ROOT setting, which collectstatic uses to move the files for you. I advise you to set this location to somewhere outside of your source code directory, i.e. outside of your Django project. As an example, I have these settings in settings/production.py:

BASE_DIR = os.path.dirname(os.path_dirname(os.path.dirname(os.path.abspath(__file__))))
STATIC_ROOT = os.path.abspath(os.path.join(os.path.dirname(BASE_DIR), 'static'))

The BASE_DIR is my project directory (inside which manage.py lives). This will give me the following directory structure:

/var/www/my_django_site.com
    |_ virtualenv
    |_ static  <-- This is where we'll collect all static files
    |_ my_project  <-- This is your code repository (under git control), i.e. BASE_DIR
        |_ manage.py
        |_ my_project
        |   |_ settings
        |   |   |_ __init__.py
        |   |   |_ development.py
        |   |   |_ production.py   <-- This is my settings file for production
        |   |_ wsgi.py
        |   |_ urls.py
        |   |_ static
        |       |_ ...   <-- non-default, needs to be added to STATICFILES_DIRS
        |_ blogapp
            |_ models.py
            |_ views.py
            |_ static
            |   |_ ...  <-- here are my static files for the blogapp, default location for Django
            |_ ...

• Set your STATIC_URL to "/static/", which means that whenever you use {% static %} in your templates, Django will know it should prepend the path with /static/. 
• Now let's tell our webserver, nginx, where it can find our static files. When it receives a request for the location /static/ (which is what we set STATIC_URL to), we want it to go fetch the corresponding file in your STATIC_ROOT directory. So in your nginx.conf file, you do the following:

location /static/ {
    root /var/www/my_django_site.com;  <-- will search files in /var/www/my_django_site.com/static/
    autoindex off;
    expires +1y;
}

location / {
     proxy_pass http://unix:/tmp/my_django_site.socket;  <-- gunicorn process
     ...
}

We're telling nginx that anything on the path /static/ should be served directly, by-passing gunicorn in the process (which is configured on the catch-all location /). We're also telling nginx to set the expiry header for static files to 1 year, so browsers that have downloaded your files once will not download them again. This is especially useful for site-wide scripts and css, making subsequent pages loading much faster than the first one.

• Finally, ssh into your production machine, go to your source directory (where manage.py is located) and run

../virtualenv/bin/python manage.py collectstatic --settings=my_project.settings

or whatever you do to run django-admin commands on production (you could activate your virtualenv first, your path to production settings might be different).

That's it! Make sure you restart nginx and gunicorn if you just made these changes. Now every time a user requests a static file, nginx is handling it super efficiently, without any python script (and Django framework) being spawned. It's just fetched from disk.

But what if my static files change? Aren't they cached for one year?

Ok, in the above configuration, we have a problem. Images aren't usually a big problem, they don't change. If you replace an image like the logo of your website with another image, you can just give it a different name. No big deal. But still prone to error. Bigger problems are your css and js files. Let's say you have a javascript file app.js that contains all your site's javascript. You'll probably minify it to app.min.js everytime you make changes. 

So now we deploy a new version of app.min.js, run collectstatic to copy it to our STATIC_ROOT folder. A user who visited our website yesterday already downloaded a file called app.min.js and nginx told his browser that it's safe to cache it for 1 year! So he won't see your beautiful changes.

This is where Django provides a smarter static files storage class, the ManifestStaticFilesStorage. The idea being that every time a file changes, it gets moved to your STATIC_ROOT with a different name and everywhere you reference that file in your templates, it gets replaced by the new name. So now in your STATIC_ROOT directory you'll see app.min.55e7cbb9ba48.js and this is the file that will be referenced by your src="{% static 'js/app.min.js' %} statement when Django resolves the template.

There's only one thing you need to do for this to work, in your settings.py:

STATICFILES_STORAGE = 'django.contrib.staticfiles.storage.ManifestStaticFilesStorage'

Now you don't have to worry about filenames yourself. Just update your code, commit, deploy to your production server and run collectstatic. Your users will see the new files because they have a different name in the HTML they download. And until you change them, they will be cached for 1 year in their browser's cache.

Let's summarise where we are now

These are our settings in settings.py for production:

BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
STATICFILES_DIRS = (os.path.join(BASE_DIR, "my_project", "static"), )
STATIC_URL = "/static/"
STATIC_ROOT = os.path.abspath(os.path.join(BASE_DIR, '../static'))
STATICFILES_STORAGE = 'django.contrib.staticfiles.storage.ManifestStaticFilesStorage'

Our single server is letting nginx handle serving our static files. Django is filling in our templates with references to the latest version of our static files (using the ManifestStaticFilesStorage) and our users' browsers cache these files for one year. For completeness, here's my nginx configuration (I run on SSL only, port 80 is redirected to port 443).

server {
        listen 443;
        ssl on;
        ssl_certificate /etc/ssl/cert/www_my_django_site_com_bundle.crt;
        ssl_certificate_key     /etc/ssl/cert/www_my_django_site_com.key;
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

        server_name www.my_django_site.com;

        access_log /var/log/nginx/www.my_django_site.com.access.log;
        error_log /var/log/nginx/www.my_django_site.com.error.log;

        add_header Strict-Transport-Security "max-age=63072000; includeSubdomains;";
        client_max_body_size 25M;

        # All our permanent redirects come here
        rewrite ^/mymodel/(.*)$ /mymodels/$1 permanent;

        location /static/ {
                alias /var/www/my_django_site.com/static/;
                autoindex off;
                expires +1y;
        }

        location / {
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_pass http://unix:/tmp/my_django_site.socket;
        }
}

If you're using an EC2 instance with Elastic Block Storage (EBS) as your root storage, your static files are on EBS and nginx has fast access to your files. For websites with not too much traffic you'll be fine. But there is one issue that even a low traffic website has with this configuration:

Users that experience a bit of lag to reach your EC2 instance (because you host it in eu-west-1, in Ireland, for example, and your users are in the USA) will experience the same lag for each image and each script and each css file on your website. You can't do much about your main HTML page having that lag, but you can make your website feel much faster if you serve all static files from a Content Delivery Network (CDN). And if you compress your files (css and js mainly), using gzip compression.

So in Part 2 of this series on static files, we'll explore:

• How to setup CloudFront (Amazon's CDN) to serve your static files
• Why using django whitenoise makes sense in such a scenario

In Part 3, we'll look at a scenario where we want multiple EC2 instances to handle your traffic:

• How to setup EFS (Elastic File System) to centralise our static files location so it can be accessed by all EC2 instances
• How to setup an autoscaling group where instances are automatically added and removed depending on the traffic requirements.

Finally in Part 4, we'll discuss deployment strategies:

• Accordingly we'll discuss the required deployment steps to update a static file
• How to change the ManifestStaticFilesStorage's behaviour so we can handle a mixed state of old and new instances during a deployment.

So follow me on Twitter at @dirkgroten to keep posted for parts 2, 3 and 4. 

Part 2 is now published here.