How do I configure Django to serve my Robots.txt file ?

Within this article we will look at how to configure Django to serve your ‘robots.txt’ file.

Though there are a number of methods available to achieve this, this example we will only look at the Django method. In that only the the file within your Django project will need to be changed.


Below is an example of the file. The additional lines that are required are displayed via the plus symbol.

Based on this configuration the robots.txt file would need to be placed directly within your templates directory.

  from django.conf.urls.defaults import patterns, include, url
+ from django.views.generic.simple import direct_to_template

  urlpatterns = patterns('',
      url(r'^example_home/$', 'example.views.home'),
+     url(r'^robots\.txt$', direct_to_template, {'template': 'robots.txt', 'mimetype': 'text/plain'}),
Rick Donato

Want to become a Django expert?

Here is our hand-picked selection of the best courses you can find online:
The Complete Web Development Bootcamp course
Django Practical Guide course
Django Full Stack Developer Bootcamp
and our recommended certification practice exams:
AlphaPrep Practice Tests - Free Trial