.py`.
+* Don't forget the `__init__.py` files! Django can only discover the commands if `management` and `commands` are proper Python package directories.
+* The example uses `custom_command`, but you can name your command with whatever valid Python module name that you want.
+
+Unfortunately, we can't slap some Python code into `custom_command.py` and assume that Django will know how to run it. Within the `custom_command.py` module, Django needs to find a `Command` class that subclasses a `BaseCommand` class that is provided by the framework. Django requires this structure to give command authors a consistent way to access features of the command system.
+
+With the `Command` class, you can add a `help` class attribute. Adding help can give users a description of what your command does when running `./manage.py custom_command -h`.
+
+The `Command` class will also help you with handling arguments. If your command needs to work with user input, you'll need to parse that data. Thankfully, the class integrates with Python's built-in `argparse` module. By including an `add_arguments` method, a command can parse the data and pass the results to the command's handler method in a structured way. If you've had to write Python scripts before, then you may understand how much time this kind of parsing can save you (and for those who haven't, the answer is "a lot of time!").
+
+Other smaller features exist within the `Command` class too. Perhaps you only want your command to run if your project has satisfied certain pre-conditions. Commands can use the `requires_migration_checks` or `requires_system_checks` to ensure that the system is in the correct state before running.
+
+I hope it's clear that the goal of the `Command` class is to help you with common actions that many commands will need to use. There is a small API to learn, but the system is a boon to making scripts quickly.
+
+## Command By Example
+
+Let's consider a powerful use case to see a command in action. When you initially start a Django app, all of your app's interaction will probably be through web pages. After all, you were trying to use Django to make a web app, right? What do you do when you need to do something that doesn't involve a browser?
+
+This kind of work for your app is often considered *background* work. Background work is a pretty deep topic and will often involve special background task software like {{< extlink "https://docs.celeryproject.org/en/stable/getting-started/introduction.html" "Celery" >}}. When your app is at an early stage, Celery or similar software can be overkill and far more than you need.
+
+A simpler alternative for some background tasks could be a command paired with a scheduling tool like {{< extlink "https://en.wikipedia.org/wiki/Cron" "cron" >}}.
+
+On one of my projects, I offer free trials for accounts. After 60 days, the free trial ends and users either need to pay for the service or discontinue using it. By using a command and pairing it with the {{< extlink "https://devcenter.heroku.com/articles/scheduler" "Heroku Scheduler" >}}, I can move accounts from their trial status to expired with a daily check.
+
+The following code is very close to what my `expire_trials` command looks like in my app. I've simplified things a bit, so that you can ignore the details that are specific to my service:
+
+```python
+# application/management/commands/expire_trials.py
+
+import datetime
+
+from django.core.management.base import BaseCommand
+from django.utils import timezone
+
+from application.models import Account
+
+
+class Command(BaseCommand):
+ help = "Expire any accounts that are TRIALING beyond the trial days limit"
+
+ def handle(self, *args, **options):
+ self.stdout.write(
+ "Search for old trial accounts..."
+ )
+ # Give an extra day to be gracious and avoid customer complaints.
+ cutoff_days = 61
+ trial_cutoff = timezone.now() - datetime.timedelta(days=cutoff_days)
+ expired_trials = Account.objects.filter(
+ status=Account.TRIALING, created__lt=trial_cutoff
+ )
+ count = expired_trials.update(
+ status=Account.TRIAL_EXPIRED
+ )
+ self.stdout.write(
+ f"Expired {count} trial(s)"
+ )
+```
+
+I configured the scheduler to run `python manage.py expire_trials` every day in the early morning. The command checks the current time and looks for `Account` records in the trialing state that were created before the cutoff time. From that query set, the affected accounts are set to the expired account state.
+
+How can you test this command? There are a couple of approaches you can take when testing a command.
+
+If you need to simulate calling the command with command line arguments, then you can use `call_command` from `django.core.management`. Since the example command doesn't require arguments, I didn't take that approach.
+
+Generally, my preference is to create a command object and invoke the `handle` method directly. In my example above, you can see that the command uses `self.stdout` instead of calling `print`. Django does this so that you could check your output if desired.
+
+Here is a test for this command:
+
+```python
+# application/tests/test_commands.py
+
+from io import StringIO
+
+from application.management.commands.expire_trials import (
+ Command
+)
+from application.models import Account
+from application.tests.factories import AccountFactory
+
+def test_expires_trials():
+ """Old trials are marked as expired."""
+ stdout = StringIO()
+ account = AccountFactory(
+ status=Account.TRIALING,
+ created=timezone.now() - datetime.timedelta(days=65),
+ )
+ command = Command(stdout=stdout)
+
+ command.handle()
+
+ account.refresh_from_db()
+ assert account.status == Account.TRIAL_EXPIRED
+ assert "Expired 1 trial(s)" in stdout.getvalue()
+```
+
+In this test, I constructed a command instance and checked the account state after the command invocation. Also, observe that the `StringIO` instance is injected into the `Command` constructor. By building the command this way, checking the output becomes a very achievable task via the `getvalue` method.
+
+Overall, this scheme of making a command and running it on a schedule avoids all the work of setting up a background worker process. I've been extremely satisfied with how this technique has worked for me, and I think it's a great pattern when your app doesn't have to do a lot of complex background processing.
+
+## Useful Commands
+
+Django is full of {{< extlink "https://docs.djangoproject.com/en/4.1/ref/django-admin/" "useful commands" >}} that you can use for all kinds of purposes.
+{{< web >}}
+Thus far in this series,
+{{< /web >}}
+{{< book >}}
+Thus far in this book,
+{{< /book >}}
+we've discussed a bunch of them, including:
+
+* `check` - Checks that your project is in good shape.
+* `collectstatic` - Collects static files into a single directory.
+* `createsuperuser` - Creates a super user record.
+* `makemigrations` - Makes new migration files based on model changes.
+* `migrate` - Runs any unapplied migrations to your database.
+* `runserver` - Runs a development web server to check your app.
+* `shell` - Starts a Django shell that allows you to use Django code on the command line.
+* `startapp` - Makes a new Django app from a template.
+* `startproject` - Makes a new Django project from a template.
+* `test` - Executes tests that check the validity of your app.
+
+Here is a sampling of other commands that I find useful when working with Django projects.
+
+### `dbshell`
+
+The `dbshell` command starts a different kind of shell. The shell is a database program that will connect to the same database that your Django app uses. This shell will vary based on your choice of database.
+
+For instance, when using PostgreSQL, `./manage.py dbshell`
+will start `{{< extlink "https://www.postgresql.org/docs/current/app-psql.html" "psql" >}}`.
+From this shell, you can execute SQL statements directly to inspect the state of your database. I don't reach for this command often, but I find it very useful to connect to my database without having to remember database credentials.
+
+### `showmigrations`
+
+The `showmigrations` command has a simple job. The command shows all the migrations for each Django app in your project. Next to each migration is an indicator of whether the migration is applied to your database.
+
+Here is an example of the `users` app from one of my Django projects:
+
+```bash
+$ ./manage.py showmigrations users
+users
+ [X] 0001_initial
+ [X] 0002_first_name_to_150_max
+ [ ] 0003_profile
+```
+
+In my real project, I've applied all the migrations, but for this example, I'm showing the third migration as it would appear if the migration wasn't applied yet.
+
+`showmigrations` is a good way to show the state of your database from Django's point of view.
+
+### `sqlmigrate`
+
+The `sqlmigrate` command is *very* handy. The command will show you what SQL statements Django would run for an individual migration file.
+
+Let's see an example. In Django 3.1, the team changed the `AbstractUser` model so that the `first_name` field could have a maximum length of 150 characters. Anyone using the `AbstractUser` model (which includes me) had to generate a migration to apply that change.
+
+From my `showmigrations` output above, you can see that the second migration of my `users` app applied this particular framework change.
+
+To see the Postgres SQL statements that made the change, I can run:
+
+```bash
+$ ./manage.py sqlmigrate users 0002
+BEGIN;
+--
+-- Alter field first_name on user
+--
+ALTER TABLE "users_user" ALTER COLUMN "first_name" TYPE varchar(150);
+COMMIT;
+```
+
+From this, we can tell that Postgres executed an `ALTER COLUMN` {{< extlink "https://en.wikipedia.org/wiki/Data_definition_language" "DDL" >}} statement to modify the length of the `first_name` field.
+
+### `squashmigrations`
+
+Django migrations are a stack of separate database changes that produce a final desired schema state in your database. Over time, your Django apps will accumulate migration files, but those files have a shelf life. The `squashmigrations` command is designed to let you tidy up an app's set of migration files.
+
+By running `squashmigrations`, you can condense an app's migrations into a significantly smaller number. The reduced migrations can accurately represent your database schema, and make it easier to reason about what changes happened in the app's history. As a side benefit, migration squashing can make Django's migration handling faster because Django gets to process fewer files.
+
+## Even More Useful Commands
+
+The commands above come with the standard Django install. Adding in third-party libraries gives you access to even more cool stuff to help with your project development!
+
+A package that I often reach for with my Django projects is the {{< extlink "https://django-extensions.readthedocs.io/en/latest/index.html" "django-extensions" >}} package. This package is full of goodies, including some great optional commands that you can use!
+
+A couple of my favorites include:
+
+### `shell_plus`
+
+How often do you fire up a Django shell, import a model, then do some ORM queries to see the current state of the database? This is something I do *quite* often.
+
+The `shell_plus` command is like the regular shell, but the command will import all your models *automatically*. For the five extra characters of `_plus`, you can save your fingers a lot of typing to import your models and get directly to whatever you needed the shell for.
+
+The command will also import some commonly used Django functions and features like `reverse`, `settings,` `timezone`, and more.
+
+Also, if you have installed a separate REPL like {{< extlink "https://ipython.org/" "IPython" >}}, `shell_plus` will attempt to use the alternate REPL instead of the default version that comes with Python.
+
+### `graph_models`
+
+When I'm live streaming my side projects on {{< extlink "https://www.youtube.com/c/MattLayman" "my YouTube channel" >}}, I will often want to show the model relationships of my Django project. With the `graph_models` command, I can create an image of all my models and how those models relate to each other (using UML syntax). This is a great way to:
+
+* Remind myself of the data modeling choices in my apps.
+* Orient others to what I'm doing with my project.
+
+This particular command requires some extra setup to install the right tools to create images, but the setup is manageable and the results are worth it.
+
+Aside from `shell_plus` and `graph_models`, there are 20 other commands that you can use that may be very useful to you. You should definitely check out django-extensions.
+
+## Summary
+
+{{< web >}}
+In this article,
+{{< /web >}}
+{{< book >}}
+In this chapter,
+{{< /book >}}
+you saw Django commands.
+
+We covered:
+
+* Why commands exist in the Django framework
+* How commands work
+* How to create your own custom command and how to test it
+* Useful commands from the core framework and the django-extensions package
+
+{{< web >}}
+In the next article,
+{{< /web >}}
+{{< book >}}
+In the next chapter,
+{{< /book >}}
+we're going to look into performance.
+
+You'll learn about:
+
+* How Django sites get slow
+* Ways to optimize your database queries
+* How to apply caching to save processing
+
+{{< web >}}
+If you'd like to follow along with the series, please feel free to sign up for my newsletter where I announce all of my new content. If you have other questions, you can reach me online on Twitter where I am {{< extlink "https://twitter.com/mblayman" "@mblayman" >}}.
+{{< /web >}}
+
diff --git a/content/understand-django/2022-01-19-go-fast.pt.md b/content/understand-django/2022-01-19-go-fast.pt.md
new file mode 100644
index 0000000..0077d3f
--- /dev/null
+++ b/content/understand-django/2022-01-19-go-fast.pt.md
@@ -0,0 +1,453 @@
+---
+title: "Go Fast With Django"
+description: >-
+ How do you make your Django app fast? You measure what is slow, scale your system when necessary, and use a combination of fast database queries and strategic caching. In this Understand Django article, we'll explore those topics and more to help you get a performant Django app.
+image: img/django.png
+type: post
+categories:
+ - Python
+ - Django
+tags:
+ - Python
+ - Django
+ - performance
+ - caching
+series: "Understand Django"
+
+---
+
+In the last
+{{< web >}}
+[Understand Django]({{< ref "/understand-django/_index.pt.md" >}}) article,
+{{< /web >}}
+{{< book >}}
+chapter,
+{{< /book >}}
+we learned about commands. Commands are the way to execute scripts that interact with your Django app.
+
+{{< web >}}
+With this article,
+{{< /web >}}
+{{< book >}}
+With this chapter,
+{{< /book >}}
+we're going to dig into performance. How do you make your Django site faster? Keep reading to find out.
+
+{{< understand-django-series-pt "performance" >}}
+
+## Theory Of Performance
+
+{{< web >}}
+>There are two ways to make a website faster:
+>
+> * Do **more** work.
+> * Do **less** work.
+{{< /web >}}
+{{< book >}}
+There are two ways to make a website faster:
+
+ * Do **more** work.
+ * Do **less** work.
+{{< /book >}}
+
+How we do more work *and* less work depends on the type of work that we're trying to address on the site.
+
+When we're talking about doing more work, what I'm really saying is that we often need to increase the ***throughput*** of a site. Throughput is a measure of work over time. By increasing throughput, a site can serve more users concurrently.
+
+A very natural throughput measure on a Django site is *requests per second*. A page view on a site could contain multiple requests, so requests per second isn't a perfect analog to how many users your site can handle, but it's still a useful measure to help you reason about site performance.
+
+On the flip side of doing more work, what does it mean to do less work to improve performance?
+
+> The fastest code is no code.
+
+Every line of code that you write must be processed by a computer when it runs. If your code is inefficient or if too much is running, that will naturally mean that it will take longer to produce a final result.
+
+The time from when an input happens to when an output is received is called ***latency***. If a user clicks on a link on your site, how long does it take for them to get a response? That time delay is latency.
+
+Less work doesn't just mean that you're running too much code! There are a lot of factors that might contribute to latency, some of them are easier to optimize than others.
+
+* Inefficient code - mistakes in development can make a computer slower than necessary
+* Data size - sending more data requires more effort to deliver to users
+* Geographic location - the speed of light is a real limit on network communication
+* and more!
+
+If you can reduce latency on your site, you can improve the experience of the people using the site.
+
+{{< web >}}
+In the rest of this article,
+{{< /web >}}
+{{< book >}}
+In the rest of this chapter,
+{{< /book >}}
+you'll learn how you can do both more and less work to make a better site that will benefit your users.
+
+## Measure First
+
+Before optimizing, we have to recognize what kind of work is impacting an app. In other words, *what is the resource constraint that is preventing an app from performing better?*
+
+Measuring applications, especially those that are running with live traffic, can be a tricky endeavor. I think we can look at an app from a zoomed out macro level or a zoomed in point of view.
+
+I would start my analysis from inspecting the system overall for patterns. Broadly, you will find that performance tends to fall into a couple of major bottleneck categories:
+
+* I/O bound
+* CPU bound
+
+*I/O bound* means that the system is limited (that's the "bound" part) by the inputs and outputs of the system. Since that's still a really vague statement, let's make it more concrete. An I/O bound system is one that is waiting for work to be available. Classic examples include:
+
+* waiting for responses from a database
+* waiting for content from a file system
+* waiting for data to transfer over a network
+* and so on
+
+Optimizing an I/O bound system is all about minimizing those waiting moments.
+
+Conversely, *CPU bound* systems are systems that are drowning in immediate work to calculate. The computer's **C**entral **P**rocessing **U**nit can't keep up with all that it's being asked to do. Classic examples of CPU bound work are:
+
+* Data science computations for machine learning
+* Image processing and rendering
+* Test suite execution
+
+Optimizing a CPU bound system focuses heavily on making calculations faster and cheaper.
+
+Since we understand that an application that is underperforming is likely to be I/O bound or CPU bound, we can start to look for these patterns within the system. The easiest initial signal to observe is the CPU load. If the processors on your production machines are running at very high CPU utilization, then it's a pretty clear indicator that your app is CPU bound. In my estimation, you'll rarely see this for web applications. **Most underperforming web applications are likely to be I/O bound.**
+
+The reason that web apps are often I/O bound has to do with their typical function. Most apps are fetching data from a database and displaying it to a user. There isn't a massive amount of computation (comparatively to something like machine learning) that the app needs to do. Thus, your app is probably waiting around for data from the database.
+
+If you observe that the system is not CPU bound, then the next action is to dig deeper and find where the application is spending its time waiting. But how can we do this? Philosophically, you should now have an ok understanding of what to be looking for, but what tools can you use to accomplish the task of measurement?
+
+We need to rewind a bit. A moment ago, I also made an assumption that you know how to find the CPU load of your application. That may not be the case. Let's look at some tools that help you categorize where your app's resource bottleneck is.
+
+The easiest way to monitor basic resource information about your app including CPU, memory, disk usage, and more may come from your hosting vendor. My preferred hosting vendor, Heroku, displays all these kinds of metrics on a single page so I can assess system performance at a glance. Other vendors like Digital Ocean or AWS provide tools to help you see this information too.
+
+If your hosting vendor doesn't provide these tools, then you'll have to use other techniques. Presumably, if you're using a Virtual Private Server (VPS) for hosting, you have access to the server itself via ssh. On the server that's running your application, you can use a program like `top`. This is a classic program for checking which processes are using the "top" amount of resources. `top` will show the list of processes ordered by what is consuming the most CPU and will refresh the order of the list every second to provide a current snapshot in time. (**Tip**: use `q` to quit `top` after you start it.)
+
+While `top` is useful and gets the job done to learn about CPU usage, it's not exactly the friendliest tool out there. There are alternatives to `top` that may offer a better user experience. I personally find `top` sufficient, but I know `htop` is a popular alternative.
+
+If you don't have tools from your hosting provider and don't want to use ssh to log into a server, there are other options to consider. Broadly, this other category of tools is called Application Performance Monitoring (APM). APM tools are vendors that will monitor your application (go figure!) if you install the tool along with your app. These tools help show *both* CPU problems and I/O issues. Application performance is very important to businesses, so the software industry is full of vendors to choose from with a wide range of features.
+
+To see what these tools can be like for free, you might want to check out {{< extlink "https://www.datadoghq.com/" "Datadog" >}} which has a free tier (Datadog is not a sponsor, I've just used their service, enjoyed it, and know that it's free for a small number of servers). Other popular vendors include {{< extlink "https://scoutapm.com/" "Scout APM" >}} and {{< extlink "https://newrelic.com/" "New Relic" >}}.
+
+Finally, we've reached a point where you can diagnose the performance constraints on your application using a wide variety of tools or services. Let's see how to fix problems you may be experiencing!
+
+## Do More
+
+We can address throughput and do more by tuning a few different knobs.
+
+When thinking about doing more, try to think about the system in two different scaling dimensions:
+
+* Horizontally
+* Vertically
+
+Horizontal and vertical scaling are methods of describing *how* to do more in software systems.
+
+Let's relate this to a silly example to give you a good intuitive feel for scaling. Imagine that you need to move large bags of dirt (approximately 40 lbs / 18 kg per bag) to plant a huge garden. The job is to unload the hundreds of bags from a delivery truck to your imaginary back yard. You enlist the help of your friends to get the job done.
+
+One strategy is to get your *strongest* friends to help. Maybe there aren't as many of them, but their strength can make quick work of moving the bags. This is *vertical scaling*. The additional power of your friends allows them to move the bags more easily than someone with an average or weaker build.
+
+Another strategy is to get *lots* of friends to help. Maybe these friends can't move as many bags as the stronger ones, but many hands make light work. This is *horizontal scaling*. The increased number of people allows the group to move more bags because more individuals can do the work simultaneously.
+
+We can apply this same thinking to computer systems.
+
+### Vertical Scaling
+
+To achieve vertical scaling, you would run your application on a more powerful computer. Cloud vendors give you all kinds of tools to do this. Picking a faster computer is naturally going to cost you more, so vendors make many options available (check out {{< extlink "https://aws.amazon.com/ec2/instance-types/" "this page from AWS" >}} to see the dizzying array of options).
+
+When should you think about vertical scaling? One natural case is when your application is CPU bound. If the processor is struggling to process the requests from an application, a faster processor may help. With a higher clock speed from a faster individual CPU, a computer will be able to process an individual request faster.
+
+Moving to a larger computer is typically considered vertical scaling, but it may be possible to have horizontal effects by moving to a larger computer because of how modern computers are designed. These days, larger computers typically come with a higher number of CPUs. Each individual CPU may be faster than a smaller computer configuration *and* there will be more CPUs on the single machine. Because of this characteristic, you will likely need to change your application configuration to take advantage of the additional power supplied by the extra CPU cores. While the traditional definition of vertical scaling (i.e., a faster individual CPU can do work quicker than a slower one) still applies, the line between vertical and horizontal scaling is somewhat blurred because of the multi-CPU core paradigm of modern CPUs.
+
+{{< web >}}
+In the Understand Django deployment article,
+{{< /web >}}
+{{< book >}}
+In the deployment chapter,
+{{< /book >}}
+we discussed Gunicorn's `--workers` flag. Recall that Python application servers like Gunicorn work by creating a main process and a set of worker processes. The main process will distribute incoming network connections to the worker processes to handle the actual traffic on your website. If you vertically scale the server machine from a size that has 1 CPU to a machine that has 2, 4, or more CPUs, and you don't change the number of workers, then you'll waste available CPU capacity and won't see most of the benefits from the upgrade in server size.
+
+If modern vertical scaling uses more CPUs when moving to a bigger machine, then what is horizontal scaling? The difference is primarily in the number of computers needed to do the scaling. Vertical scaling changes a single machine to achieve more throughput. Horizontal scaling pulls multiple machines into the equation.
+
+### Horizontal Scaling
+
+Conceptually, how does horizontal scaling work? With the vertical scaling model, you can see a clear connection between users making a request to your website's domain and a single machine handling those requests (i.e., the main process from your application server distributes requests). With the horizontal model, we're now discussing multiple computers. How does a single domain name handle routing to multiple computers? With more computers!
+
+Like the main process that distributes requests, we need a central hub that is able to route traffic to the different machines in your horizontally scaled system. This hub is usually called a **load balancer**. A load balancer can be used for multiple things. I see load balancers used primarily to:
+
+* route traffic to the different application servers in a system
+* handle the TLS certificate management that makes HTTPS possible
+
+Since the load balancer doesn't do most of the actual work of processing a request, your system can increase its throughput by increasing the number of application servers. In this setup, each application server "thinks" that it is the main server that's handling requests. The load balancer behaves like a client that's making requests on behalf of the actual user. This kind of configuration is called a proxy setup.
+
+If you want to learn more about horizontal scaling with a load balancer, then I suggest you check out {{< extlink "https://www.nginx.com/" "Nginx" >}} (pronounced "engine X"), {{< extlink "http://www.haproxy.org/" "HAProxy" >}} (which stands for "high availability proxy"), or {{< extlink "https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html" "AWS ALBs" >}} (for "application load balancer"). These tools are commonly used and have a reputation for being strong load balancers.
+
+### What's Better?
+
+*What are the tradeoffs between horizontal scaling and vertical scaling?*
+
+When you add more pieces to a system, you're increasing the complexity of the system. Thus, vertical scaling can, at least initially, produce a design with lower operational complexity. Personally, if I ran a service on some VPS like Digital Ocean or AWS, I would probably reach for vertical scaling first. A bigger machine would allow me to use a higher number of concurrent worker processes to increase throughput, and I would avoid the complexity of deploying multiple application servers.
+
+In reality, I run my side projects on a Platform as a Service, Heroku. With my choice of Heroku, the service already includes a load balancer by default. This means that I can trivially scale horizontally by changing a setting in Heroku that will start multiple application servers.
+
+While vertical scaling may be a good fit if you don't have an existing load balancer, that scaling path does have downsides to consider.
+
+First, in a vertically scaled world, downtime on your server could mean downtime *for your service*. Whether a site is reachable or not reachable is called "availability" by the software industry. If your entire site is tied to a large vertically scaled server, then it can act as a single point of failure if there is a problem.
+
+Secondly, a vertically scaled service may potentially have more cost for you. In my experience, most websites have high and low periods of usage throughout the day. For instance, my current employer is a US healthcare company that provides telemedicine visits for people that need to speak with a doctor virtually. When it's the middle of the night in the US, the site utilization is naturally lower as most people are sleeping.
+
+One common cost optimization is to use fewer computing resources during periods of lower utilization. On a vertically scaled service, it is harder to change computer sizes quickly. Thus, that computing resource's usage is relatively fixed, even if no one is using your service. In contrast, a horizontally scaled service can be configured to use "auto-scaling."
+
+Auto-scaling is the idea that the infrastructure can be resized dynamically, depending on the use of the site. During periods of high activity, more computers can be added automatically to join the load balancer distribution and handle additional load. When activity dies down, these extra machines can be removed from use. This cost saving technique helps ensure that your system is only using what it needs.
+
+The truth is that if your system reaches a large enough size and scale, then picking horizontal or vertical scaling is a false choice. As a system matures and grows, you may need to have a mix of the two scaling types, so that your service has the characteristics that you need (like availability).
+
+I hope that I've helped equip you with some mental modeling tools. With these tools, you should have some idea of how to handle more traffic when your site becomes wildly popular. ๐
+
+In this section, we focused on increasing throughput by changing your service's infrastructure to handle more load. Now let's shift from the macro point of view to the micro view and talk about how to improve throughput by doing less.
+
+## Do Less
+
+*How do you make your Django site do less work?* We should always measure, but since I believe most websites are I/O bound, let's focus on techniques to improve in that dimension.
+
+### Optimizing Database Queries
+
+The most common performance problem that I've encountered with Django applications is the N+1 query bug (some people will describe it as the 1+N query bug for reasons that may become evident in a moment).
+
+The N+1 bug occurs when your application code calls the database in a loop. How many queries are in this made up example?:
+
+```python
+from application.models import Movie
+
+movies = Movie.objects.all()
+for movie in movies:
+ print(movie.director.name)
+```
+
+It's a bit of a trick question because you might have a custom manager (i.e., `objects`), but, in the simplest scenario, there is one query to fetch the movies, and one query for each director.
+
+The reason for this behavior is that Django does a lazy evaluation of the movies `QuerySet`. The ORM is not aware that it needs to do a join on the movie and director tables to fetch all the data. The first query on the movie table occurs when the iteration happens in the Python `for` loop. When the `print` function tries to access the `director` foreign key, the ORM does not have the director information cached in Python memory for the query set. Django must then fetch the director data in another database query to display the director's name.
+
+This adds up to:
+
+* 1 query for the movies table
+* N queries on the director table for each iteration through the `for` loop
+
+Hence the name, "N+1" query bug.
+
+The reason that this is so bad is because calling the database is way slower than accessing data in Python memory. Also, this problem gets worse if there are more rows to iterate over (i.e., more movies to process and, thus, more directors to fetch individually).
+
+The way to fix this issue is to hint to Django that the code is going to access data from the deeper relationship. We can do that by hinting to the ORM with `select_related`. Let's see the previous example with this change:
+
+```python
+from application.models import Movie
+
+movies = Movie.objects.select_related(
+ "director").all()
+for movie in movies:
+ print(movie.director.name)
+```
+
+In the reworked example, the ORM will "know" that it must fetch the director data. Because of this extra information, the framework will fetch from both the movie and director tables *in a single query* when the `for` loop iteration starts.
+
+Under the hood, Django performs a more complex `SELECT` query that includes a join on the two tables. The database sends back all the data at once and Django caches the data in Python memory. Now, when execution reaches the `print` line, the `director.name` attribute can pull from memory instead of needing to trigger another database query.
+
+The performance savings here can be massive, especially if your code works with a lot of database rows at once.
+
+While `select_related` is fantastic, it doesn't work for all scenarios. Other types of relationships like a many to many relationship can't be fetched in a single query. For those scenarios, you can reach for `prefetch_related`. With this method, Django will issue a smaller number of queries (usually 1 per table) and blend the results together in memory. In practice, `prefetch_related` operates very much like `select_related` in most circumstances. Check out the Django docs to understand more.
+
+### Caching Expensive Work
+
+If you know:
+
+* execution will likely happen many times
+* is expensive to create, AND
+* won't need to change
+
+then you're looking at work that is a very good candidate to cache. With caching, Django can save the results of some expensive operation into a very fast caching tool and restore those results later.
+
+A good example of this might be a news site. A news site is very "read heavy," that is, users are more likely to use the site for viewing information than for writing and saving information to the site. A news site is also a good example because users will read the same article, and the content of that article is fixed in form.
+
+Django includes tools to make it simple to work with the cache to optimize content like our news site example.
+
+The simplest of these tools is the `cache_page` decorator. This decorator can cache the results of an entire Django view for a period of time. When a page doesn't have any personalization, this can be a quick and effective way to serve HTML results from a view. You can find this decorator in `django.views.decorators.cache`.
+
+You may need a lower level of granularity than a whole page. For instance, your site might have some kind of logged in user and a customized navigation bar with a profile picture or something similar. In that scenario, you can't really cache the whole page and serve that to multiple users, because other users would see the customized navigation bar of the first user who made the request. If this is the kind of situation you're in, then the `cache` template tag may be the best tool for you.
+
+Here's a template example of the `cache` tag in use:
+
+{{< web >}}
+```django
+{% load cache %}
+
+Hi {{ user.username }}, this part won't be cached.
+
+{% cache 600 my_cache_key_name %}
+ Everything inside of here will be cached. The first argument to `cache` is how long this should be cached in seconds. This cache fragment will cache for 10 minutes. Cached chunks need a key name to help the cache system find the right cache chunk.
+
+ This cache example usage is a bit silly because this is static text and there is no expensive computation in this chunk.
+{% endcache %}
+```
+{{< /web >}}
+{{< book >}}
+```djangotemplate
+{% load cache %}
+
+Hi {{ user.username }}, this part won't be cached.
+
+{% cache 600 my_cache_key_name %}
+ Everything inside of here will be cached. The first argument to `cache` is how long this should be cached in seconds. This cache fragment will cache for 10 minutes. Cached chunks need a key name to help the cache system find the right cache chunk.
+
+ This cache example usage is a bit silly because this is static text and there is no expensive computation in this chunk.
+{% endcache %}
+```
+{{< /book >}}
+
+With this scheme, any expensive computation that your template does will be cached. *Be careful with this tag!* The tag is useful if computation happens during rendering, but if you're doing the evaluation and fetching *inside of your view* instead of at template rendering time, then you're unlikely to get the benefits that you want.
+
+Finally, there is the option to use the cache interface directly. Here's the basic usage pattern:
+
+```python
+# application/views.py
+from django.core.cache import cache
+
+from application.complex import calculate_expensive_thing
+
+def some_view(request):
+ expensive_result = cache.get(
+ "expensive_computation")
+ if expensive_result is None:
+ expensive_result = calculate_expensive_thing()
+ cache.set(
+ "expensive_computation",
+ expensive_result
+ )
+
+ # Handle the rest of the view.
+ ...
+```
+
+On the first request to this view, the `expensive_result` won't be in the cache, so the view will calculate the result and save it to the cache. On subsequent requests, the expensive result can be pulled from the cache. In this example, I'm using the default timeout for the cache, but you can control the timeout values when you need more control. The cache system has plenty of other cool features, so check it out in the docs.
+
+As fair warning, caching often requires more tools and configuration. Django works with very popular cache tools like Redis and Memcached, but you'll have to configure one of the tools on your own. The Django documentation will help you, but be prepared for more work on your part.
+
+Database optimization and caching are go-to techniques for optimization. When you're optimizing, how do you know that you're doing it right? What gains are you getting? Let's look at some tools next that will let you answer those questions.
+
+## Tools To Measure Change
+
+We'll look at tools at an increasing level of complexity. This first tool is one that is massively useful while developing in Django. The other tools are more general purpose tools, but they still are worth knowing about, so that you'll know when to reach for them.
+
+Each of these tools helps you get real performance data. By measuring the before and after of your changes, you can learn if the changes are actually producing the gains that you expect or hope to achieve.
+
+### Django Debug Toolbar
+
+The {{< extlink "https://django-debug-toolbar.readthedocs.io/en/latest/index.html" "Django Debug Toolbar" >}} is a critical tool that I add to my projects. The toolbar acts as an overlay on your site that opens to give you a tray of different categories of diagnostic information about your views.
+
+Each category of information is grouped into a "panel," and you can select between the different panels to dig up information that will assist you while doing optimization work.
+
+You'll find panels like:
+
+* SQL
+* Templates
+* Request
+* Time
+
+The SQL panel is where I spend nearly all of my time when optimizing. This panel will display all the queries that a page requests. For each query, you can find what code triggered the database query and you even get the exact SQL `SELECT`. You can also get an `EXPLAIN` about a query if you really need the gory details of what the database is doing.
+
+With a little bit of eye training, you'll learn to spot N+1 query bugs because you can see certain queries repeated over and over and "cascading" like a waterfall.
+
+I'll often test with the debug toolbar when I'm trying to sprinkle in `select_related` to visually confirm that I've reduced the query count on a page. The debug toolbar is open source and is a great free resource. The toolbar is totally worth the investment of configuring it for your next Django project.
+
+### hey / ab
+
+There are two tools that are very similar that I use when I need to get a crude measure of the performance of a site. These tools are {{< extlink "https://github.com/rakyll/hey" "hey" >}} and {{< extlink "https://httpd.apache.org/docs/2.4/programs/ab.html" "ab" >}} (Apache Bench). Both of these tools are *load generators* that are meant to benchmark a site's basic performance characteristics.
+
+In practice, I prefer `hey`, but I mention `ab` because it is a well known tool in this space that you are likely to encounter if you research this load generator topic.
+
+Operating this kind of tool is trivial:
+
+```bash
+$ hey https://www.example.com
+```
+
+In this example, hey will try to open up a large number of concurrent connections to the URL and make a bunch of requests. When the tool is done, it will report how many of the requests were successful and some related timing information and statistics. Using a load generator like this lets you synthesize traffic to learn how your site is going to perform.
+
+I'd suggest you be careful where you tell these tools to operate. If you're not careful, *you could cause a Denial of Service attack on your own machines.* The flood of requests might make your site unavailable to other users by consuming all your server's resources. Think twice before pointing this at your live site!
+
+### Locust
+
+The previous load generator tools that I mentioned act as somewhat crude measurements because you're limited to testing a single URL at a time. What should you do if you need to simulate traffic that matches real user usage patterns? Enter {{< extlink "https://locust.io/" "Locust" >}}. Locust is not a tool that I would reach for casually, but it is super cool and worth knowing about.
+
+The goal of Locust is to do load testing on your site in a realistic way. This means that it's your job to model the expected behavior of your users in a machine understandable way. If you know your users well (and I hope you do), then you can imagine the flows that they might follow when using your site.
+
+In Locust, you codify the behavior patterns that you care about, then run Locust to simulate a large number of users that will act like you expect (with randomness to boot to really make the test like reality).
+
+Advanced load testing is something you may never need for your site, but it's pretty cool to know that Python has you covered if you need to understand performance and the limits of your site at that deep level.
+
+### Application Performance Monitoring (APM)
+
+{{< web >}}
+Earlier in this article,
+{{< /web >}}
+{{< book >}}
+Earlier in this chapter,
+{{< /book >}}
+I mentioned that Application Performance Monitoring tools can show you CPU and memory utilization of your site. That's usually just the tip of the iceberg.
+
+An APM tool often goes far beyond hardware resource measurement. I like to think of APMs as a supercharged version of the debug toolbar.
+
+First, an APM is used on live sites typically. The tool will collect data about real requests. This lets you learn about the *real* performance problems on the site that affect *real* users.
+
+For instance, New Relic will collect data on slow requests into "traces." These traces are aggregated into a set to show you which pages on your site are the worst performers. You can drill into that list, view an individual trace, and investigate the problem.
+
+Maybe you've got an N+1 bug. Maybe one of your database tables is missing an index on an important field, and the database is scanning too many records during `SELECT` statements. These traces (or whatever they are called in other services) help you prioritize what to fix.
+
+In fact, an APM highlights the true value of measurement. If I can leave you with a parting thought about optimization, think about this: ***optimize where it counts***.
+
+Here's a simple thought experiment to illustrate what I mean. You have an idealized system that does two things repeatedly:
+
+* One task (A) is 90% of all activity on the site.
+* The other task (B) is the remaining 10%.
+
+If you have to pick a target to try to optimize because your system performance is inadequate, which one do you pick?
+
+Let's assume that you know an optimization for each type of task that could cause the task to execute in 50% of the time. If implementing each optimization is the same level of effort, then there is a clear winner as to which task you should optimize. You could either:
+
+* Optimize A for 90% * 50% for a total system saving of 45%.
+* Optimize B for 10% * 50% for a total system saving of 5%.
+
+In most circumstances, spend your optimization effort on the area that will have outsized impact (i.e., pick task A as much as you can). Sometimes the hard part is figuring out which task is A and which task is B. Monitoring tools like an APM can help you see where the worst offenders are so you can focus your limited time in the right spot.
+
+## Summary
+
+{{< web >}}
+In this article,
+{{< /web >}}
+{{< book >}}
+In this chapter,
+{{< /book >}}
+we looked into making Django apps go fast.
+
+We saw:
+
+* A mental model for thinking about performance optimization
+* Different types of performance bottlenecks
+* How to get your system to do more by either horizontal or vertical scaling
+* How to get your app to do less work by optimizing database queries and caching
+* Tools to aid you in all of this optimization work
+
+{{< web >}}
+In the next article,
+{{< /web >}}
+{{< book >}}
+In the next chapter,
+{{< /book >}}
+we'll look into security.
+
+You'll learn about:
+
+* How Django helps you be more secure with some of its design features
+* What those different warnings from `./manage.py check --deploy` mean
+* Fundamental things you should consider to help keep your site secure
+
+{{< web >}}
+If you'd like to follow along with the series, please feel free to sign up for my newsletter where I announce all of my new content. If you have other questions, you can reach me online on Twitter where I am {{< extlink "https://twitter.com/mblayman" "@mblayman" >}}.
+{{< /web >}}
+
diff --git a/content/understand-django/2022-03-10-secure-apps.pt.md b/content/understand-django/2022-03-10-secure-apps.pt.md
new file mode 100644
index 0000000..62f9dba
--- /dev/null
+++ b/content/understand-django/2022-03-10-secure-apps.pt.md
@@ -0,0 +1,296 @@
+---
+title: "Security And Django"
+description: >-
+ You want to protect your users' privacy, right? The goal is noble and users demand it, but how do you do it? In this Understand Django article, we'll look at some areas that improve the security of your application.
+image: img/django.png
+type: post
+categories:
+ - Python
+ - Django
+tags:
+ - Python
+ - Django
+ - security
+series: "Understand Django"
+
+---
+
+In the last
+{{< web >}}
+[Understand Django]({{< ref "/understand-django/_index.pt.md" >}}) article,
+{{< /web >}}
+{{< book >}}
+chapter,
+{{< /book >}}
+we learned about where apps slow down. We explored techniques that help sites handle the load and provide a fast experience for users.
+
+{{< web >}}
+With this article,
+{{< /web >}}
+{{< book >}}
+With this chapter,
+{{< /book >}}
+we will look at security. How does a Django site stay safe on the big, bad internet? Let's find out.
+
+{{< understand-django-series-pt "security" >}}
+
+## A Security Confession
+
+I have a confession to make. Of all the topics that I've covered about Django
+{{< web >}}
+in this series,
+{{< /web >}}
+{{< book >}}
+in this book,
+{{< /book >}}
+*this is my least favorite one.*
+{{< web >}}
+Perhaps that's why I've pushed the subject so far into this list of articles.
+{{< /web >}}
+
+I have a very hard time getting excited about security because it feels like a pure cost to me. As developers, we're in this arms race against malicious people who want to steal and profit from the data of others. In a perfect world, everyone would respect the privacy of others and leave private data alone. Alas, the world is far from perfect.
+
+The bad actors have devised clever and tricky methods of exploiting websites to steal data. Because of this, application developers have to implement guards in an attempt to prevent these exploits. Implementing those guards detract from the main objective of site building and often feels like a drag on efficiency.
+
+All that being said, **security is super important**. Even if you're like me and the topic doesn't naturally interest you (or actively feels like a waste of time), the security of your application matters.
+
+* Privacy matters.
+* Trust matters.
+
+If we cannot protect the information that users of our Django sites bring, then trust will rapidly erode and, most likely, your users will disappear along with it.
+
+As noted in this section, security is not my favorite topic. I'm going to describe some security topics as they relate to Django, but if you want to learn from people who *love* security, then I would recommend reading from the {{< extlink "https://owasp.org/" "Open Web Application Security Project" >}}. This popular group can teach you far more about security than I can, and do it with gusto!
+
+## The Three **C**s
+
+Learning about security involves learning a bunch of acronyms. I don't know if this is something that security researchers like to do, or if the reason is because the problems that the acronyms stand for are challenging to understand. Either way, let's look at three common acronyms that start with C and the problems they address.
+
+### CSRF
+
+{{< web >}}
+In a number of these Understand Django articles,
+{{< /web >}}
+{{< book >}}
+In a number of these chapters,
+{{< /book >}}
+I have discussed CSRF briefly.
+{{< web >}}
+In the forms article,
+{{< /web >}}
+{{< book >}}
+In the forms chapter,
+{{< /book >}}
+I did some hand waving and stated that you need a CSRF token for security reasons and basically said "trust me" at the time.
+
+CSRF stands for *Cross Site Request Forgery*. In simple terms, a CSRF attack allows an attacker to use someone's credentials to a different site without their permission. With a bit of imagination, you can see where this goes:
+
+* Attacker socially manipulates a user to click a link.
+* The click activity exploits the user's credentials to a site and changes something about the user's account like their email address.
+* The attacker changes the email address to something they control.
+* If the original site is something like an e-commerce site, the attacker may make purchases using the user's stored credit card information.
+
+Django includes a capability to help thwart this kind of attack. Through the use of **CSRF tokens**, we can help prevent bad actors from performing actions without user consent.
+
+A CSRF token works by including a generated value that gets submitted along with the form. The template looks like:
+
+{{< web >}}
+```django
+
+```
+{{< /web >}}
+{{< book >}}
+```djangotemplate
+
+```
+{{< /book >}}
+
+When this renders, the result would be something like:
+
+```html
+
+```
+
+The value would naturally be different from my example. When the form is submitted, the CSRF token gets checked for validity. A valid CSRF token is required to make a `POST` request, so this level of checking can help prevent attackers from changing a user's data on your site.
+
+You can learn more about CSRF with Django's {{< extlink "https://docs.djangoproject.com/en/4.1/ref/csrf/" "Cross Site Request Forgery protection" >}} reference page.
+
+### CORS
+
+Imagine that you've built `myapp.com`. For your user interface, instead of Django templates, you built a client UI using a JavaScript framework like {{< extlink "https://vuejs.org/" "Vue.js" >}}. Your application is serving static files at `myapp.com`, and you built a Django-powered API that is handling the data which gets called at `api.myapp.com`.
+
+In this scenario, browsers will require you to set up CORS. CORS is *Cross-Origin Resource Sharing*. The goal of CORS is to help protect a domain from undesirable access.
+
+In our example, your API at `api.myapp.com` may only be designed to work with the user interface at `myapp.com`. With CORS, you can configure your API so that it will reject any requests that do not come from the `myapp.com` domain. This helps prevent bad actors from using `api.myapp.com` in the browser.
+
+Django does *not* include tools to handle CORS configuration from the core package. To make this work, you'll need to reach for a third party package. Since CORS configuration is handled through HTTP headers, you'll find that the very appropriately named {{< extlink "https://github.com/adamchainz/django-cors-headers" "django-cors-headers" >}} package is exactly what you need.
+
+I won't walk through the whole setup of that package because the README does a good job of explaining the process, but I will highlight the crucial setting. With django-cors-headers, you need to set the `CORS_ALLOWED_ORIGINS` list. Anything not in that list will be blocked by CORS controls in the browser. Our example configuration would look like:
+
+```python
+CORS_ALLOWED_ORIGINS = [
+ "https://myapp.com",
+]
+```
+
+As you read about CORS on the internet, you'll probably run into advice to set the HTTP header of `Access-Control-Allow-Origin: *`. This wildcard is what you'll get if you set `CORS_ALLOW_ALL_ORIGINS = True` in django-cors-headers. *This is probably not what you really want. Using this feature opts your site out of CORS protection.* Unless you have some public web API that is *designed* to work for many domains, you should try to avoid opting out of CORS.
+
+CORS is not a core concept that you will find in Django. If you want to learn more about the specifics of CORS, check out {{< extlink "https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS" "Cross-Origin Resource Sharing (CORS)" >}} from the Mozilla Developer Network (MDN).
+
+### CSP
+
+The final **C** in our tour is *Content Security Policy* or CSP, for short. You might roughly think of CSP as the inverse of CORS. Where CORS defines what parts of the internet can access your domain, CSP defines what your domain can access from the internet.
+
+The goal of CSP is to protect users on your site from running JavaScript (and other potentially harmful resources like images) from places that you don't want.
+
+To understand how your site can be vulnerable to these kinds of attacks, we need to understand a well-known attack vector called Cross-Site Scripting (XSS). XSS is when a bad actor finds a way to run a script on your domain. Here's a classic way that XSS can happen:
+
+* A site has a form that accepts text data.
+* Then the site displays that text data *in its raw form*.
+
+At first, that seems harmless:
+
+```html
+
+ What could possibly go wrong?
+
+```
+
+For an honest interaction like "What could possibly go wrong?" as user input, that is truly harmless. What about this?
+
+```html
+
+ I am getting sneaky.
+
+```
+
+Now, the user added a bit of HTML markup. Again, this is fairly benign and will only add some unanticipated italics. What if the user is a bit more clever than that?
+
+```html
+
+
+
+```
+
+Here's where a site gets into trouble. If this is rendered on a page, a little alert box will appear. You can imagine this happening in a forum or some other sharing site where multiple people will see this output. That's annoying, but it's still not horrible.
+
+What does a really bad scenario look like? A really bad scenario is where the bad guys figure out that your site is unsafe in this way. Consider this:
+
+```html
+
+
+
+```
+
+Now your users are really in trouble. In this final version, the bad guys won. A user's browser will download and execute whatever JavaScript is in `owned.js`. This code could do all kinds of stuff like using the `fetch` API to run AJAX requests that can change the user's account credentials and steal their account.
+
+How do we defend against this kind of attack? There isn't a singular answer. In fact, multiple layers of protection is often what you really want. In security, this idea is called "defense-in-depth." If you have multiple layers to protect your site, then the site may become a less appealing target for attackers.
+
+For this particular scenario, we can use a couple of things
+
+* HTML escaping of untrusted input
+* CSP
+
+The real problem above is that the site is rendering user input without any modification. This is a problem with HTML because the raw characters are interpreted as HTML code and not just user data.
+
+The simplest solution is to make sure that any characters that mean something specific to HTML (like `<` or `>`) are replaced with escape codes (`<` or `>`) that will display the character in the browser without treating it like the actual HTML code character. **Django does this auto-escaping of user data by default**. You can disable this behavior for portions of a template using a variety of template tags like `autoescape` and the (ironically named?) `safe` tag.
+
+Because there are ways to opt out of safe behavior from HTML escaping and because clever attackers might find other ways to inject script calls into your site, CSP is another layer of protection.
+
+Primarily, CSP is possible with a `Content-Security-Policy` HTTP header. You can read all of the gritty details on the {{< extlink "https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP" "Content Security Policy (CSP)" >}} article on MDN. Like CORS, CSP is not something that Django supports out-of-the-box. Thankfully, Mozilla (yep, the same Mozilla from MDN), offers a {{< extlink "https://django-csp.readthedocs.io/en/latest/index.html" "django-csp" >}} package that you can use to configure an appropriate policy for your Django site.
+
+In a content security policy, you mark everything that you want to allow. This fundamentally changes what requests your site will connect to. Instead of allowing everything by default, the site operates on a model that denies things by default. With a "deny by default" stance, you can then pick resources which you deem are safe for your site. Modern browsers respect the policy declared by the HTTP header and will refuse to connect to resources outside of your policy when users visit your site at your domain.
+
+There is something obvious that we should get out of the way. This setup and configuration requires *more work*. Having more secure systems requires effort, study, and plenty of frustration as you make a site more secure. The benefits to your users or customers is that their data stays safe. I think most people expect this level of protection by default.
+
+So, do you have to become a security expert to build websites? I don't think so. There is a set of fundamental issues with web application security that you should know about (and we've covered some of those issues already), but you don't need to be prepared to go to black hat conventions in order to work on the web.
+
+{{< web >}}
+Before finishing up this security article,
+{{< /web >}}
+{{< book >}}
+Before finishing up this security chapter,
+{{< /book >}}
+let's look at what Django provides so that you can be less of a security expert and use the knowledge of the community.
+
+## Check Command Revisited
+
+The Django documentation includes a good overview of the framework's security features on the {{< extlink "https://docs.djangoproject.com/en/dev/topics/security/" "Security in Django" >}} page.
+
+Aside from the content outlined on the security page, we can return to the check command
+{{< web >}}
+discussed in previous articles.
+{{< /web >}}
+{{< book >}}
+discussed in previous chapters.
+{{< /book >}}
+Recall that Django includes a `check` command that can check your site's configuration before you deploy a site live. The command looks like:
+
+```bash
+$ ./manage.py check --deploy
+```
+
+The output from this command can show where your configuration is less than ideal from a security perspective.
+
+The security warnings that come from running the `check` command are defined in `django.core.checks.security`. A more readable version of the available security checks is on the {{< extlink "https://docs.djangoproject.com/en/4.1/ref/checks/#security" "System check framework" >}} reference page.
+
+Scanning through the list of checks, you'll find that:
+
+* many checks center around configuring your site to run with HTTPS. Secure connections used to reference SSL for Secure Sockets Layer. Along the way, that layer changed names to TLS for Transport Layer Security. In practice, if you see either of those terms, think `https://`.
+* other checks confirm that your site has the kinds of {{< extlink "https://docs.djangoproject.com/en/4.1/ref/middleware/#security-middleware" "middleware" >}} installed that offer some of the protection discussed previously (like CSRF).
+* still other checks look for core settings that should be set like `DEBUG = False` and defining the `ALLOWED_HOSTS` setting.
+
+There is a good comment in the Django security checks reference docs that is worth repeating here:
+
+> The security checks do not make your site secure. They do not audit code, do intrusion detection, or do anything particularly complex. Rather, they help perform an automated, low-hanging-fruit checklist, that can help you to improve your siteโs security.
+
+When you're thinking through security, do some homework and don't let your brain go on autopilot. Remember that users develop a trust relationship with your websites. That trust is easy to break and may cause people to leave your site forever.
+
+## Summary
+
+{{< web >}}
+In this article,
+{{< /web >}}
+{{< book >}}
+In this chapter,
+{{< /book >}}
+we explored security topics and how they relate to Django.
+
+We covered:
+
+* CSRF
+* CORS
+* CSP
+* Cross-site scripting (XSS)
+* The security checks and information available from the `check` command
+
+{{< web >}}
+In the *last* article of the Understand Django series,
+{{< /web >}}
+{{< book >}}
+In the *last* chapter,
+{{< /book >}}
+we'll get into debugging.
+
+You'll learn about:
+
+* Debugging tools like `pdb`
+* Browser tools
+* Strategies for finding and fixing problems
+
+{{< web >}}
+If you'd like to follow along with the series, please feel free to sign up for my newsletter where I announce all of my new content. If you have other questions, you can reach me online on Twitter where I am {{< extlink "https://twitter.com/mblayman" "@mblayman" >}}.
+{{< /web >}}
+
diff --git a/content/understand-django/2022-05-31-debugging-tips-techniques.pt.md b/content/understand-django/2022-05-31-debugging-tips-techniques.pt.md
new file mode 100644
index 0000000..d981f5c
--- /dev/null
+++ b/content/understand-django/2022-05-31-debugging-tips-techniques.pt.md
@@ -0,0 +1,307 @@
+---
+title: "Debugging Tips And Techniques"
+description: >-
+ Your Django app is up. You've got users. Your users are hitting bugs. How do you debug to fix the problems? That's the focus of this Understand Django article.
+image: img/django.png
+type: post
+categories:
+ - Python
+ - Django
+tags:
+ - Python
+ - Django
+ - debugging
+series: "Understand Django"
+
+---
+
+In the last
+{{< web >}}
+[Understand Django]({{< ref "/understand-django/_index.pt.md" >}}) article,
+{{< /web >}}
+{{< book >}}
+chapter,
+{{< /book >}}
+we looked at security. How does a Django site stay safe on the big, bad internet?
+{{< web >}}
+The article explored some core elements
+{{< /web >}}
+{{< book >}}
+The chapter explored some core elements
+{{< /book >}}
+for making a Django app more secure.
+
+{{< web >}}
+With this article,
+{{< /web >}}
+{{< book >}}
+With this chapter,
+{{< /book >}}
+we will investigate problem solving techniques for Django apps. The goal is to equip you with tools to fix the real problems that you'll hit when building your Django site.
+
+{{< understand-django-series-pt "debugging" >}}
+
+## Systematic Problem Solving
+
+When you have real users using your website and they report problems with the site, what can you do to fix the problems? Having a mental framework for how to fix problems is really useful, because it provides a repeatable process that you'll be able to use as long as you're building on the web. Let's discuss the mental framework that I use, which I think of as *systematic problem solving*.
+
+The first thing to drill into your head is that computers are deterministic. Given a set of inputs, they will produce a repeatable output. This is crucial to remember because the art of problem solving with computers is about sleuthing the state of your system (i.e., a Django website in this context) to determine why a given set of inputs produced an undesirable output, also known as a *bug*.
+
+When debugging a problem, the challenge is figuring out what all those inputs are. The inputs can vary wildly. Successful problem solvers are those who can find the inputs that caused the bad output.
+
+This is the core of systematic problem solving. With systematic problem solving, you want repeatable patterns that help you build up a mental model of how a problem occurred. Once you understand the problem, you'll be ready to create the solution and fix your bug.
+
+*How do you repeatably figure out problems with a website?*
+
+In my experience, the most fruitful way to understand a problem is to reproduce it. To reproduce a problem requires context and data about what happened in the first place.
+
+What are the sources of this data?
+
+{{< web >}}
+* Error monitoring services like Rollbar or Sentry can be a fantastic source of data.
+* Log data, which we'll discuss later in this article, can be another excellent source.
+{{< /web >}}
+{{< book >}}
+* Error monitoring services like Rollbar or Sentry can be a fantastic source of data.
+* Log data, which we'll discuss later in this chapter, can be another excellent source.
+{{< /book >}}
+
+Error monitoring services help show what happened. These services will collect tracebacks of Python exceptions and display them in context of a particular request. Some of these tools can even show the sequence of events that led to the failure case.
+
+My favorite strategy for cases like this is to produce an automated test that can trigger the same exception as the one reported by the error monitoring service. This is often a mechanical transformation of the data provided on an error report page into the setup data of a unit test.
+
+The great part about having a unit test is that you now have something that you can repeat with ease. This is a lot better than clicking around on your site and trying to recreate a problem manually. Because you've captured the problem scenario in test code, you can know *exactly* when you've **fixed** the problem. You can work on the site's code and run the test over and over until you've devised a solution that overcomes your error.
+
+This process is a systematic approach that you can apply again and again to solve problems. The process in a nutshell looks like:
+
+* Collect data from the failure scenario
+* Transform that data into an automated test that demonstrates the problem in a repeatable fashion.
+* Fix your site's code until the test passes and the problem is resolved.
+
+The secondary benefit of this systematic model is that your site has a passing unit test when you're done. This test acts as a guard against future failures of a similar kind (i.e., a "regression" test).
+
+I don't want to oversell the value of these kinds of tests. Regression tests can be useful, but in my experience, I've rarely seen regression tests fail after the offending code is fixed. Nonetheless, if the test is fast enough, these kinds of tests are worth keeping in your automated test suite for those rare cases where you *do* encounter a regression failure.
+
+Maybe my unit testing approach doesn't appeal to you. That's fine. Do what works well for you. My larger point is that you should try to take a systematic approach to problem solving. **Find repeatable patterns that you can apply to your context.**
+
+Approaching problems with a process in hand helps you avoid feeling stuck. Having methods for solving problems also helps when the pressure is on, and you're trying to fix a critical and time sensitive bug for your customers. Not knowing what to do in a high pressure scenario can add even more stress to an already stressful situation.
+
+## `print` Without Shame
+
+In the previous section, I wrote "Fix your site's code until the test passes," but I didn't explain how you'd actually do that.
+
+When you're working with a failing test and you're trying to make it pass to solve a problem, you have to understand the scenario at a pretty deep level.
+
+One tool to reach for is the `print` function. You can sprinkle some calls to `print` into the code you're working with to gain an understanding of what is happening.
+
+In your journey in programming, you'll run across this idea that "print debugging" is a bad idea. Not everyone espouses this idea, but it exists out there. In the same breath, you'll read that you should be using a debugger instead or some other tool.
+
+Debuggers, which are great tools that we'll cover shortly, are not the only tool worth knowing. In fact, in many cases, debuggers are a terrible tool for the job.
+
+What does the `print` function do that makes it so useful? **`print` provides a chronology.**
+
+With the `print` function, you can observe changes *over time*. The `print` function can answer questions like:
+
+* What is the value of my variable in each iteration through the loop?
+* Is this block of code even reached by the interpreter?
+* What is the before and after state after a section of code executes?
+
+The nice part about using `print` in an automated test is that you can see all the data, all at once. The results of a handful of `print` calls starts to tell a story of what happened during execution. If your "story" isn't rich enough and meaningful enough to understand what's happening, you add more `print` calls and run the test again.
+
+I'd estimate that I am able to use `print` for 70-80% of my debugging needs.
+
+What should you print?
+
+* One simple strategy is to print a number sequence. Adding `print(1)`, `print(2)`, or more to various lines can show what parts of code are executing or what patterns are happening in loops.
+* Sometimes I'll print the values of conditionals for branches that I'm interested in. This helps me figure out when certain branches in the code are taken versus when they are not.
+* Other times I'll add prints with a bunch of newlines to visually separate things. `print('\n\nHERE\n\n')` is surprisingly effective.
+
+The answer to "what should you print?" is really "what do you want to know?" As you use `print`, you'll develop a good feel for what information you find most useful.
+
+Tip: As of Python 3.8, you can print the variable name in an f-string along with its value:
+
+```python
+>>> foo = 5
+>>> print(f'Hello world {foo=}')
+Hello world foo=5
+```
+
+`print` is pretty great, but it isn't always my tool of choice. For instance, `print` is not the best tool when problem solving a live site. `print` also doesn't work as well when I need to get really detailed information about the state of my program.
+
+When do I reach for something else? Let's look at debuggers next.
+
+## Debuggers
+
+If 70-80% of the time I can get away with `print`, what about the rest of the time?
+
+Sometimes I'll use a debugger. A debugger is a specialized tool that allows you to go through your Python code, one line at a time.
+
+Considering how much code we write and how much we use from other packages, this means that a debugger can be a slow tool to utilize. A debugger makes up for this slowness with an unparalleled amount of information about what is going on at the exact moment that the Python interpreter executes a line of code.
+
+How do you use a debugger?
+
+Assuming that you're following my strategy of writing an automated test when you're working on a problem, the simplest way to start a debugger is by adding `breakpoint()` before the code you want to check.
+
+In a standard Python installation, adding this function call will pause your program by running the Python debugger, `pdb`, starting from the call to `breakpoint`.
+
+If you do this, you'll be left at a prompt that starts with `(Pdb)`. From here, you'll need to know some commands to navigate within the debugger. I'll cover the primary commands I use here, but you can type `h` to see a list of the available commands with instructions for how to get more help info.
+
+My natural inclination in the debugger is to know where I am. The `l` command will *list* code that the interpreter is about to execute, along with an arrow showing the next line that Python will run. I may also want to know where I am in the call stack (i.e., the history of calls that go back all the way to the main function that started the Python process). By using the `w` command to show *where* I am, pdb will show the call stack with the current line of code listed last by the prompt and the oldest function listed at the top of the output. These two commands give me the context at any particular moment.
+
+Next, I'll often want to know the values of local variables when I'm debugging. I can either use `p` to *print* the value or `pp` to *pretty print* in cases when I have a structure like a list or dictionary.
+
+All the previous commands orient me to where my code is and what values are in the data structures. With that context, I'm ready to work through my code with two additional commands. In a debugger, you advance the Python interpreter a line at a time. There are two styles to do this.
+
+One way is with the `n` command to go to the *next* line. This command is what you want when you don't really care about how a line runs. For instance, if the line calls a function that you know works, `n` is the way to pass that line to let it execute in its entirety.
+
+The other command to advance the interpreter is the `s` command. The `s` command lets you *step* through the code at the smallest possible increment. That means if you're on a line with a function call, the `s` command will move *into* the function call to show what's executing inside of it. I view the step command as the very fine adjustment command to run as I get close to the problem area in my code.
+
+Whenever I'm done debugging my code and have seen all that I need to see, I finish with the `c` command to *continue* normal execution.
+
+What's great about having a unit test is that I can run the debugger in a very coarse way by liberally using the `n` command. As I hit the error state (like an exception happening), I can continue, restart the test, then quickly return to right before the point in time when the error occurs. At that juncture, I switch to the `s` command until I find the exact moment when something goes wrong. This process helps me avoid wasting time by stepping through parts of the code that don't matter.
+
+The debugger has more interesting features like setting breakpoints and checking data at different parts of the call stack, but I hope this description gives you an idea of how to apply a debugger in your workflow.
+
+Learn more about the debugger and `pdb` by checking out the {{< extlink "https://docs.python.org/3/library/pdb.html" "pdb standard libary documentation" >}}.
+
+## Use The Source, Luke
+
+At times, you're going to run into problems with code that is not part of your project. This might be code in the Django source or it might be code from some other library that you happen to use in your project.
+
+What I have found over time is that many less experienced developers will hit this situation and suddenly freeze. There's some kind of mental barrier preventing them from looking any further than the boundary of their own code.
+
+Here's my tip: **don't freeze!**
+
+When you wake up to the reality that it's all just code that people wrote, you cross that mental barrier and move into realms that others may fear to tread. The funny part is that this code is often no more special than anything that you might write yourself! In fact, many open source developers are fantastic programmers, and fantastic programmers often know how to write clear, maintainable, and well-documented code.
+
+So, listen to your inner "Obi-Wan" and use the *source*, Luke!
+
+* If you're hitting a problem with a package, look at the source code on GitHub (or wherever the code is hosted). You can study the modules and trace through how the code would execute.
+* If that's not enough, don't be afraid to step through functions in other libraries with your debugger as you are testing out whatever problem that you are trying to fix.
+* Remember `print` debugging? Who says you can't do that with other software? If I've got a virtual environment and I need to figure out what's going on with how my software interacts with a library, I will sometimes add `print` statements directly to the library files in my virtual environment's `site-packages` directory. This is a great trick!
+
+There are limitations to this approach because some packages will include code sections that are compiled with C. That kind of code is harder to peek into, but I've found that the majority of Python packages that you find on PyPI are Python-only. That makes most packages great candidates for digging into the source code.
+
+## Logging Chronology
+
+Logging is a topic that we will only scratch the surface of, but the subject is important to round out a discussion of debugging tools.
+
+A logging system provides the ability to record messages during the execution of your application. The easiest way to think of logging is as a permanent set of `print` calls that record information every time your functions or methods call the logging system.
+
+When logging is used in an application, you can capture what happened within your app even if you weren't there to observe the system at the moment that it happened. The logging system forms a chronological log of events that happen within a system.
+
+This has some big benefits, but comes with some tradeoffs to consider.
+
+* **Pro**: When you have sufficient logging, you can see what is happening in the app when your users are working in the app.
+* **Pro**: More advanced logging configurations can include metadata that you can filter on to look for specific kinds of activity. This can aid your diagnostic abilities.
+* **Con**: Logging generates data. While this enables the points above, you now have a new challenge of *managing* this new source of data.
+* **Con**: Log management is even more of a challenge as an application system grows. When you have multiple servers, where does all that log data go?
+
+This concept of logging is not a Django-specific idea. In fact, you can find a `logging` module in the Python standard library that serves as the basis for logging in the Python world. Django builds its logging features on top of the `logging` module.
+
+The short version of logging in Django is that you can use logging by configuring the `LOGGING` settings in your Django settings module. This process is described in the documentation at {{< extlink "https://docs.djangoproject.com/en/4.1/howto/logging/" "How to configure and use logging" >}}.
+
+In a basic workflow, logging can be little more than writing lines of data to a log *file*. If you're running an app on a VPS like Digital Ocean and have control over your filesystem, this is a good starting option. If you choose that path, keep in mind that disk space is a limited resource. One way to tank your app is to fill your machine's disk with logging data. To prevent this, you should investigate how to use logrotate. {{< extlink "https://linux.die.net/man/8/logrotate" "logrotate" >}} is a command that can archive your log data on whatever frequency you desire and clean up old data to prevent your storage from filling completely.
+
+You may follow a path like I've recommended frequently and deploy your app on a Platform as a Service (PaaS) like Heroku. In that kind of environment, you have less control over the tools that are available to run your application. In the case of Heroku, the platform does not guarantee that your app will stay on the same machine. Because of that constraint, you can't rely on a log file on the machine. Instead, you would need to add an additional service to the application that will store logs for your later use. {{< extlink "https://www.papertrail.com/" "Papertrail" >}} is an example of a log aggregation service that works with Heroku.
+
+As you can see, logging brings some complexity to manage if you choose to use it. Logs can be a great tool to understand what happened when a user used your app, especially if there was an error that didn't raise an exception. For me, personally, I *don't* reach for logging right away. While logging can be useful, it can be an overwhelming amount of data that can turn debugging into a "needle in a haystack" problem. Logging is still a potentially useful tool for your toolbox as you think about how to debug your apps.
+
+## Summary
+
+{{< web >}}
+In this article,
+{{< /web >}}
+{{< book >}}
+In this chapter,
+{{< /book >}}
+we saw debugging tips, tools, and techniques to make you a bug crushing machine in Django.
+
+We discussed:
+
+* A systematic method for finding and fixing bugs in your app
+* How `print` is an awesome debugging tool that should be used without shame
+* Debugging with a "proper" debugging tool for particularly thorny problems
+* Using the source code of packages you didn't write to figure out what's going on
+* Logging and the ability to see a history of activity in your application
+
+{{< web >}}
+That's the end of this series!
+{{< /web >}}
+{{< book >}}
+That's the end of this book!
+{{< /book >}}
+Can you believe there are still more topics
+{{< web >}}
+that this series could cover?
+{{< /web >}}
+{{< book >}}
+that this book could cover?
+{{< /book >}}
+Django has so much available that,
+{{< web >}}
+even after twenty articles,
+{{< /web >}}
+{{< book >}}
+even after all these chapters,
+{{< /book >}}
+I've not covered everything. But this is the end of the line for me.
+
+{{< web >}}
+I knew when I started this series
+{{< /web >}}
+{{< book >}}
+I knew when I started this book
+{{< /book >}}
+that I was going to cover a huge number of topics. What I didn't know when I started this in January of 2020 is how wacky the world would be.
+{{< web >}}
+I thought that I might produce an article every other week
+{{< /web >}}
+{{< book >}}
+I thought that I might produce a chapter every other week
+{{< /book >}}
+and be done in less than a year. *How wrong I was!* More than two years later, I'm writing the words
+{{< web >}}
+of this last article.
+{{< /web >}}
+{{< book >}}
+of this last chapter.
+{{< /book >}}
+
+{{< web >}}
+As I wrap up this series,
+{{< /web >}}
+{{< book >}}
+As I wrap up this book,
+{{< /book >}}
+my hope is that you, dear reader, have come to understand Django better. Django is a web framework written by people like you and me. That community, through years of collaboration, polished a tool that can bring your ideas to life on the web.
+
+While the framework might initially seem magical in all that it is able to do, we can see through study and examination that the code is comprehensible. The magic becomes less magical, but we can grow in our respect for those who contributed untold hours to making something so useful for others (for *free* in most cases!).
+
+{{< web >}}
+I would like to conclude this series
+{{< /web >}}
+{{< book >}}
+I would like to conclude this book
+{{< /book >}}
+by thanking all of you readers out there. Along this lengthy journey, so many of you have reached out
+{{< web >}}
+and told me how this series has helped you grow
+{{< /web >}}
+{{< book >}}
+and told me how this book has helped you grow
+{{< /book >}}
+as a Django developer. I'm hopeful that developers have used these words to learn and create websites that help their own communities. Knowing that I've impacted some of you and, by extension, the communities that you've helped,
+{{< web >}}
+made writing this series worth it.
+{{< /web >}}
+{{< book >}}
+made writing this book worth it.
+{{< /book >}}
+
+> Thank you for reading!
+
+{{< web >}}
+If you'd like to see more content like this series, please feel free to sign up for my newsletter where I publish all new things first. If you have other questions, you can reach me online on Twitter where I am {{< extlink "https://twitter.com/mblayman" "@mblayman" >}}.
+{{< /web >}}
+