Runtime caching?

[kalafut]

Hi. I'm fairly new to this project, but my first use is going smoothly, and I'm really impressed!

As I've gone over the docs and peeked into the code, one topic I've not seen much of is caching. Watch mode involves some caching, but I was thinking about an in-memory cache for templates/components.

If a component is a pure function that only references other pure functions/components, and its inputs are value types (not pointers), then a cache of {component, args} → HTML should be possible. This could be beneficial with a highly composed app. For example, a styled Button might used in many places but only with a few labels. A form might only have a few different sets of fields, etc.

Knowing what components can be safely cached sounds non-trivial for the general case, since a template can call any function. Instead, this could be the developer's responsibility, e.g. by marking a template as const to indicate it is cacheable. The compiler+runtime could, over time, get better at detecting and warning if a const template can't be safely cached.

I was going to start playing around with this concept, but I'm already overcommitted and probably won't get to it for a while. I'm curious if this idea has been discussed before? I'm not even sure it is worth it. But if templates are calling deterministic but slow functions, maybe it is.

[a-h]

Yes, @joerdav and I have discussed introducing a cache component for partial output caching. The main issue is, as you might guess, cache invalidation - one of the two hard problems of computer science. 😁

Here's a working example of a partial output cache in templ:

package main

import (
	"bytes"
	"context"
	"io"
	"log/slog"
	"os"
	"time"
)

type cacheContent struct {
	expireAt time.Time
	content  []byte
}

var cacheKeyToContent = make(map[string]cacheContent)

type cacheComponent struct {
	cacheDuration time.Duration
	key           string
}

func (c cacheComponent) Render(ctx context.Context, w io.Writer) error {
	// Get children.
	children := templ.GetChildren(ctx)
	ctx = templ.ClearChildren(ctx)
	if children == nil {
		return nil
	}
	cc, isCached := cacheKeyToContent[c.key]
	if isCached && cc.expireAt.After(time.Now()) {
		_, err := w.Write(cc.content)
		return err
	}
	// Render children to a buffer.
	var buf bytes.Buffer
	err := children.Render(ctx, &buf)
	if err != nil {
		return err
	}
	// Cache the result.
	cacheKeyToContent[c.key] = cacheContent{
		expireAt: time.Now().Add(c.cacheDuration),
		content:  buf.Bytes(),
	}
	// Write the result to the output.
	_, err = w.Write(buf.Bytes())
	return err
}

func Cache(key string, duration time.Duration) templ.Component {
	return cacheComponent{
		cacheDuration: duration,
		key:           key,
	}
}

func slowDependency() (string, error) {
	time.Sleep(5 * time.Second)
	return "I took a while", nil
}

templ SlowComponent() {
	<div>{ slowDependency() }</div>
}

templ DemonstrateCache() {
	@Cache("slowCacheKey", 10*time.Second) {
		@SlowComponent()
	}
}

func main() {
	log := slog.New(slog.NewTextHandler(os.Stderr, nil))
	// Render the template.
	ctx := context.Background()

	log.Info("Rendering template, this should take 5 seconds")
	start := time.Now()
	err := DemonstrateCache().Render(ctx, os.Stdout)
	if err != nil {
		log.Error("failed to render template", slog.Any("error", err))
		os.Exit(1)
	}
	log.Info("Render complete", slog.Duration("time-taken", time.Since(start)))
	log.Info("Rendering template again, this should be instant")
	start = time.Now()
	err = DemonstrateCache().Render(ctx, os.Stdout)
	log.Info("Render complete", slog.Duration("time-taken", time.Since(start)))
}

The result of this is:

time=2024-09-08T10:50:09.918+01:00 level=INFO msg="Rendering template, this should take 5 seconds"
<div>I took a while</div>time=2024-09-08T10:50:14.920+01:00 level=INFO msg="Render complete" time-taken=5.001465792s
time=2024-09-08T10:50:14.920+01:00 level=INFO msg="Rendering template again, this should be instant"
<div>I took a while</div>time=2024-09-08T10:50:14.920+01:00 level=INFO msg="Render complete" time-taken=37.917µs

Use of a cache is a tradeoff between process RAM consumption, and the cost of rendering again. Unless you're doing a network fetch in the component, then the cost of rendering a templ component is very low.

You could argue, therefore, that a better pattern would be to cache the data, before attempting to render components, as per the https://templ.guide/core-concepts/view-models/ concept, rather than making template rendering more complex.

re: your idea of computing whether something can be cached, constants in Go aren't really very useful, and AFAIK, you can't analyse whether something is a pure function (sadly): https://go.dev/play/p/aiYFPU8URBQ so the burden of deciding whether to cache the output of a template rendering operation falls to the programmer, hence the API above having a cache key parameter.

Other considerations would be out-of-process caching in Redis or something else, where the tradeoff is then the cost of pulling the data back over the network from the cache vs rendering it again.

I think a partial output cache could be useful even if the output of a template render isn't deterministic. For example, if I'm displaying available stock in a shop, the value might be unlikely to change on every page load, so I might want to avoid hitting a backend DB by caching the result.

For entire page level output caching, then I think most people would/should/could use a CDN, or some more general purpose HTTP middleware.

As you can see, there's no need per se for this to be in the core of templ until we work out the right API. We've got https://templ.guide/experimental/overview packages which might be the best place to start such a thing, and see whether anyone actually values it.

[kalafut]

I think something experimental is very reasonable, and your proposal is a good start to test interest. This approach allows one to easily bake parameters into their cache key too, if they want, through a simple fmt.Sprintf.

A couple of comments on the caching itself:

• I agree this should be a small, in-process cache like the one above. Redis, etc., seems overkill, and whole-response caching is well covered by frameworks, CDN, etc.
• Expiration is a good field (ideally optional, though, or perhaps 0 for never expiring).
• The cache structure could be LRU (or some eviction scheme), with a cap on items or maybe better, total bytes (which we readily know in templ's case). So I could say at a high level "cache up to 1MB of templates," and then there would be no surprises. The surprises I'm thinking about would be something like you've included some dynamic data in a cache key that accidentally results in thousands or millions of entries.

[kalafut]

💯 on mutexes and that's in the POC I'm drafting.

re: scoping, I think your idea is good, and isn't necessarily mutually exclusive with a default/global cache. I'll play around and share some concepts in the PR.

[kalafut]

Just pushed templ-go/x#3 for an initial review.

[a-h]

Brilliant. I've had a quick look and it looks great. I'll have a closer look when I can.

I already have a use case for this myself! I just built a markdown to html component that loads markdown from disk, so caching the result is something I want.

The tiny default cache size and short default cache time caught my eye though.

I thought a default of something like 64MB instead of 64KB might be reasonable. I think defaults are there to stop you making big mistakes, most containers and Lambda functions are going to have 1GB of RAM, so we can afford more.

Maybe worth looking at some other tools to see what their defaults are, and what people set.

[kalafut]

Yeah, it's too small. 64MB sounds ok, though I probably wouldn't go higher. Given that the items are lazily expired, if the dev accidentally picks a key that will be variable per request, the cache will fill pretty easily. But 64MB is ok even on a Raspberry Pi so that's a reasonable pick.

[kalafut]

I pushed the change to 64MB. I also changed the default TTL to 5 minutes after thinking about all of this.

We want this to be useful out of the box. As one data point, when I look at much of the content I consume, reading a page in 1 minute would be a bit quick, but I'm rarely on the same page for 5 minutes. (I'm basing this on articles and news that often list reading time.) If we default to too short, I think the OOTB improvements will be undercut for the normal case. Things will be expiring too soon without good reason. Even longer than 5 minutes would be OK by me, in fact.

The flip side is, of course, what about invalidation issues. My thoughts are that as soon as you start caching dynamic data, you're already playing in a fraught area and you should be deliberate about choosing your TTLs based on your requirements. My preference would be to give better default performance and expect user to handle shorter TTLs as needed. Some precedence here is Redis. The basic SET command doesn't have any TTL at all, and if expiration is a concern you use SET ... EX.