Advanced Render Cache Debugging

[00:00:00] Jody Hamilton: Okay, so I'm Jody, this is Janez, and we're here to go deep with the render cache. Hopefully everyone knows a little bit about the cache API, because we're not going to go too much into the basics; we'll try to get more into the weeds of it. We've been going pretty deep into Drupal caching and found out there's a lot to know to really improve your caching. Here's a link to our slides if you want to follow along.

[00:00:59] So my name is Jody. I've been around the Drupal community for a long time; I've been going to DrupalCon since 2008. For a long time I was CTO of a Drupal agency called Zivtech. The past few years I've been working for a Japanese semiconductor company called Renesas. I always had an agency, and I wished I could get really deep into a project long term, so now I run one website all the time, renesas.com, a big semiconductor company. I'm leading a team of 25 or 30 engineers there, and I'm also on the product team, so this performance stuff is more of a passion project than my actual job.

[00:01:53] I really love digging into complex performance problems, and I really think fast websites are, well, a good website is a fast website. It is the feature; it is what good software is. If it's slow, it's not good. It's not easy, but it's what we need to work on.

[00:02:21] Janez Urevc: I come from Tag1 Consulting. We are the second biggest contributor to Drupal. We also donate a full-time infra expert to the Drupal Association to help them run the infrastructure, drupal.org and all the tooling. In the past I used to be the lead of the Media Initiative, and I also worked at examiner.com, which used to be the biggest Drupal website on the internet, where we had quite a lot of interesting performance challenges. Even before that, performance has always been dear to my heart, so here I am.

[00:03:15] Jody: The reason we're doing the talk together is that when I came to Renesas, the site was really slow, like 10 seconds to load a page, nothing was cached on any level, and I was like, this is going to change, this is totally unacceptable. I'm not going to sit here and work on a site that takes 10 seconds; we're going to fix this whether it's my job or not. As I got into this, I realized I really needed some serious professional help, some experts. Our dev team at the time didn't really know a lot about Drupal caching or performance, so I remembered that the people I needed to call were Tag1. So we got a hold of Tag1 and got to work with Janez and his team, who really helped me learn more about the cache API, which I thought I understood well. I think most Drupal developers think they grasp it, but it's pretty complex, and there was a lot we dug into and found out.

[00:04:28] The reason renesas.com has such serious cache challenges: first of all it wasn't built with performance in mind, so of course it's so much harder when you have a site that's been around since, I think it started in Drupal 6, under active development with dozens of developers for a decade, maybe adding code without understanding, measuring, or thinking about performance. So we have a huge amount of technical debt to dig out of. But we also have other unique challenges. For example, we have tens of thousands of pages; we're basically an e-commerce site with tens of thousands of unique product variations. We also have three different languages on the site and nine different regional variations, so we have 27 URLs for every single page, and of course a lot of caching happens on the page URL level.

[00:05:20] We have a lot of content editors and a lot of automated integrations updating content constantly that needs to be invalidated all the time. And we support authenticated users; we have users who log into the site to access secure content, so we have a lot of people logged in who need it to be fast. We can't just make the site fast for anonymous users, because our customer workflow is, we're going to do everything we can to convince you to make an account on our site, that's how we get a conversion, and as soon as you do, it takes 15 seconds to load a page. So that's your reward.

[00:06:16] We also have probably almost a hundred custom modules, endless custom code, hundreds of contributed modules, and we had very heavy use of the views module, building everything all around the site; every page was a view of a view of a view. We also have lots of pages with hundreds and sometimes thousands of entity references on them, really big complex pages that could cause a lot of cache invalidations or just be slow if uncached. So we had serious challenges. It's not drupal's fault; there is no system out there where you can have authenticated users with per-user experiences and dynamic content that's always up to date, and tens of thousands of URL variations, and have this actually cached well and be fast without tons and tons of custom development.

[00:07:41] Janez: We'll first go through some of the render cache basics, explain the concepts and how it works, then some more advanced techniques used in Drupal, like placeholdering and the layers of caches we can experience in Drupal, and then we'll go into debugging and getting stats out of the cache system, and things that can go wrong.

[00:08:19] But first, the basics. What is render cache? Render cache was introduced in Drupal 8. Before, it could have been a contributed module that did it, but in uniform, as we have it now, it came into Drupal in version 8. It caches markup that was rendered, so next time you need this specific piece of markup, instead of re-rendering it again you can get it from your cache, and it's way faster. It is enabled by default, so if you're producing your own markup, it will be cached, and if you don't think about that, you'll get potentially very weird results.

[00:09:22] By default everything is cached indefinitely; it doesn't have any lifetime, and it uses tags to invalidate. It does significantly improve performance, and it's so powerful that, as Jody mentioned, it lets you cache things dynamically for authenticated users. It's not like page cache that only works for anonymous users. But it can cause really strange and weird problems if it's not used correctly, and in order to use it correctly we have to understand some basic concepts behind it, which are really not that hard to understand once you grasp them.

[00:10:18] The first concept is tags. Tags are used to invalidate things that are cached. We mentioned that by default everything is cached indefinitely, so if we didn't have tags, when we render the page it would never change, and cache tags let us achieve that. They look like this. The first one essentially means that when node with ID 11 updates, everything that has this cache tag on it will also invalidate. The second one is similar but for user with ID 4. Then we can have tags that are invalidated when a certain config is updated, like the one for views here, or tags that are way more broad, like "rendered," which invalidates everything being rendered, basically the entire page. You can also create your own custom cache tags to fine-grain the invalidation logic you want. When you're building anything that produces markup, usually a block plugin, you are responsible for defining cache tags on that markup, otherwise it won't be cached correctly.

[00:11:58] The second concept in the render cache is contexts. Contexts are the thing that defines how a markup varies. If we have markup that's the same for every user, then it won't vary, but if we have markup that's different for different situations, we have to use contexts, and contexts tell the caching system how to vary. A few examples: route, so this markup will be cached separately for every route; if it appears on route A it will be different than on route B. Then session will be different for every user, because every user has a different session.

[00:12:58] Then we can vary by URL, and this one is interesting, because you can use the URL cache context, which means the item will be cached separately for every URL, or you can control it more in detail. You can say I don't want it different for every URL, but just for some part of the URL, in this case just the query arguments, and even more, you can limit it to a single query argument, which then means you get a higher cache hit ratio because you're not varying as much. There is a huge difference between varying just by the page query argument and varying by every URL. So it's wise to define this cache context as detailed as possible. Again, when you write custom markup, you are responsible for defining the context on that markup.

[00:14:09] There are also default cache contexts configured in the services.yml file, and those contexts will always be used. The defaults are language, theme, and user permissions, and if you think about it, it makes a lot of sense. If you have a site where somebody can switch a theme, you probably always want to cache markup in those two themes separately, and language, because you don't want to cache something in Spanish the same way as something in English, that wouldn't work.

[00:14:50] Then at the end we have the max age attribute, which basically defines how long something will be cached. It's an integer, the number of seconds something will be stored in the cache system. Default is -1, which means forever, and that is usually the best approach, because we want to cache as long as we can and use tags to invalidate, not time-based invalidation.

[00:15:29] Now we come to metadata bubbling. If you imagine a page as a tree structure, let's say on a page you're displaying an article, and inside that article you have an image, and the image itself is a component that has the actual image and the caption. Imagine this as a tree structure, and at the root or the top of the tree is the entire page, and the leaves are the smallest individual components, like the image tag. When cache metadata is configured on the leaf, it will always bubble up towards the top, so all the cache metadata that every leaf defines will end up being assigned to the top, so the entire page will inherit all the cache metadata from all these small components.

[00:16:52] When things are bubbling up, tags and contexts are merged. So we saw node 11 before, and then another component on the page adds node 22; this means the page will have both cache tags at the end, and same with contexts. Which also means that if you put some problematic context or tag on a really small, maybe not significant part of the page, the entire page will be affected. And max age, when bubbling, uses the shortest age of all, so if you say max age zero, effectively making something uncacheable, the entire page becomes uncacheable. There is a concept of placeholdering that can stop this bubbling, and we'll cover placeholdering a little later.

[00:17:59] We already talked a little about the consequences. This is very common: we have a block or some piece of the page that behaves in a weird way, and a lot of the time people will go and put max age zero on it to solve the problem, quote unquote, but then it can happen that the entire page becomes uncacheable, which is obviously not what we want. And as much as max age zero is a problem, using cache contexts that vary a lot, like vary per user, is also causing problems, because it makes things harder for Drupal to cache.

[00:18:56] Render cache is not the only caching layer we have. We have internal page cache, which has been part of Drupal for a very long time, but that one works only for anonymous users, which also means it doesn't need cache context, because you just cache one version of the markup. Then we have dynamic page cache, which works also for authenticated users and uses this cache metadata to try to be smart, to have dynamic content but still be able to cache it as efficiently as possible. Dynamic page cache also works with Big Pipe, which can stream placeholder elements later in the HTTP response.

[00:19:50] Then we also have external page caches like Varnish or CDNs like Akamai, and using the Purge module you can use cache metadata to also invalidate items in those caches. So if you've set your tags correctly, you can send the tags via a header to a CDN, and then when you want to invalidate that, you do an API call to the CDN and instruct it to invalidate, which is also very powerful, but it requires that you provide correct cache metadata.

[00:20:39] So, the most common problems we see. This one we already mentioned: usually this is done because there are cache-related bugs, and the solution is to make something uncacheable. Please don't do it ever, because there are really almost practically never good reasons to do that, and if you still think you have to, it can pay off to still cache for a short period, maybe a few seconds or a minute, and then you have to placeholder it.

[00:21:23] Another really common problem is missing cache tags. For example, we have a block that prints a list of five most recent articles, and you forget to add cache tags for those five articles to the output of the block. This means that if one of those five articles updates the title, this title won't update in the block. You solve that by adding cache tags for all five nodes you displayed onto that markup you're generating.

[00:22:12] If you don't do that, and you forget some cache tags, then content won't update, and admins and editors, in order to make the change appear, usually need to go and clear the caches on the site, because we have a very nice button in the UI that clears all the caches on the entire site, which we don't want. The problem is when people get into the habit of doing that, every time they see something not updating, let's clear the cache, and that's a really bad thing because it affects performance a lot, and when people develop a habit it's really hard to stop.

[00:22:59] Then we also have cache tags that are too general, like node list, user list; those will invalidate anytime any node in this example updates, and views, by the way, adds this cache tag by default. It's way better to use cache tags that are more specific, like the one that only invalidates when a node of content type article is updated. A lot of times people will just invalidate cache tags just in case, which also results in frequent invalidations that affect your performance, so don't do it; do it when you know why you're doing it and what you want to achieve.

[00:23:54] Contexts: usually the problem is that there are missing contexts, so things won't vary the way we want, which means potentially we'll display content that was meant for another user to yet another user. So we really have to think about which contexts we need, because if we put too many on a markup, you'll get a poor cache hit ratio. A common example is an empty block: you have a block that displays something, but under certain conditions it doesn't print anything. Usually what people would do is, when those conditions are met, return the empty render array. But you still have to put cache metadata even on that empty render array, because you don't want it cached indefinitely and not varying at all. Sometimes that's not needed, but in a lot of cases you have to do it, so good practice is to prepare the render array, start putting basic metadata on it, then have the condition and add the markup to the existing render array.

[00:25:31] Which brings us to placeholdering. Placeholdering is a tool that lets us cache pages that have items that vary a lot or are frequently invalidated. It works by, instead of including the problematic part of the page in the cached item for the entire page, it will only put a placeholder there and then render it separately, and then inject the problematic part into the markup of the entire page after the fact. This lets us cache the entire page while still having pieces that are really dynamic and change a lot.

[00:26:34] Drupal will do this automatically for you and try to be smart. For example, the core blocks use that by default, so if you create a block that varies per user, it will be automatically placeholdered; you don't need to do anything, which is very powerful. But then we found out that if you're using the Context module, Context module doesn't do that, and if you have a block that varies per user and you're using Context, then suddenly the entire page will be varied by user, which really impacts performance. There is an issue; it has been open for six years and has a patch, but it's still not committed, so if you're using Context, check that issue and use the patch.

[00:27:29] An example of placeholdering is the status messages. Status messages are meant just for the user currently viewing the site, just this one time, so it would effectively make the entire page uncacheable if you didn't placeholder them. If we check how status messages are rendered, it looks like this: it doesn't load status messages or try to display them, it only defines the lazy builder callback and says please placeholder it. Then Drupal renders the entire page, and after rendering the entire page it calls the lazy builder callback, which renders the status messages and injects that markup into the page. This is how the page can be cacheable and status messages that are only displayed once are rendered.

[00:28:40] On renesas.com we used this technique for this block, which looks quite static; it has a few links that are the same for every user, but then you have "subscribe to document updates," which, if you are subscribed, will say "unsubscribe from document updates." So only that link varies per user, because it has to check the subscription status, and we placeholder just that link to be able to cache the block for every user and then put that link into it to vary for every user.

[00:29:29] Jody: I'm going to give you some tricks and tips and types of stuff I wish I had known earlier, how to really dig into this stuff. First of all, you need to be able to debug these cache tags and cache contexts and see which tags and contexts are on everything, to improve them. The first place to start is in services.yml, which should be in sites/default, and that's where it has these render configurations that set what your default contexts are, your auto-placeholdering settings, the kinds of stuff Janez was just talking about.

[00:30:30] There's also something you can turn on that you would not do in production but would do on local or a preview environment: you can enable this debug cacheability headers. That lets you see, on the page level, after all the tags and contexts bubble up to the page level, you can look in your response headers and view all the tags and contexts for a certain page. From there you can start to dig in and see, this context is a problem. For example, I think that URL query args is a problematic cache context, because as soon as somebody puts a random string into the query, now they've got a cache miss.

[00:31:21] user.node_grants view is a problem, because if you're logged in and you have a specific grant to a node, now you're varying per user, which means you basically have no caching at all for your authenticated users. Then you look at the cache tags and try to see if any of these are problematic cache tags that start to get cleared, and I'll show you how you could figure that out in a bit. Down here this is really bad: this whole page is set to uncacheable. But that's a separate heading you would have by default anyway. So yes, you definitely want to enable that by default locally and just always be looking at it.

[00:32:11] There's another way to debug that in more detail, on the render array level instead of the page level, and this I think Janez actually helped this patch finally get landed. There's a setting also in services.yml called debug, so it's like renderer config debug, you set that to true, again for your local, and then it gives you, in your markup in the HTML right before each block or each rendered piece of the page, just for that little bit, what all the cache tags are, all the cache contexts, and it even shows you how long it took to render that block and whether it was a cache hit or a cache miss.

[00:33:02] I use that all the time, because you have a mix of the custom tags and contexts you're adding and all kinds of things core and contrib might be adding in there, and ultimately you just have to see what the end result is. It even shows you the pre-bubbling cache tags and contexts. So turn on that debug mode; that's where you start to get into this stuff. But that will show you what's going on at the development level. The thing about performance is it's always a combination of the code and the real user traffic, and you have to see what's really going on on your production site with the real traffic and patterns actually happening to understand what's really important with the caching.

[00:34:01] So Janez introduced me to this Cache Metrics module, which is by Moshe, who I think is here. Moshe made this module, Cache Metrics, that can send cache data into New Relic. I think it can probably send it to some other places if you're not using New Relic. This way you get all the data on production of every time anything invalidates a cache tag on production, and which URL it happened on, when, and by which user, and you can also get all the data for cache hits and misses for dynamic page cache, render page cache, everything, and see everything going on in production with the cache tags.

[00:34:46] That was a real game changer once we started investing in trying to get better data and really see what was going on, and making sure we were spending our time fixing the things that were actually going to help, because you could really spend your lifetime trying to fix these cache issues and maybe not make much of a difference. So you really need good data. This is a screenshot of a dashboard I got from using Cache Metrics, and you can see which cache tags are getting invalidated, and pages that are cache misses and all this kind of stuff.

[00:35:32] But it doesn't answer the whole question of which cache tags are really problematic, because it will tell you which cache tags are getting invalidated, and which cache tags you have on a certain page, but you can't really query it together. So if you look here, these cache tags are getting invalidated a lot, but are we even using these cache tags? So what I'll do is pick a certain URL that I think has a poor cache hit rate and a lot of traffic, and then I'll go grab what tags it has, coming from dynamic page cache. I copy those, and then I do another query from the tag invalidations to check what's invalidating the actual tags I'm using.

[00:36:43] So I copy the actual tags that are on that page, and in the past week this one got cleared 14 times, this one three, and the others not at all, so these really didn't matter versus these other ones where nothing was clearing them. This 14 isn't bad, but it's not good. Sometimes I've done it and it says 5,000 times, and you're like, oh great, the best cache tag, and then you realize actually this cache tag is on every single page on our site and it's getting cleared 5,000 times a week, so we really don't have any caching at all. That's how easy it is to have no caching, and that's why this stuff is so tricky, it's very easy to destroy your caching in so many amazing, easy ways, that not doing it is kind of a modern miracle.

[00:37:42] This is showing, for those same cache tags, what URLs are invalidating them, so you can understand, okay, this is the one getting invalidated fairly frequently, it's happening when this node was edited by a content editor, or this one was deleted, and you can start to understand, maybe we need to be more specific about when we're invalidating them or how we're using these tags. So, Cache Metrics module, really like that.

[00:38:15] Okay, let me talk about views for a little bit. First of all, if you're doing an enterprise site or a high performance site with a big team, I would just not use views. I really think that Drupal as a development community needs to get a bit more clear about what is a site builder tool and what is a serious engineering tool for big sites. Views is a great tool for small sites you're building quickly, or for things that don't get used a lot like an admin dashboard, but it's not a great tool for high performance or for doing a lot of custom development on top of, and there's really no reason to be using it; just write a database query and build your stuff.

[00:39:12] So here are some of the gotchas with views. When you go into views and see the caching options, there are three: tag-based, time-based, and none. I think the default is tag-based, so if you use tag-based, which is the default, it automatically adds this node list, well, usually views are mostly nodes, but whatever the entity type is, it adds that cache tag automatically, because it doesn't know which cache tag to add, so it just adds the one that's going to wipe everything out as much as possible, because it's not able to understand closely enough when it really needs to invalidate.

[00:39:52] Well, node list is just one of these toxic cache tags, and so adding that is kind of like having no caching at all, and if your content gets updated a lot, because node list gets invalidated every time you edit or add any node, that's not a cache tag you want on things. So then you might think, maybe I should switch it to time-based, and they have the option for time-based. Well, here's the messed-up thing about time-based: time-based adds max age but it doesn't remove the tags. You think it's going to stop invalidating things by tag because you switched it to time-based, but it doesn't, it still uses the same tags and then also adds a time basis, which is completely unnecessary because it's already getting invalidated all the time anyway. So time-based is worthless; there's an issue in core about it.

[00:40:56] Then the other option is none. If you set it to none, I think that's going to give a max age of zero. Hopefully that will cause auto-placeholdering to happen, so you can still get a dynamic page cache hit for the rest of the page, but it might result in you not even getting any caching on the entire page at all, so I wouldn't mess around with none. Also, if you set sorting to random, that's another good way to have no caching at all on a view.

[00:41:32] So if you want to use views and get rid of these tag problems, you have to install this contrib module, Views Custom Cache Tag, and when you do, that gives you a new option, so you'll have tag-based, time-based, or custom. When you use the custom option, first of all it'll remove that node list general entity list tag and allow you to put in the cache tags you actually want. So one way you can do it is put in a more specific one, like node list article, or put in some other custom cache tag.

[00:42:14] So we got into this idea of custom cache tags, which was really necessary for us. The reason was that we wanted to use cache tags to invalidate Varnish and our CDN, so that things would update right away when they needed to on those levels, and not require the editors to wait a certain number of hours or to manually invalidate it. What we found was, at least on Acquia, there's a size limit to how big of a header you can send to Varnish. The way varnish purging by cache tag works is it sends the cache tags in the response header, so if you have too many cache tags, you're going to get fatal errors all over your site.

[00:43:16] We had way too many cache tags, because what we were doing is we'd have a big table of 500 nodes and we'd have node one, node two, node three, 500 node cache tags on it, so if any of those nodes changed we'd change it. Actually we had some pages that had ten thousand in a table, these crazy pages with way too many cache tags, so we had to add some custom data to Cache Metrics so that in New Relic we could see the length of our cache tags, and then we'd know which pages we needed to work on, because otherwise we'd keep getting these fatal errors.

[00:44:04] So Howell on our team did a ton of work around this to get both the purging working and all this cache tag stuff working, so he'd have to go figure out which pages have too many cache tags and what to do about it. What we did was, instead of putting a separate cache tag for each node, we started making our own custom cache tags. So for example, in this part of the site there's 2,000 records in this table, each one a node; since it's 2,000 cache tags, like we used to do, he added a custom one called parametric table, and this is the page we're on.

[00:44:46] And then we'd make custom logic for when to clear that cache tag, so we could say, we don't need to clear that cache tag when any of these pages are updated, we need to clear them if their title changes, or some other little piece of it, or if their URL changed, something that's actually relevant here, so that we can minimize the invalidation events based on what actually has to change. That was something we had avoided at first; we were like, oh, we'll just use the core cache tags that are already there, but once we started doing that it made a lot more sense that we really need to have our own cache tags and our own invalidation rules, because then we can control the invalidation much better.

[00:45:47] I could go on, but does anybody have any questions?

[00:46:00] (Question from the audience about the link to the slides.) He just wanted the link to the slides. Sure.

[00:46:37] (Question from the audience about a block on every page.) You said you have one block that's on every page and how to cache it. So we have, for example, our main menu. We have this big giant main menu, it's the same on every single page on the site, and Drupal by default adds a cache context of active menu trail, so we're caching each menu per page, so we have to render the whole entire menu fresh on every page and cache it separately. So we had to go in and change the cache context on that thing, because really, the important thing if you have the block on every single page is to make sure it doesn't vary by URL or by route or anything like that. Really, if it's on every single page, it doesn't even need a cache context, it should have no cache context at all.

[00:47:56] (Question from the audience about caching real-time data.) You're getting real-time data, so how do you cache that? Well, I would say you can't really cache it. I would try to grab it from JavaScript. We haven't really talked about that, but that's kind of the key to all this stuff: try to do it on the front end, so that your back end stuff is all cached, and then do a call in JavaScript to get this stuff and stick it on the page, so the whole rest of the page will be cached, if that makes sense.

[00:48:37] Or placeholder it. Or, if you try to compromise, maybe you can cache it just for a minute or whatever; more than zero is better. So if you're pulling in tweets and you can be fine with it not updated for 15 minutes, then cache it for 15 minutes, use the time-based approach. Ask individual questions and we'll say goodbye. Thanks.