Performance Audits: The Top 8 Problems We Find & How To Fix Them

[00:00:03] Thank you all for coming. This is the talk about performance audits and the most common issues we see when doing them. If you'd like to follow along with the slides on your laptop or phone, here you can get access to the slides. I'll show the QR code one more time a few slides later, so if you miss it, don't freak out.

[00:00:36] My name is Janez. I'm a senior engineer at Tag1 Consulting. I used to be the lead of the Media Initiative for Drupal 8, and I was also at the largest Drupal website on the internet at the time, called examiner.com. That was not the first time I was introduced to performance, but it was a totally different beast, and since then performance has been dear to my heart.

[00:01:16] A few words about the company I work for. It's the second all-time contributor to Drupal. We are a globally distributed team of experts with the largest concentration of Drupal core contributors of any organization, and we contribute a full-time infrastructure expert to the Drupal Association, so we basically help you when you're using issue queues and everything else on drupal.org. We work with many clients from different industries, a lot of enterprise, government, nonprofits, you name it.

[00:01:57] First I'd like to explain what a performance audit is. We are usually approached by clients when they notice they have problems on their website, either it's too slow for their needs, or users start to complain, or even worse, sometimes sites go down. We're approached to try to figure out what the causes are and how to fix them. Usually it's way too late, sometimes weeks or days before the go-live, or it can be too late in the sense that there were years of technical debt that accumulated, and then it's really hard to roll that back.

[00:03:02] What we do is review the site first. We talk to the client, they tell us what the problems are, is the site generally slow or are there specific pages that are problematic, and based on that info we go in and review the site. We try to identify the biggest problems and also the smaller ones, and of course we usually want to fix the lowest-hanging fruit first, things that give us the biggest bang for the buck. Sometimes we just provide recommendations and the internal teams implement them; other times we either do them ourselves, or we assist or train the client's team to do it themselves.

[00:04:11] Now let's talk about what we see the most. By far the most common problem we see on Drupal 8, 9, and 10 websites is incorrect use of cache, specifically incorrect use of cache metadata. By cache metadata we mean cache tags, cache contexts, and cache max age. Those are the things that define when something that is cached will be cleared or invalidated, how it will vary, and for how long it will be cached. If this information is correct, Drupal's cache system is really powerful and will help you a lot.

[00:05:03] If you don't provide the correct metadata, or if you don't provide it at all, you start getting problems, and these problems are sometimes really weird and hard to debug because they won't show up immediately. They only show up when you have multiple users on the site, and then you see that somebody is seeing something they shouldn't, because it was some other user that generated that part of the markup, and since the cache context is not correct and it's not varying correctly, it's being displayed to another user that shouldn't see it, or should see it in some other form.

[00:05:52] It's often easy to overlook this when you're developing locally, because you're either running your local environment with caches disabled, or even if you're testing with caches you don't test with every user in all the different combinations of how these users interact with the site. So it usually appears on UAT or, even worse, at production.

[00:06:27] In my opinion, the most important thing when it comes to fixing this really common problem is to learn and understand how caching in Drupal works. There are plenty of blog posts and DrupalCon sessions, and it's really not that hard. We have to remember that in Drupal everything is cached, and when you're doing something you have to think: let's say you have a block and you need to rely on the user's role when generating it. That probably means it needs to vary by role, or by permission, so you have to add the user role or user permissions cache context on it, and if you don't, you have problems.

[00:07:25] Core also provides debugging tools that help with that. First is our HTTP headers, where all the cache metadata for the page you're looking at is displayed. Here we can see a header that tells us there was a cache miss on this page, which contexts are used, how long it will be cached, when there will be a cache hit, and all the cache tags. This is useful if you're trying to debug a page in general, but when you have a lot of blocks or a lot of pieces of a page, it's sometimes hard to figure out which part of the page caused a session or a certain cache context to appear.

[00:08:30] This is why we also have this debug output that, when you enable it in services.yml, adds an HTML comment above the block. Any piece of the page that can be cached, with this setting enabled, will have something like this above it. Here you see basically the same information as before, except it's specific to that part of the page, so for the most recent news block you see which cache contexts are attached, for how long it will be cached, and whether it was a cache miss.

[00:09:07] You can also see the rendering time below, which means this is also a performance measurement tool in a way. If your page is slow and you disable caches so nothing is cached, then run this, you'll see that most blocks render in 50 or 100 milliseconds, and then you notice there is one block that takes 10 seconds to render. You'll know that's the block causing your problems.

[00:10:06] Related to the previous problem is the next one. Very often, when people experience the first problem and don't know how to fix it, they resort to this one: they just disable cache. The problem goes away because now things work, but your performance starts to suffer. Everything is okay when you have low traffic, but then your site grows. This is actually the worst scenario, because when you start doing this it becomes a habit. The first time, you see okay, setting max age zero on some render array fixed my problem, and next time I have a similarly weird problem I can use the same fix. If this goes on for years, your site is full of these things, and at some point you hit the tipping point where everything falls apart.

[00:11:17] The problem, and why this is so bad, is that over the years you accumulated so much technical debt that it's now really hard to remove. You have to spend a lot of time going through the codebase, figuring out where these things are and removing them, but then you also have to fix the bug that caused you to resort to this fix in the first place.

[00:11:44] Another similar thing is if you're building views. In views configuration you have the caching setting, and by default there are three options: time-based, tag-based, and none. Tag-based is default, and none basically disables cache; it's the equivalent of max age zero and it can potentially affect the whole page it sits on. So even if this is just a small part of the page, it can affect the entire page. The fix is to never do it. Seriously, I've never seen a valid use case for that. Even if you have a part of the page that is updated really frequently, it's still useful to cache it for at least 5 or 10 seconds, some short period of time; it's still better than caching it to zero.

[00:12:53] With lazy loading and Big Pipe, for example, if you have something updated really frequently you can separate the rendering of that out of the rest of the page, so the page can still be cached and then this piece that updates frequently is added at the end of the rendering pipeline, which is actually what Drupal does by default if you do things correctly. But, and I'll come to contributed modules later, there are cases where contributed modules cause this to stop working, and then this frequently changing block again affects the whole page. If you think you need it, you probably have the first problem, so the recommendations from the first one apply.

[00:14:03] The next one is a little bit hidden functionality in Drupal, and many people don't know about it, and the situation that makes it bad is kind of blurred, so we have to dissect it a little to understand when it can cause problems. Drupal core by default has an entity list cache tag for every entity type it has. Even if you have a custom entity type, core adds it by default, so you don't have to think about it, it's always there. That means that if you update any node, or create a new node, any create or update operation on nodes, it will invalidate this cache tag, which in turn means that everything that has this cache tag added to it will be cleared or invalidated.

[00:15:22] The thing is, views automatically adds this cache tag to every view. The reason it does that is that when you build a view, views are cached by default. If you have a list of most recent news, and it wouldn't do that, and you added a new article, that list wouldn't update. So it's a good thing we have that. The problem is that it's too general. If you're just listing your articles in a block and you have dozens of content types, any of the other content types will also clear this block, which is only about articles. So it's overdoing it, and it makes sense that it's overdoing it, because Drupal needs to make sure views work in every situation, but it also causes a performance hit.

[00:16:25] Now imagine a scenario where you basically build the entire site with nodes, all the content you have are nodes, and you update those nodes very frequently. An example would be a content type coming from some other source, with a script that runs regularly and imports this stuff all the time. If you have a lot of those, you're basically clearing this cache tag all the time. So yes, you're using cache, but it's invalidated all the time, so you're basically not using cache. And everybody uses views for everything, and everybody uses nodes for everything, so what could possibly go wrong?

[00:17:26] There are actually quite easy solutions for that. The simplest is the Views Custom Cache Tag module, which adds another option next to those three in the cache configuration, called custom cache tag. When you choose that, it removes the entity list cache tag from the view so it doesn't cause problems anymore, and it gives you a text area where you can type in the cache tags that are more optimal for that specific use case. Core already has content type specific cache tags, so you can use node, colon, article, and that means this view will only be invalidated when an article updates, which is what you want. You can probably solve most use cases by using these.

[00:18:37] If you don't, you can very easily invalidate your custom cache tags yourself. I remember an example where we had a block that showed which colleagues of yours have a birthday today. Users don't update that often, so this shouldn't be a problem; you shouldn't be invalidating the cache of this block frequently. Except if you're using single sign-on of some sort, because SSO systems usually send fields about the user to the site every time you log in, and that gets saved. So often when you have SSO, users are updated every time the user logs in, which means a block like this is being invalidated all the time.

[00:19:40] It's really easy to fix: you hook into the saving of users and you check if the birthday changed, or maybe even something more specific than that. It won't change every time you log in, because your birthday doesn't change, and then this block can stay cached basically for the entire day after it has been generated.

[00:20:13] Another issue is a lot of joins, particularly in views. This is basically caused by Drupal's data model, which puts each field we create into a separate table. That's great, because it's powerful and flexible, you can have single-value fields, you can have multi-value fields. The problem is that when you have listings, aka views most of the time, and you want to filter on those fields, especially with a really complex view with a lot of filters on different fields, you need to add joins to bring these fields into the query and then add where clauses on those joins. MySQL sometimes can optimize this, but often it can't, and often these situations cause queries to become really slow.

[00:21:29] If there is a left join, which is when you have this checkbox that says require this relationship, if you don't require the relationship that's usually a left join, and a left join is worse, because the size of the data set grows way quicker than with a normal join. Instead of joining just rows that exist, it adds null rows for everything that doesn't have this field, and when you have a few joins doing this, the data set the database needs to work with explodes quite quickly.

[00:22:18] The most efficient fix for that, in my opinion, is to use custom entities where it makes sense. It's seriously not a sin to create a custom entity in Drupal; the Entity API is quite powerful, so it's not that hard to do, and even when you do it, you still get all the goodies, the entity form, formatters, use it in views, everything works more or less out of the box. The reason this solution is great is that if you have a custom entity with single-value fields that are base fields, everything is in one table in the database, so it's much easier to query, especially if the data set is large, and it's easier to index.

[00:23:16] Also, if everything is nodes, everything on the page is in one table, node core field data or just node. If you have custom entities that each handle their separate thing, you also separate the data into multiple tables, so your data sets by default are not that big. Obviously it depends on the use case, but it's seriously not a sin and it's seriously not that hard.

[00:23:53] Another thing that's almost a sin in the Drupal world is not to use views when you need some custom display. Creating a custom block that loads whatever you need and displays it exactly how you want is not that hard to do, and sometimes it's way easier, because views has to fire up all the plugins it has and generate a query that needs to be very general, so it's usually not that optimized. Templating it is harder because you end up with a gazillion templates potentially, but if you have a block where you load stuff and print it out in a render array, it's also easy to maintain and not that hard to do.

[00:24:51] If you have to use views, we often do what we call the IN subquery trick. On the left side you have what views would do out of the box: you have a field on a node, you put a condition on it, and this is roughly how the query looks. If you figure out this query is slow and that this is the field causing the biggest damage, you can use the trick on the right side, which is basically querying the field table first, getting entity IDs out, and then on the main entity table using where on the entity ID, which is a primary key and is indexed, so it's generally much faster.

[00:25:49] To do that you usually need to create a custom views plugin for a condition or contextual filter, but it's not that hard. You can do a subquery, or you could do two separate queries and bring the IDs into PHP and put them into the next query; both work. Often the difference between these two is substantial, like a few seconds or even more than 10 seconds down to 200 milliseconds.

[00:26:33] I already touched on this one. Views are great, they're one of the reasons Drupal is so successful, in my opinion, but you can also overuse them. There are situations where they probably shouldn't be used. I've seen views used as a field formatter. Let's say you have a reference field and you want to display the things referenced in the field, and instead of a formatter that loads it from the entity and displays it, there was a view that joined from the field to the nodes table back to the field again. It already had a few joins before it even started doing what it needed to do; it was basically using view as a template layer, because you can add fields in the UI and it's easy to do, but it's not really the use case it was made for.

[00:27:50] Also, as we've seen, views out of the box have to be really general when it comes to caching, so if you do a custom field formatter or a custom block instead, you can be much more specific about how this thing needs to be cached, which brings your cache hit ratio up and improves performance.

[00:28:27] Another very commonly used trick, and another thing caused by views wanting to be very general, is the count query. As soon as you have a view that needs a count, and that would usually mean either a full pager or the total number of items displayed somewhere in the footer or header, it doubles the execution time. It does that because, in order to get the count, it takes the query it already has and just puts count around it and runs it again, which means if you have a slow query that takes 10 seconds, you just added another 10 seconds on top.

[00:29:21] So this is one of those low-hanging fruits. If there is a slow query that's hard to optimize and we want immediate improvement, we suggest: would you be happy using a mini pager instead of a full pager? A mini pager doesn't need a count, because it doesn't have first or last numbered pages, it's just back and forth, and you go forward until nothing is displayed, so it doesn't need to know how many items there will be.

[00:29:54] It's really okay to custom code your block or formatter. Views are great for prototyping and for a lot of things, but sometimes when you have more complex things, going custom is the better approach. Also, measure what you're doing. When you're building a view, enable the setting that displays the query below the view, which gives you the performance statistics and you'll see how long your query will take. If you add a semi-realistic data set in your table by using the Devel generate module or something like that, you'll have a rough idea how long it will take for that view to execute, and if you see that even in development it takes a few seconds, that's probably not okay, so we have to figure out some other way.

[00:31:05] Another one that's views-related is overly complex queries. It's easy to click things together and not look at the query, and then if you do, you see it's like this, and even if you throw contrib modules into the mix, it can get even worse. Easy to configure and looks good in the UI doesn't equal well done and performing well.

[00:31:45] I remember a case where we had a view with really complex language fallback options and conditions, and it was hard to do, but there was a contrib module that did it. The condition section of the view was this big, but it worked and it performed well. Then somebody found a module that did the same thing kind of, and that module was used, the old condition set was removed, and just a single line was added to the conditions section. It looked great, it was made smaller, but the query that module created was awful, so it was actually way worse.

[00:32:41] So again, same recommendations: look at the query while you're building the view, use explain on the query to figure out what it's doing, what the sizes of data sets are. I also always recommend making this part of a peer review process. If there is a significant change to some query or view, add a comment: this is covered by this index. Add the explain output into the issue so reviewers can review whether it actually is covered by the index and the explain does look good. It's about culture. If you really want to achieve great performance, it also needs to become the culture in the team and be part of the day-to-day process.

[00:33:54] This one kind of goes against the "there is a module for that" mentality in Drupal. Contrib space is awesome, Drupal wouldn't be the same without it, but if you just blindly install 500 modules on your site, you're doing yourself a disfavor, because more modules means more complexity means a slower site. You also have to be aware that contrib is not perfect. Even core gets weird bugs and regressions sometimes, and in contrib this is even more frequent, because there are fewer eyes looking at it. Sometimes a module is maintained and built by a single dev and nobody ever reviews it.

[00:35:01] When was the last time you installed a module and actually reviewed it and tried to understand what it does? If everybody does that, great, but if everybody is just blind-installing modules and there's a single person working on it, there's not a lot of quality assurance going on, and things can sneak in, not just in smaller modules but in quite widely used ones. One example I was really surprised about is the Context module, still being used, probably not as much as in Drupal 7, but I've seen it used in Drupal 8, 9, and 10.

[00:35:53] It has a bug that destroys the auto-placeholdering Drupal provides. This is the thing I mentioned before, where you have a piece of the page that is updated frequently or varies for every user, and Drupal detects that, removes it from the page, renders it separately, and injects it at the end, so it can cache the rest of the page. Context module has a bug that completely disables that, so if you have, say, a cart block that varies per user, it ruins the cacheability of every page that cart appears on. It also completely disables Big Pipe, so if you have Context module and Big Pipe, you're basically not using Big Pipe at all. This issue has been open for six years, and there is a patch, so if you're using Context, use that patch, but it's still not committed.

[00:37:02] My recommendation is: don't be afraid. When you bring a contributed module into your project, treat it as a custom module initially and do a review on it, try to understand what it does and figure out if there are things that are potentially problematic, and try to fix them, work with the maintainer, provide a patch. Also, it's not a sin to have custom modules on your project. If the decision is between a small fix in a custom module and a huge extra contributed module you'd need to install, go with the first option; it will be easier to maintain long term and cause fewer problems down the road.

[00:37:59] The problem with this is that as time goes on and you keep accumulating, it's really hard to scale back, because even your team members will change, and there will be a module that at some point nobody knows why it's there, and you'll be afraid to disable it because you never know what it will break. So it's better, when you don't need something, to disable it and remove it from the codebase, so it's clear it's not used anymore.

[00:38:36] The last one, and the hardest one to fix, are data model issues. In Drupal you create things in the UI and rely on Drupal to create a data model, and we're coming back to the fields table and stuff like that. Using nodes for everything, as we've seen, can be problematic. Using complex field types like paragraphs for things they were not meant for is also wrong.

[00:39:12] Once I had a situation where we had a Drupal instance that was data storage for data scientists, and they were using APIs built with views for these data scientists to pull the data and work on it. In order to get nice filters and views, they used paragraphs for nested complex data, which, as soon as you had a decently large data set and started using filters on those API endpoints, took forever to query. On the other hand, if you thought about it before, you could use a custom field type with multiple properties, because most of those paragraphs were exactly that, a few fields on another field.

[00:40:23] The problem with this one is that if you don't do it in the architecture phase, when you have the site live with real data in it, it's really expensive to fix, because you basically have to redo the entire architecture, migrate the data, which costs money and is time consuming. At some point you're rebuilding your entire site again, which is not ideal. I promoted custom entity types and custom field types before, and this is one of the solutions. Don't blindly trust Drupal to generate a good data model for you with the content types you create and the modules you install. Do it and then check what's in the database, try some queries on realistic data sets, and see what happens. It could be fine, or you could find problems that would bring you a lot of headache down the road.

[00:41:27] To conclude: the type of site matters. All the things I've been telling you about were with an enterprise project in mind. You have to be pragmatic. If you're building a hobby site, like a site for a small local sports club, it probably doesn't matter; you can do all of those and it will be fine. It's still nice to fix the lowest-hanging fruit. On the other hand, if you're building an enterprise thing, or something that could become one in the future, you absolutely have to think about these things, because otherwise, by the time you notice there are problems, it will cost you a lot to fix them.

[00:42:26] Never guess, always measure, or don't blindly trust. When you're testing performance, only change one thing at a time, otherwise you won't know what brought the improvement. Measure on realistic data sets, use the available tools we mentioned, build it into a peer review process, integrate it into your culture, don't be afraid of custom code and custom entities, and you always have to be learning, because there's always something new.

[00:43:06] Speaking about tools, I'd like to mention Goose, an open source load testing framework we're developing at Tag1. It's built in Rust, but we love it, and there are examples to use it on Drupal, so check it out. We're also building Gander, a new thing in Drupal core that brings performance testing to Drupal core, which should in the long term prevent performance regressions from sneaking into core itself. It's open source, there's a base test class in core, and it's coming out in 10.2, so you can start using it to performance test your websites, either immediately using the patch in the issue, or wait until 10.2 and you won't need to patch.

[00:44:01] Especially if you already have CI and tests running for your project, it's really simple to do. Just look at these issues in core; there are a few example tests already to performance test a single page, and you need four or five lines, so if you already have this running it's really easy to add. If you want to learn more, there's a blog post on the Tag1 website with links and a little background. Thank you for your attention.

[00:44:42] For questions, there's a microphone there, or I can repeat the question.

[00:45:33] (Question from the audience about reusing paragraphs.) So the question was about paragraphs, whether reusing paragraphs would help. In general I think it would, because if you reuse them a lot your data set would become smaller. So it really depends on how much you'd be able to reuse, but in general the answer is yes.

[00:46:17] I'd also add that paragraphs are great if you're using them as components. If you have a paragraph and you use it to display content on a node page, it's great, it works fine. The problem with paragraphs is if you start using them in listings and need to add where conditions on paragraphs, because then, say you have a node and there's a paragraph field on the node, that's one join, then you have to join the main paragraph table, another join, then the field on the paragraph, another join, and then you add a condition on that, so that doesn't work well. But if you're using paragraphs as a component tool, it works great.

[00:47:22] When you're just retrieving values it's not a problem; it's when you're trying to filter on the values in a paragraph. I've also seen pages that used paragraphs as components but used hundreds of paragraph entities, and that was a problem, because it needed to load hundreds of entities to display the page and was rendering forever. But if you're not overdoing it and using it as a component tool, I think it's a great tool.

[00:48:15] (Question from the audience about Ajax performance.) So the question is about Ajax performance, specifically Ajax forms, because the Ajax request on forms can sometimes be slow. The thing is, if you do a lot of things during that Ajax request that take time, of course it will be slow, so it comes down to backend performance. You need to make sure the things you're doing are fast, that's obvious, but also, when you're doing Ajax you usually update just a part of the page. Every time an Ajax form request goes, the entire form is rebuilt, so if you're doing some compute-heavy things in the form build that aren't needed for that specific Ajax use case, maybe you can make sure those things are not executed at all.

[00:49:35] (Comment from the audience.) So to repeat the comment, there is a proposal to fix that, because the problem is that every Ajax request rebuilds the entire form, and there's an idea to change that to only build whatever needs to be built. It's not done yet, but if you want to have fun, you're welcome to.

[00:50:00] (Comment from the audience about Search API.) The problem with the joins is only for filters and sorts, not for displaying fields; I just checked and I didn't see this added to the query. And one way to avoid these joins is to use Search API. Exactly, and then you can index the paragraph fields, so you can go three levels into the paragraph and configure your search index. Of course, if you go too deep and have a super nested big thing, maybe you get pollution of your index. I also had this problem on the display side, that you had to index some stuff just for display, so Search API maybe needs to be improved so that for display we actually load the real entity and for filtering we use the indexed data. You're exactly right with everything you said, and you just made me realize I have to update my slides to mention Elasticsearch or Solr. We don't have time for more questions, we're out of time. Thank you for coming, and have a nice evening.