From Reactive to Proactive: Automated Performance Testing Success Stories

[00:00:00] All right, this is my talk on performance testing success stories. I'm Nathaniel Catchpole, catch on drupal.org. I'm a senior performance engineer at Tag1, where a lot of the work has been performance audits of sites that would like to be faster, or not fall over. And I'm a half-time Drupal core framework manager at Third and Grove, just working on core twenty hours a week.

[00:00:27] Those are the slides. If you want them locally you don't need to, because they're up there, but you can. Before I get going, I want to do a quick survey. Who here has used PHPUnit, written a test? Who has used New Relic, Web Profiler, KCachegrind, or any of those tools to inspect the application performance of a site? Okay, most people. What about Lighthouse or front-end tools? Okay, most people. So if you haven't, I apologize in advance. And who knows what the dynamic page cache module in core is? Yeah, okay, you'll be all right.

[00:01:22] So, why performance? The reason we should care about performance is that a slow site is not very nice to look at, and you want people to actually look at the website you serve and not wait for it all the time. You use the web day to day, and when pages don't load it takes ages, you can't see what you want, and then it does load and it puts loads of stuff in and it jumps around. It's not nice.

[00:01:49] So when Drupal CMS 1.0 was about to be released, I did a comparison between Drupal and WordPress performance. I installed them both locally and tried to do as similar a set of things as I could. This is not the result of that testing; this is from Google. Google tracks Core Web Vitals for sites that use tag manager and things like that. The percentages are the percentages of origins with good Core Web Vitals, however they define it. You can see Drupal's doing pretty well at 61%; WordPress is only 45.

[00:02:24] I was actually surprised when I compared them how heavy WordPress is. We do fewer database queries, we do fewer function calls. Drupal is lighter than WordPress out of the box, or Drupal CMS is lighter than a WordPress site. Drupal core is much lighter. But for comparable very basic sites with no contrib, we're a bit faster. That's not the only story, though. Once you start adding in different platforms, especially the hosted platforms, we're still okay, but we're not as good.

[00:02:52] WordPress.com, where they control the hosting and everyone's on the same version of core, is like 70% good, compared to our 61. Wix, and Squarespace, which bizarrely dropped in the past months; assuming they fix that, they're closer to 80%. So they're ahead of us, and a lot of that is because they control the hosting platform. They've got the same CDN and everything. But there are things we can do to get back past them. We were faster than all those four years ago.

[00:03:25] And if you see everything going up, that's not because everyone's getting faster. That's because people are getting fiber installed in their house and they've got faster phones over the past four years. This is not websites getting better; this is infrastructure and technology compensating for things getting worse. So only the relative lines really matter on this graph. It's not a sign of the web getting better; it's like fiber came to our house, or 5G, and now it's a bit faster. That's it.

[00:03:56] So, why performance testing? We didn't have performance testing in Drupal for a very long time. The way you find performance issues is you have a website, and your website is very very slow, and it's so slow that someone says you actually need to fix it. And then eventually you figure out what the problem was, and you open a core issue and say, oh my god, you made my site really slow. That's not a good way to find things. Sites going down in the wild is just not very nice.

[00:04:27] It's also really hard to spot when you're about to cause a problem when you're reviewing code. I've reviewed thousands of core issues, and I can visually review whether the code is good, or where the code is optimized, but the interaction between that subsystem and how it's going to impact things, whether an access check is going to happen more often, or whether a cache tag is going to get invalidated more often, is very hard to track in your head. Sometimes you can spot it and most of the time you can't. And sometimes you think things will be a problem and they're completely fine, because no one ever runs that code.

[00:05:03] So it's not the actual quality of the code, or how optimized that particular line is; it's how it runs in the context of the whole request and the site, and many many requests over a day. So I opened this issue in 2009. I said, this is not good, we should add performance testing to core. We had a good discussion at the start about what we could record and how we could make it reliable so it doesn't fail on different people's machines. And we decided it was quite hard, and gave up for about a decade.

[00:05:38] And then Google started funding some work. They wanted open source CMSes to have better Core Web Vitals, so they funded some Tag1 work on lazy loading images and things like that. Later on they asked how we could make it generically faster, more at a system level rather than specific pieces. I came up with the idea of performance testing, and it landed in 10.2. So it only took fourteen years to happen, but it was only about a year from when we started to when it landed. It wasn't that long once we actually figured it out.

[00:06:19] It's not run by Google anymore, and lots of other people have worked on it, not just Tag1. It's just a part of core now that we're using like every other part of core. So we figured out how to have repeatable performance tests, and that was the big thing that was so hard to think about for over a decade. You cannot have a test that passes if something takes 50 milliseconds and fails if it takes 51, because it needs to run on everyone's computer, on different operating systems, when you're running a PHP server in a different tab in your terminal. So you cannot have an assertion on how long something takes; it's just going to be random.

[00:07:07] So we split it into two things. There are things we can count: how many database queries are on the page, how many cache gets, how many asset requests, how big the assets are, loads of things that are the same every time when you write the test correctly, which we don't always do, but they should be and they can be. But we also added a Grafana Tempo dashboard using OpenTelemetry for the things you can't count, because you still want to see the time, you just don't want your test to fail when it changes.

[00:07:37] This is a screenshot of the dashboard; it's all live, you can go and click around on it. There are specific tests that run every hour, I think, on a GitLab pipeline, part of the Drupal core GitLab pipeline. So you can see how long they take, and you can click through to a trace. If you're familiar with New Relic, it's very similar. It has what the database query was, how long it took, in the order it ran during the page, and you can add more events to this. I don't think we have image downloads, but you can add when the image is loaded, when it finishes loading. There's stuff we could add that's not in there yet.

[00:08:17] There's also a DDEV add-on; you can spin this up locally. You just add it in, there's a link to it, you run a test and it will send it there as long as it's a performance test. So you can look at all this locally and click around visually and see what's going on. And then on the other end there's a PHPUnit test. So if you have a merge request and it adds a database query to one of the things we test, your MR will fail, and it will say, here's this query we weren't expecting, your test failed. So as long as we test it, you cannot just randomly add database queries to core anymore.

[00:08:53] We spent so many years just finding database queries that snuck in somewhere we didn't want them, and it's not impossible, but it's very hard, and the more test coverage we have, the less likely it becomes. So when Drupal CMS 1.0 was being prepared, we wrote performance tests for that as well, not just for core. Drupal CMS is a site template; it has lots of contrib modules and also configuration for those modules. So if you write tests for Drupal CMS you're testing about 30 contrib modules.

[00:09:28] The first thing we saw was 250K of JavaScript for anonymous users. That's a lot of JavaScript; what's that? I had a look around and found it was from the Klaro cookie consent module, and someone had already opened an issue, so the test did find it but it wasn't the first to. The packaging was broken upstream and they were packaging something four times as big as it was supposed to be. They fixed that, updated the module, and it went down to 64K.

[00:09:56] But also, why do you need cookie consent on a brand new install that doesn't do any third-party cookie stuff? So I opened an issue against Drupal CMS: do you really need this, can't it be optional? And Jürgen Haas, the Klaro maintainer, said, okay, first of all we're going to fix Klaro so it progressively enables each cookie provider. If you've got YouTube videos on your site you need cookie consent for YouTube, but if you haven't uploaded a YouTube video you don't need it.

[00:10:32] So he did that in Klaro, and then he added a Klaro recipe to Drupal CMS that enables it when you upload a YouTube video. So out of the box Drupal CMS has zero JavaScript from Klaro, because there's nothing; when you upload your first YouTube video it switches it on, and then it's down to 64K from 240. That's the difference in the test: you can see the assertions and the numbers go down. If the numbers go down, it's good, right? That's the Drupal CMS commit after it updated to the Klaro version and the recipe that had the fix in it.

[00:11:18] The other thing we did: Drupal CMS ships with the navigation module, which is still experimental, hopefully not for long. It's not enabled by default in core, so the core performance tests were not testing it; they test the old Drupal 7-era toolbar. I noticed you'd hit one page to warm the cache and then record the performance data on the second page. All the performance tests are like this: you set up the exact scenario you need with the exact cache warmth, the scenario you want to test. So you hit node one, then you start collecting performance data, and you hit node two and see what happens when other stuff has already been done, so the tests are predictable.

[00:12:05] Otherwise it could be different every time, or you only test cold caches, or only warm caches. In that test you go to the other page and it would build the whole navigation toolbar again. The toolbar is always there on every page in Drupal core; if you have access to it, it's on every page. We have a dynamic page cache, and that caches the HTML of the toolbar, but the dynamic page cache can only cache per URL, because pages are different. So you're duplicating not only the HTML in the cache, but also all the work to build the toolbar for every new page you go to that it's not in yet.

[00:12:46] That's potentially thousands of pages if you've got loads of admins clicking around your site, or if you're on a site like drupal.org where you use it and you're also an admin. It could be built thousands and thousands of times a week, and it's a complete waste of time. Even in local development when you're building a site, you go from one page to the other and you do not want that all the time. So we need to cache it by itself and not rebuild it.

[00:13:10] But if you just cache it and add a render cache entry for that, and also have it in the dynamic page cache, you still have the HTML duplicated. You're still storing the HTML in the big cache item in all those places, taking space in Redis or Memcache or wherever it sits, even your database if you have a massive cache table. So what we did: Drupal's render layer has a placeholdering system. Normally we only placeholder little things, like your username. That means if you've got four people with the same permissions who all have access to the toolbar, their names are different because they're different people, but you only have to swap that out on the fly, and everyone else gets the same thing.

[00:13:57] So we can do that; we can placeholder the navigation toolbar, which is a bit bigger, and that way you've got one item in the cache that's loaded on every page. But that introduced a problem: BigPipe. Does everyone know how the BigPipe module in Drupal core works? Wow, no hands. I know one, no, okay, four hands, all right. I was hoping there'd be more. Let's see what we can do.

[00:14:27] So you have a placeholder, and Drupal will print the HTML without the placeholders replaced, they're empty, to the page as soon as it can. That way you get something loading, and maybe some CSS in the header can load, and the browser's got it. Then the response keeps working; the same PHP response goes through the placeholders it needs to replace and builds the replacements. When that happens, there's some JavaScript that looks for the replacements and swaps them back into the placeholders where they'd end up, so you can see them, and then your page gets built.

[00:15:11] There is no Ajax request, no separate request to the application. It's all done in one big request; it's just out of order and then put back together by JavaScript. I hope that's clear; that's all I can do, I'm not going to try it again. So the issue is, when JavaScript is loading parts into your page, it happens after it's sent, and it's going to be janky, so you get content layout shift, or at least flickering and things loading in, and you don't want that.

[00:15:42] You can have a preview. We added support a few years ago and use it a little bit, you know what they look like, that's my MS Paint version. But it's still not good, because you don't want to be clicking on every page and getting the little gray bars and then it comes in. You really want, when the page loads, all the HTML that's there to be there, if it's not expensive. And then if there's something that's a cold cache, that can be swapped in, but the stuff that should be there should be there. So what we did was we added a cache placeholder strategy.

[00:16:18] What this does is it uses the same API that BigPipe does, and it just runs before it. So if your navigation is in the render cache, which it will be after it's been built once, here it is. That's all of the code, by the way; it's little, you don't have to read it, there's not much there. It just goes around the placeholders and asks, are they in the render cache? If they are, it immediately swaps them into the HTML, and that gets sent in the initial response. So BigPipe does not get to see it; it doesn't know it's there, because it's already been replaced. There are no placeholders to replace. And that's like half a millisecond, one millisecond, if it's a cache hit.

[00:17:03] If there are multiple levels of placeholders, like the username inside the navigation bar, obviously it can't do them all at once, but once it's got the first one it goes down to the next level and does those, and the next level. So at any level of nesting it works, and it completely cuts BigPipe out of the system. If you've got a warm cache, there's no BigPipe.

[00:17:30] That matters because BigPipe responses cannot be put in the CDN; it messes up the page headers and everything. Wim, I don't think he's here, has a contrib module for BigPipe that depends on the internal page cache, which puts the page in the page cache so it can go to the CDN. But because we don't have BigPipe on render cache hits in Drupal core anymore, we can just allow those pages to go in the CDN. When you get a cold cache, BigPipe runs; when you've got a warm cache, it doesn't and it's cacheable.

[00:18:03] That means when you have your first response after a deployment, or after someone's created a very frequent cache tag like a node list, everyone will start getting fast responses, and async-rendered pages much faster, regardless of whether they're logged in or not. HTMX is also already in 11.3. BigPipe under the hood uses the JavaScript API, about 150K of JavaScript if you add it all together. It's been converted to HTMX, a new API that just got added, and that's like 15 or 20K. So even when it does kick in, it's going to be much lighter than it currently is.

[00:18:52] Once we fixed that problem, now we can use it everywhere. We made it so most blocks provided by Drupal core and contrib modules can specify, it's a couple of lines, that they can be placeholdered. Everything can be placeholdered now, because you don't have that jumping around and content layout shift with BigPipe when the render cache is warm; it's all instant. There's no drawback, and it's even a single cache lookup from the render cache for all of those pieces. And then that allows us to do other stuff.

[00:19:33] So we've had an issue for a very long time, like ten years. Who knows what cache tags are? Oh, that's a high number of people, good, I won't explain them. Each one of those you have to check whether it's valid or not, so you have to look up the cache tags, and they're all independent of each other, so you look them up separately. Cache tags ended up being most of the database queries we ran on a lot of pages. It's nuts, it's loads. It's better than doing something else, but it was not a light system.

[00:20:10] So instead of that, it now looks like this. Because we can do a multiple render cache get for the placeholder replacements, when we do that, we can also get all of the cache tags and load the invalidations at the same time, in one single lookup. So instead of doing five, we just do one. And again, that works at any level of those things. Separately, Christian added a service parameter that does the every-request ones. On every single request we check things like routes or library info; every HTML request in Drupal core has these cache tags, so they also get loaded in one query.

[00:21:01] So instead of eight or nine, that's one query, and instead of five or six for the placeholders, that's also one. So dozens are reduced, and that's a lot of round trips. Even if it's Redis or Memcache, those are still round trips; they still take some time, and they add up when you have 70 or 80 on a page. Once we did that, there was more we were able to do.

[00:21:26] Who knows what fibers are? Okay, I'll quickly explain. PHP 8.1 added fibers, there's just been a talk about it. It lets you have a chunk of code and execute that code inside a fiber as a callback. It will start, and then when you're inside a fiber you can suspend, which takes you out to the thing that called you, and then that thing can decide what to do. In Drupal we added a fiber loop to the placeholder rendering system, so each placeholder replacement is generated inside a fiber. If something says, oh, I've got nothing to do, suspend, we move on to the next one, go around, come back, and keep doing it. You can suspend and resume as many times as you need, and once they've finished, we swap the results out.

[00:22:26] It's mostly for actual async things, like asynchronous database queries or HTTP requests, where you can fire one off, wait for it to come back, and do other stuff in the meantime. But because of the entity system, we can abuse it for good things as well. When you've got a page, everyone knows roughly what this looks like, you've got a block, it's got a node, it's got a media item inside it, and it references an image, and then there's another one that does the same, and another. In each block they could only be loaded one at a time, sometimes two; if it's a list maybe you can load ten at a time. But once you get into references, it all breaks down.

[00:23:19] So there's a lot of entity loading on every page, and if it's a cold cache, that's potentially ten database queries per entity for each field, so you can end up with hundreds and hundreds of queries to the field tables. What the fiber does is: you start, and you find you're going to have to load node nine from the persistent cache or the database, and it suspends and goes to the next placeholder. Oh, I need to load node one, so that suspends, and it goes to the next and suspends, and it picks them all up. It gets back to the beginning and says, okay, I need to load node nine, node one, node two, and node six, so I'm going to load them all. They go into the static entity cache, and then when it goes around, they're there and already done.

[00:24:05] So you've taken four entity loads and converted them to one. It'll be a single Memcache round trip, or instead of 50 database queries it's like ten, however many it is, it depends on the entity. And then you can do it as long as you've got the same type going around: all the media, all the files. So instead of twelve entity lookups it's now three; again, it depends on the page.

[00:24:29] Then you can do the same trick with path aliases. Path aliases are entities, but they're not loaded the same way, because there can be hundreds needed on a page. From 2008 we had a per-page cache of the system paths that need to be looked up, and in Drupal 7 that made sense because there was no render caching, so every request had to check all the path aliases and render all the HTML pretty much from scratch. Once you add in render caching, that was kind of useless: you'd write it, the next page the caches would be warm and it just wouldn't be consulted. So you're just filling up the cache and not really getting any benefit, and it didn't work when the caches were cold.

[00:25:11] But you can do the same thing. You've got the node nine path, the user three path for your username, the other nodes, and it's exactly the same: it goes around, I need to do one, adds it to a list, goes around again, and once it gets back to the beginning it does them all. If there's a second layer it's two lookups, a third layer three. That's another five queries down to one.

[00:25:37] So what I did was I took the 11.3 performance tests, and all of these are committed, this is done, it's in 11.3 or some in earlier versions, and backported them to Drupal 10.3, then ran the tests against that code base. It was the same test, because the tests have changed a little bit in the meantime, which is why I had to do this. You can see we've reorganized core CSS into different components. We used to put out 6K of CSS on every page; it's now one. The reduction is not quite as good, I think, because we added some different CSS, but it's a 6K reduction. There are 40 fewer database queries, 110 fewer cache gets, 40 fewer cache sets, and 20 fewer cache tags. That's a specific page; it's different for every site, but all sites will see these benefits to a greater or lesser extent.

[00:26:37] All right, what's next? This is all stuff that's not done: asynchronous database queries. Unfortunately, PDO does not support that, but mondrake has already added a mysqli driver to Drupal core that does support it. We need to add a connection pool and things to actually enable it. But that will mean you have a views listing, you fire off a query, you can go around and do all the other things, and then see if it comes back. So if you've got a listing query that's 200 milliseconds and you've got 200 milliseconds of anything else you can do, it doesn't have to be async, it could just be rendering a block, that will come off your total page response time and compact it all into the same simultaneous time.

[00:27:23] We need to fix multiple module install. We made the Drupal installer about ten times faster; it's literally 3 seconds instead of 30. If you haven't installed Drupal, you just install with Drush or from the UI and it's installed, and it used to take quite a long time. But it doesn't work for recipes, so if you've got a site template like Drupal CMS, it's still one by one, it still takes 30 seconds. So please, there are links here for a reason, click on the link and help if you can.

[00:28:03] Because our CSS weight has gone down, one thing I noticed WordPress does out of the box that we don't is it inlines its CSS. If you run Lighthouse, it always complains that you've got render-blocking assets, CSS and fonts. Because of the library system, where everything is on demand, there's hardly any unused CSS out of the box in Drupal core. Themes are a bit different, but you can fix your themes. So we could inline CSS for anonymous users. The reason to do it for anonymous users is that authenticated users are clicking around, and if you inline CSS you don't get the benefit of browser caching and CDN caching; you're sending more with every page. Anonymous users generally click around less, so you can probably get away with it. We can make it an option.

[00:28:47] We want to add async rendering of views rows. So if you had 50 entities in a grid, they'd all be able to do the same thing I just showed with the individual blocks: all the path aliases, the entity loads, and the references can all be scooped up into one thing. Canvas and Layout Builder don't support block placeholdering; it was never done in Layout Builder, the issue has been there for like eight years, and Canvas has the same issue. If you add that, then all of this stuff, which currently would not work inside a Canvas content template, would just immediately work. We just need to add support for it. And we've been looking at the Revolt event loop to formalize how we're doing some of these things.

[00:29:30] There are lots of issues I didn't mention, but I didn't want to go on for two hours. If you want to get involved, there's a performance tag for core issues, that's the link to the open issues, and just join the core development channel on Drupal Slack. There are lots of people who work on Drupal core performance, and they're all really happy if anyone else helps, because it's not enough. You can write your own performance test; we've got tests for Drupal core, but we can extend the coverage. We're going to need to update things with Drupal CMS 2.0 and the individual site templates, because they all have different content modules and configuration, which has different impacts on performance.

[00:30:10] We just added the first performance test for a real site, for london.gov.uk, that I know of anyway. We did a performance audit for them, wrote a performance test, and it showed their cold cache request was like 75,000. We got it down to 10,000. Still a lot, but you can see that on a real site. The way we did that was we added Drupal Test Traits support for performance tests. Drupal Test Traits lets you test with an already installed database, so if you've got a massive complex site, which is probably what you're going to write performance tests for, and the database is 4 GB, you cannot recreate that in a PHPUnit test by creating content, or even loading a database dump is expensive.

[00:30:50] Drupal Test Traits has been around for ages. It lets you test against a database that's just sitting there; you connect to the same one and run a test on it. As long as your test isn't destructive, it's fine. And that now has performance test support, so you can do all the same things you do in a core performance test, you just don't have the setup, you point it at the site that always exists with all its content and configuration. So the actual real site that's on production is what you're testing against.

[00:31:18] There's documentation; I don't need to talk about that. And Sasha has just added performance tests this morning to the redirect module. The redirect module has to do a database query on every page to check if there's a redirect from that page, because it doesn't know. He added a deny list that's dynamically generated from the redirects, and he wrote a performance test to show it works. So now the module does not do that query on every page, as of this morning, it's not released yet, the commit just went in, and the performance test demonstrates it, so it cannot be reverted unless you revert the test. It's going to be fixed. Any questions?

[00:32:09] [Applause]

[00:32:14] (Question from the audience.) That's a lot of really exciting stuff, thank you very much. The data loader pattern you showed, is that something happening at the render level, or is it now in the core entity API and will show up wherever you load entities?

[00:32:31] So it's in, you mean the pre-loading stuff? (Question from the audience.) I mean specifically the pattern where, rather than going and fetching an entity immediately, you pause for a second to gather them all up.

[00:32:43] It's baked into the entity API. It's like five lines in the entity storage handler, maybe more than five but not more than ten. So if it's in a fiber and it gets to the point where it needs to go to the persistent cache or the database, past that point it adds them to the list and tries to suspend. If it's not in a fiber, it doesn't do that. So anywhere you can load entities within fibers, it will work, but obviously you need more than one; if you've only got one, it's just going to go out and back in again. You always need two, which is where the placeholdering helps us, because we've taken the page and, instead of doing things one by one, we can control eight or ten things and how they get executed. Thank Matt Glaman for that; it was his original idea to use it in that place.

[00:33:33] That's what enables so many of these things to happen: we have control of a lot of the page rendering, more than we had six months ago. The same if we add it for views rows, it's the same thing. So anywhere you can add a loop around the fibers, all of this stuff will work. Same for path aliases, and I reckon we can do it for config. I had a little go, I didn't get it working, but there will be other things we can apply it to, at least one.

[00:34:04] (Question from the audience.) The reason I asked, as a GraphQL maintainer, is we actually have custom code, because in GraphQL that's known as the data loader pattern, and we have custom code to do that. I'm very curious to see if we can reuse this. I reckon you could use it. You don't have to do anything; you just have to execute in a fiber and then it will work.

[00:34:26] (Question from the audience.) There are a lot of factors beyond just the website that cause performance issues, for example an AWS outage or a Google Cloud outage. Would this be able to at least give a hint that something's wrong outside of the site?

[00:34:44] So the performance tests themselves are for running in a controlled setting; you wouldn't run them on your production site. You could run them on a staging install. The Drupal Test Traits support means it can be your actual production copy; if you have a copy of your production infrastructure, you can run them against that just before things get deployed, to make sure there's no misconfiguration, say you switched APCu off and sent the containers to staging. It would tell you that's off, as long as you're testing the right things. But it won't tell you that an external service has gone down. Not really.

[00:35:31] But there are other monitoring tools; you can run OpenTelemetry and monitoring tools on production, and they already exist. What you could not easily do is run stuff before, and the big thing is caching it before it gets there. So the existing stuff in production doesn't replace it; it just means you get very early warning, ideally on your own machine, that you're about to cause a problem. I nearly said it differently, sorry.

[00:36:01] (Comment from the audience.) Hey Nate, excellent presentation, this is really great. This was wonderful to watch, because the way you showed it, the username, the one thing that's different per page, that's how I originally, in 2014, came up with a placeholder system. I was working on drupal.org performance at the time. It didn't much help, but I was seeing this pattern, like, this is all the same and we're rendering it all the time, again and again, but we should really switch this out into these two modes: this is always the same, and this is always dynamic. And so we came up with BigPipe and all of these things.

[00:36:40] If someone had said in 2014, when we first introduced it as a quick hack, that's what's happening here, and now, eleven years later, it's finally implemented like this. The render cache strategies we added in 2015 and then never got to use, and Nat was telling me, oh, there's this problem you're having, and I was like, yeah, I thought about that. And that was really cool, because that worked. But my question is: are we planning to use this for entity render caching? Because that's the other point where we could be inserting it, and where we originally thought about it.

[00:37:10] I looked at that recently. It would be nice if, say you've got a grid of 20 entities, they were individually cached and multiple-loaded, because even the individual cache tags on the entities would not cache-bubble up. When you have a small piece of content that's got a cache tag, and that's cached inside a bigger part of the page, everything goes to the higher level, and when the small thing gets invalidated, that trickles back up to the largest thing and it all has to be built again, even if only the small thing changes. So when we break the page up into smaller placeholder parts, they become more interdependent and it improves cache invalidation as well. I should have put that in the talk, but anyway, thanks for asking.

[00:38:01] So yes, but there are situations, entity references get the parent entity in, where it goes a bit weird, because we overload the entity with weird request-based information for previews and different things. But I think we should do it. I don't think it will be easy. We'll just have to try and see what breaks, as usual.

[00:38:33] (Comment from the audience.) Because, if I remember correctly from 2015, one of the reasons we didn't do it at the time was forms, but we solved that in the meantime, so maybe we should just try it again. Yeah.

[00:38:46] No one else? We can call it done. [Applause]