Workshop: Gander in Practice: Start monitoring your site's performance today

[00:00:00] Hi, thank you for coming. This is a workshop about Gander, the performance testing framework we've created for Drupal. The idea is that you do hands-on work, which we'll lead you through. If you'd rather just sit and listen, that's also fine. But by the end of the workshop you'll have an environment where you can run performance tests for Drupal core, you'll learn how to write these tests, and ideally we'll add a few tests to core and submit merge requests. Let's see how that goes.

[00:00:49] This is the URL to the slides. You'll want it, because we have steps on the slides, and it's easier to copy and paste into the terminal than to type everything from the screen. The QR code appears after a few slides too, so if you miss it now, you'll get another chance. We are Janez and Nat, both at Tag1 Consulting. Tag1 specializes in performance, so this is natural work for us, and we're one of the biggest contributors to Drupal, as a company and as individuals.

[00:01:52] I used to be the lead of the Drupal core media initiative, and I worked at examiner.com, which at the time was the biggest Drupal website on the internet, and that's where I first got really interested in performance and learned a lot. And I'm Nat Catchpole. I've been working at Tag1 for about 15 years, mostly on performance audits and things like that. The other half of my time I'm a Drupal core committer and framework manager, and I created Gander, the tool we're showing you today.

[00:02:42] A few words about Tag1. We're the second all-time contributor to Drupal, and we have the largest concentration of core contributors of any organization in the community. We donate a full-time infrastructure expert to the Drupal Association, so if you use the issue queues or ever went to drupal.org, this is what we help run. We work with enterprise clients from many areas, and in the enterprise world, performance often comes up.

[00:03:34] A few words about Gander and why we've done it. First, why think about performance at all? The main reason is that slow websites turn away users, which reduces your reach, your revenue, whatever you wanted to achieve with the site. Performance is also a factor when search engines rank your site, so a slower site generates less traffic. And the real-life example: do you like to stand in a line at a bank? No. So why make our users do the same?

[00:04:22] The problem with performance is that when it's neglected, which it often is, it becomes really hard to solve once problems pile up. It's worth thinking about from the beginning, but that's often hard to justify, so it doesn't happen. Neglected performance can lead to availability issues, which is the very worst outcome. Traditionally we thought about performance mostly in the context of time to first byte, which is backend performance, but nowadays, with a lot of front-end code and interactivity, the front end matters more and more. So we have to think about both, and Gander measures things on both ends of the spectrum.

[00:05:24] At the end of the day, slow websites generally consume more CPU power, which consumes more energy, which is worse for the environment. So if you care about performance, you're doing good work for the environment as well. Nat mentioned performance audits, one type of project we do. Clients approach us when they have performance problems and ask us to analyze their site and figure out what's wrong and how to fix it. Usually it happens way too late, when there are way too many problems.

[00:06:14] The most common problems are usually present on all the audits we do: caching problems, misconfigured views, and excessive use of contributed modules, and we're talking hundreds. A bigger problem to fix is team culture. If the team culture doesn't include thinking about performance and reviewing it, that can lead to actual performance problems over time.

[00:06:54] We created Gander together with the Google Chrome team. They approached us about two years ago. They want to make the internet faster, and they decided to work with open source platforms to have the biggest reach. Initially we contributed some features to core, specifically lazy loading of images and iframe embeds, and after that we wanted to do more. We thought about the next step and decided to do performance testing. That's how Gander was born.

[00:07:39] If we go back 15 years, this gentleman here opened an issue about adding automated performance testing to core, in 2009. So many years later, the same person led the effort and designed the whole system, and now we finally have it. And this issue is now actually closed. Is it? Yeah, I actually closed it out.

[00:08:07] When we were considering doing this, we understood it's a long-term investment. It will take a while to see real benefits, but we thought it was worth it. Before Gander, performance testing in Drupal was done manually, by people who are already overworked, so it was only done occasionally. An even bigger problem was that we didn't have consistent testing, so every time the results were different and we couldn't compare apples to apples. Issues were often detected only when somebody noticed on a production website, which is not great.

[00:08:55] Automated performance testing fixes all of those problems. It's a big investment, because we need to learn to use it and develop habits, but if somebody told us to develop core without functional testing at all, which used to be the case, we couldn't imagine it. We hope it'll be the same with performance testing.

[00:09:25] Drupal core already uses it, and it can be used in two ways, which we'll show you. It can assert on performance metrics when you write a test, and you can also have tests you run regularly that send metrics to a dashboard, where you can observe long-term trends. The assertions are part of the normal test suite now, so if you have performance assertions and something bad happens, the normal test suite fails. We're also running it regularly, and on gander.tag1.io you can see the dashboard for core. Today you'll learn how to start using it on your project.

[00:10:17] This is how the dashboard looks, the one for long-term statistics. Gander isn't really old, but it already brought a lot of improvements to core. The biggest one improved the core test suite runs by 10%, which is huge considering how many test runs we have on a daily basis. Just with that improvement, it basically paid for itself. And we keep identifying more things that can be improved, and adding performance stats so regressions aren't introduced.

[00:11:05] So what is Gander? It's basically a base test class that's been in core since 10.2. It's a special type of JavaScript functional test, so in practice any functional JavaScript test can be converted to a performance test. It's currently measuring the metrics on the screen, and you can assert on these metrics after you've navigated the site and done whatever you were doing.

[00:11:41] This is roughly how it looks. Those familiar with tests in Drupal will recognize the drupalGet calls. The first two lines are what we call preparation, where you prepare the environment, usually warming up the caches or preparing some content. Then everything you want to collect performance metrics for is done in the collect-performance-data closure, and everything inside it has its metrics collected and returned as a performance data class, which you then assert on at a later point.

[00:12:35] So let's start using it. A little show of hands: has anyone already tried using Gander? Okay. Who's familiar with Drupal testing in general, running and writing tests? Okay, cool. How many of you have ever contributed a core patch? Awesome. How many of you already have a working local environment for Drupal core where you can run tests? Okay. And how many of you are using DDEV? Okay.

[00:13:36] The intersection is probably quite big, but nonetheless, in this workshop we'll be using DDEV, because that's what we use and what the official Gander documentation suggests. You don't need to use it, but it'll be impossible for us to help you if you're using some other environment. The requirement is that you can run JavaScript tests; then you should be able to write Gander tests.

[00:14:06] Prerequisites for today are to have DDEV and Composer installed on your machine. If you don't, please start now. We have a few people here to help you, so if you run into problems, just raise your hand or shout and somebody will come help. At the end we'll be writing performance tests and hopefully submitting merge requests for core. The problem is we don't have as many issues as there are people in the room, because honestly we weren't expecting you to be this many. So when we get to that, we'll work in groups, with a few people around you working on a given issue and reviewing each other's work, because otherwise there won't be enough issues for everybody.

[00:15:30] This is the first step in the workshop: creating a new environment. If you already have one you can use it, but if you don't, or you don't want to pollute it with this stuff, you can create a new environment. This is based on the docs on the Drupal wiki. This page has a lot of information about Gander, and what we're doing today is taken from the quick-start section, so you can always refer back to it.

[00:16:23] There's a slight twist. The quick-start section uses a Composer project to bring up the environment. Since we'll be contributing to core today, a Composer project would make that a little harder, because it's harder to create a merge request. For that reason we'll be using git clone instead. It's very similar; the way you get things is different, and the folder structure is a little different, but everything else should be the same.

[00:17:05] Another thing: if you don't have DDEV and you're concerned about polluting your machine, there's a slide at the end that will remove everything you've done, with no trace left, if that's a concern. So let's do it. I'll sit down, otherwise it's harder for me.

[00:18:03] As you can see, when I install Drush I'm using ddev composer. Here I'm using the host, just because DDEV isn't configured or running yet, so that won't work, but it doesn't make a difference. So we're cloning core. I hope we won't flood the Wi-Fi now, since we're all doing it. Any questions so far?

[00:19:00] The issue queue is a few slides down. This is the meta issue. It's really slow. If you really want to work on an issue and don't want to work on the same one as everyone else, and you can think of something, you're welcome to add a child issue of your own devising. The bottom three are examples of things you could come up with. But also, if five people work on the same issue, you can review each other's merge requests, RTBC them, and get it committed, so either way it should work.

[00:20:05] To speed this up, I disconnected from the conference Wi-Fi and I'm using my cell connection. If it's slow for you, that's one workaround, especially if you're from Europe and can use the EU-regulated data. So we checked out core, and now we move into that directory. We have to install all the Composer dependencies, then configure DDEV.

[00:20:59] Here it asks for the project name. I'm fine with Gander, the folder name. It usually successfully detects the docroot and what kind of project it is. Now we can start the environment. First I have to install the Selenium add-on, which lets me run JavaScript tests. And now I can start it. I'll also install Drush, because I'll use it to install Umami. It's probably a good idea to have it, though you might not need it.

[00:22:37] At this point we can start running tests. We do that by logging into the DDEV shell and running the normal PHPUnit command. We'll run a single test, because these tend not to be the fastest, since they need to install Drupal and everything. So I log into the DDEV shell and run the test. This will take a while, so in the meantime we can look at this test and see what it does. Which one was it? The authenticated performance test.

[00:23:46] So here we're testing the front page of Umami as an authenticated user with warm cache. Depending on the state of the cache, your performance metrics, your database queries, your cache sets and gets will all be different, so it's really important to know what state you're testing in. By visiting the front page twice, we warm the cache, so these two steps warm the cache and we're not collecting metrics there. Then in the collect-performance-data closure we visit the front page again and collect the performance data as a class.

[00:24:52] The performance data class is a simple class with a bunch of attributes, getters, and setters. This is one way to figure out which metrics are currently being collected, because as core moves, we're adding more. We won't remove them, because that would be backward-compatibility breaking, but we're constantly adding them. Nat just recently added the byte size of CSS and JavaScript files loaded on the page, which weren't there before, and that let us ensure that after optimizing those sizes, they stay that way. If the size of assets on a page changes, the test fails, and we have to figure out whether the change was justified.

[00:26:00] So we're asserting that there will be two stylesheet files on this page, at 44,000 bytes, and one JavaScript file at 125,000 bytes. Then we assert that we expect exactly these four queries to run, and the test will fail if another query appears or one goes away for some reason, which would probably be a good thing, but we want to know what happened. We also assert on query count, which technically isn't needed here, since asserting on the array already asserts there'll be four. And we assert on cache requests, cache tag invalidations, and checks.

[00:27:10] Now we can go back to our test, which successfully ran. We can see there were 15 assertions and everything is okay. Now we'll be a little naughty and make a change that breaks something. I want a change that would be impossible to catch with a standard functional test, because the functionality stays the same and only performance is affected.

[00:28:00] I'll do that by changing one of the views on the front page of Umami to be uncached, which is one of the things we see in our performance audits. When caching doesn't work, when cache metadata is not correct, instead of fixing the real problem, people often just disable cache. That fixes the problem functionally, but introduces a huge performance problem. Sometimes it's not noticed, because maybe the site didn't have a lot of traffic at that point, but as traffic or content grows, it becomes a bigger problem.

[00:28:59] If I go to the front page there's this view called Promoted Items, and, like all views by default, it's tag-based cached, as it should be. If I disable cache, we'll see what happens with the test. So I've changed it in the browser, but this environment installs Drupal from scratch for the test, so to achieve the same change I have to change the YAML file in the code base. I look for cache in this view, see we're using tag-based caching, change it to none, and start the test. It should fail.

[00:30:11] The important thing is that functionally everything is the same. It's impossible to detect this change with a functional test, unless you went into the Views UI and asserted on it, but that's not the point. The performance test will detect it, because there will be more queries on the page. This is how you prevent somebody from doing something like this. You can see the test failed, and it's complaining about the queries. We remember the four queries we expected, and we definitely didn't expect the fifth one. That's how we detected this unwanted change that affects performance.

[00:31:12] This screenshot you can't really see is basically the code I showed before, so we've been through that. We mentioned there are two ways of using Gander. One is what we just saw, asserting on performance metrics to keep things in good shape. But we can also run regular tests and collect metrics over time.

[00:32:03] The first way works if you're running standard Drupal JavaScript tests, without any issues. For the long-term collecting of performance metrics, we have to introduce a few more things to the stack, because we collect the data and the test sends it to OpenTelemetry, which is eventually fed into Grafana and displayed there. There are quite a few services needed, so to make your life easier we created a DDEV add-on that prepares all these services in your local DDEV environment.

[00:32:51] We're also publishing Grafana configuration files for the graphs displayed on the dashboard we use for core, so if you want to set this up for your custom project, you can use that configuration as a starting point. To start doing this, I need to add another DDEV add-on, the Gander add-on, which adds Grafana, Prometheus, Tempo, the OpenTelemetry collector, all the things we need. We restart the environment, because all these services need to be configured, and then we start tests and run them in perpetuity, because for graphs to appear you need a few data points.

[00:34:16] Just to say, the reason there's both the OpenTelemetry dashboard and the PHPUnit assertions is that the metrics on the dashboard are things we didn't want to add assertions for in PHPUnit: time to first byte, largest contentful paint, and things like that. Those are times, and times vary every run, so if you add an assertion you'll always have false negatives or positives at some level. Rather than adding assertions with loads of headroom that could still be wrong, this lets you see how they're doing over time. There are limitations, because a small regression over time you still might not see, but it's more chance than zero. You could see the Umami largest contentful paint was really bad a few months ago; it stuck out badly in the graphs, and then Mike Herchel fixed it and now it's okay.

[00:35:32] Now that we've restarted the DDEV environment, we have Grafana on this URL. It's HTTP, so if you put in HTTPS it won't work, like it didn't for me a moment ago. This is Grafana running inside your DDEV environment. I already started the test, which failed because we still have the change we did before, so let's revert that and start again. After a while, when the test runs, we'll see traces appearing here for each test and data for the metrics in the graph, but we have to wait for two or three iterations. This is a nice opportunity to take a break. Let's meet back in 15 minutes, and I'll keep the test running so we can see the graphs update.

[00:37:16] Welcome back. First, does anyone need help with installing the environment or running tests? Feel free to ask, especially now, because it could get a little more interesting. Back to where we left things: we still have these performance tests running, and now they've run a few times, enough that we should see some results in Grafana, which we do.

[00:38:15] On the graphs on the left side there are data points for some metrics: largest contentful paint, first contentful paint, and time to first byte. Those are collected on the front end, and we usually don't assert on them because they're not deterministic; they always vary a little. But they can be really useful for figuring out what's happening long term. The data points are a little all over the place, because I have other things running on the laptop and the run times vary. That's why we recommend, if you do this for your own project, having a dedicated test runner just for these tests, to keep it as consistent as possible. That's exactly what we do for core.

[00:39:34] On GitLab CI you can add custom test runners, so Tag1 has donated one, the only runner for the performance tests. There's one machine that always runs the same tests with the same configuration. I think it's still on AWS, but it's not used for any other test jobs, only for these, and it gives better consistency than if it sometimes had other tests running on it. It's not perfect, but it's pretty consistent. On the right side of the screen you get traces for each individual run, and if you inspect an individual trace you can see all the metrics in a timeline.

[00:42:08] As far as I understand, Nat, you were able to identify quite a few performance issues just by looking at this. Yeah, it gives you a browsable overview of which queries are running. For example, Drupal core tests were creating the lock table every time they logged in, because no one had ever logged in before on every test install. Actually they weren't even creating it, they were checking if it existed, because the password was always correct: it only gets created when the password is not correct. That's something we never knew happened on GitLab CI, but there was this lock query in the traces, and I had no idea why it would be there, and was able to debug it and figure out where it came from.

[00:43:11] So you kind of have to look at it to know what to expect, but once you know what should be in there, stuff sticks out really quickly. Devel query log will also find you these things, but the big difference is that if I'm testing with Devel query log or Webprofiler or Xhprof, every time you want to check a change you have to follow your five steps to get to the right amount of cache warming. The difference here is that you do that in the test, so every time you run you always get the same queries, and the only time it changes is when something actually changed, not because you forgot to refresh the page. It makes it easier to find things that stick out.

[00:43:59] (Question from the audience.) It's only I/O, so what it doesn't do is function calls and things like that that you'd get with Xhprof. It would be possible to add that; especially with modern PHP there's loads you could do to collect the number of function calls and show them in the traces. The problem is that adds overhead to the test collection, so other things get affected because you're collecting more data. But we should add the capability, because it would be useful. You might have to have tests with and without, but there's no reason not to do it, and then we'd get even more granular data. That's a to-do.

[00:45:00] We've done this, we've got data into Grafana, and now we've come to the point where we'll ask you to help us write some tests. There are a few performance tests in core right now, most of them in Umami. There's only one other, testing the standard installation profile. Some existing tests are missing assertions, because they were created before the additional data was collected, so we have a few of those where it would be nice to add them. We'd also like to add performance tests for a few pages not tested yet in Umami: specifically admin content, the content listing, viewing an article, and Nat also created one for the taxonomy listing and the menu UI.

[00:46:25] The goal is to contribute patches to core. There's a meta issue, the link on the slide, with child issues for all these things. The idea is that we split them up. I'll demo the first one on the screen, but I won't necessarily create a merge request, so you can pick that issue too, because I don't want to take attributions from you. In the meanwhile, go pick one of the child issues, work on it, and leave a comment that you're working on it so others know it's taken. As we said, maybe collaborate in smaller groups and review each other's work. Everybody in yellow t-shirts is here to help, so if you're stuck, just wave.

[00:48:20] Just to say, the first issues, that add assertions, have the least work involved and the most structure. The ones with a new page to test also need you to create the test boilerplate, because that test doesn't exist yet. So depending on how much work you want to do, that might influence your decision, but someone please choose the last three so people work on different things. Decide for yourself how much you want to take on.

[00:49:00] This issue is about the node page performance test with cold cache. The problem is that the cold-cache test doesn't have assertions for cache requests, database queries, and JavaScript and CSS files, because those metrics didn't exist when this test was created. So the goal is to add them. Let's look at the test. We're working on cold cache, and it basically doesn't do any assertions yet; it's just collecting the data. The reason is that we wanted to collect the data for the Grafana dashboard, since this was one of the first things we did with Gander, and never got to adding assertions.

[00:50:32] So what we'd do here is look at one of the tests that already has assertions and basically copy and paste them, then fix the numbers so they represent what's happening on that page. This test has many more assertions, so I'll copy them over to this test. If I ran it now, it would fail because the metrics are different. Usually we run it and then fix it with the values that come out, and also consider whether it makes sense for the numbers to be what they appear to be.

[00:51:39] Since these tests run for a little longer, it makes sense to dump the entire performance data and fix all of them in one go, rather than one by one. In this specific case, because we weren't doing any assertions, we weren't assigning performance data to a variable, so we also have to add that, otherwise we can't access the metrics. Now when we run this test, it'll fail because the assertions fail, but we'll also see what the metrics are. Then we update the assertions and, in theory, the test comes back green.

[00:52:52] (Question from the audience.) What happens to me sometimes is that tests get stuck and I have no idea why. Then I check the CPU usage to see if the test is actually running, and if I think it's not, I restart DDEV, and that usually fixes it. I suspect I'll have to do that now. Does that happen to you? Yeah, sometimes it hangs. Okay, I'm glad I'm not the only one.

[00:54:15] (Question from the audience, about network requests during collection.) If you have the add-on installed, it adds a Gander endpoint, and that sends the trace data. So there's a network request when it collects performance data: one network request to the OpenTelemetry collector. It's not that big, but it might not help.

[00:55:10] (Question from the audience, about test overhead.) It does add a bit. The performance collector collects two things: network data, and backend data. The backend data is collected by a test Drupal module that runs on the tested site and logs the queries. It's a bit like what the Webprofiler module does in contrib, using the same technique, the database query events and things like that, and it writes to key value and gets it at the end. So it's a little overhead on the tested site, but everything is in memory and it's an end-of-request listener, so it's the minimum overhead I could come up with, so it doesn't mess up time to first byte.

[00:56:14] The other end is PHPUnit getting the Chrome performance log, which is a stream: you don't get one log per request, it's just a stream of a log. So it has to wait for previous requests to finish, including image requests, which could be creating image derivatives, and that can take a while. There are a couple of sleeps in there, and then it tries to detect the beginning of the request we want, and waits until all the HTTP requests from that request have finished, keeps parsing the log in a while loop, and comes out. So it's like an extra three to five seconds for the collect-performance-data method. They do take slightly longer to run, but we're talking about five seconds longer.

[00:57:20] (Question from the audience.) A normal functional JavaScript test won't even run the performance log, but now it does. If I was going to do a contrib project, I would not just switch this on for every test. I'd have a performance test class, like we've done for core, so you've only got the overhead on the class you're actually doing this on. Also, most of these don't do much of a functional thing; they just load a page or submit one form. You don't want complex testing steps. You're doing something pretty simple to hit one interaction and see what happens.

[00:58:27] So, this being a cold-cache test, Drupal produces about 407 database queries for this request. Copying that from the console took some imagination, but I think I succeeded. Now we can also update the other metrics: cache get count, set count, delete count, and the checksum valid count, which is 81, and invalid, which is zero. And the rest are on top: two stylesheets, one script.

[01:00:03] One thing here: we originally started with hard numbers of the exact number of bytes for those assertions in core, and annoyed every core developer who had to make the tiniest change to any stylesheet or JavaScript file. So instead we basically round up to the nearest 500 bytes. That way, if you do a small refactoring to a file you don't have to touch the number, because it's still below, whether it goes down or up we don't care. But someone who does a big refactor that goes up still gets caught. The only issue is it doesn't fail when you make the file smaller, so we could make it a lower bound as well. That's a thing to add, an idea for a contribution. The number is there, so you just have to add a greater-than line; you don't need any support, just add the assertion.

[01:01:05] I think I forgot to do this before. When I first ran this test, I used var_export on the performance data class, which gave me all the metrics immediately; otherwise I'd have to go assertion by assertion. The reason we use var_export instead of var_dump is that var_export formats the output as a nice PHP syntax array. Otherwise you'd have to turn whatever var_dump gives you into a real PHP array. I still had to remove the numerical indexes it adds for the queries, but that was easy with a regular expression. Now, in theory, if we restart, this test should pass. That's how you write performance tests, and this is what we're trying to do with the issues we were talking about.

[01:02:24] Let's see if anyone picked up any issue. If you're not logged in, it might not update to tell you. We failed again. It's an interesting problem. Is it possible that the ordering of the queries will vary? Yeah, it's possible.

[01:03:47] Okay, this is a tricky one. Because it's a cold cache, the asset aggregates have not been created, and nor have image derivatives. That means, because we're actually visiting the page in Chrome, those requests also occur when the performance data is collected, and the database queries from those requests also get collected by the test module. It won't empty out the queries at the end of the request; you empty the queries out when you get the data from the module. So if you have these extra network requests, they can be issued in a different order, or even in the same order the queries can run at a different time, because the different requests run through their steps at different times.

[01:04:50] The easiest way to avoid that issue is not to do a 100% cold-cache test. If you add a step where you visit the page so it generates image derivatives and asset aggregates, then clear the cache in the test and visit the page, those files exist on disk. It's essentially like a running site that has had a cron run: the PHP request is cold, but the asset requests are hot, and that's more predictable when you collect the performance data. If you wanted to test the asset generation queries specifically, you could delete one file and visit that URL directly. So you can still test those things, but it's usually better to have a couple of cache-warming steps.

[01:06:13] For this one, if you look at one or two of the other tests, those steps are taken. I think the standard performance test does a couple of things and then clears the cache, cache rebuild all, and then collects the performance data, and that avoids the race conditions between the different requests. It's a problem because it's realistic, but it's a bit too realistic, so it becomes hard. The other thing you can do is not assert the individual queries and just assert that there are 400, or however many; that should be the same every time. And if it's not, you learn something interesting, but you might not have a test that gets committed to core, because it might have a random failure. Still, for finding out what's going on, it's useful information.

[01:07:15] (Question from the audience.) Could we sort the query array so it's always ordered the same way? We could. The only issue with sorting the query array is that if you're debugging and want to see why a query is running, it shows you the order the queries ran. For the traces you want the queries in the order they ran and how long they took, but for the assertions maybe we could do that instead. Certainly the test could do it: the test could sort the output when it gets the queries, just one sort on it. That way it's not in the API, but you can do it in an individual test when you run into this situation. So now I commented out the specific-queries assertion and left the others, and let's see what happens. That's just an example of how much pain I went through getting this to work when I was first starting.

[01:08:36] (Discussion continues.) It seems the number of queries is varying too. That might be the order the requests complete in. For example, a hook cache: if you have an image derivative request before an aggregate request, certain end-of-request caches get built differently, and then the next request has more or less things in that cache, and a different number of queries run. So if I visit this page before I collect the metric, and this rebuild-all clears all the caches, then the aggregates and the image styles should be created. Now it will obviously fail because things will be different, but we'll see.

[01:09:55] Now we are asserting on the queries, let's see how many there will be. Oh, I didn't save the file, that'll be useful. Now it's probably a good idea to export the metrics again. Okay, that seems to be running. Any other questions?

[01:11:50] (Question from the audience, restated by presenter.) The question was, how would you do a test where something doesn't happen with stock core but does happen when a contrib module causes a performance issue? The example was that path alias preloading only runs when a certain cache item is set, but there was a core patch that broke that cache item, which caused path alias preloading to fail.

[01:12:21] The way you could do the contrib module example is add a test module that recreates that, doing just enough of what a contrib module would do to cause the same problem, would be my first thought. For the core patch idea, I don't know if we have test coverage for path alias preloading, but if we have performance test coverage that looks for the preload path alias query and asserts the count when that's in place, we might be missing coverage for that functionality. So adding a test that explicitly requests a path with several aliases rendered in such a way that the path alias preloading can work, you could add that as a test scenario to core, and then the patch that broke it would break that test.

[01:13:40] That's the kind of thing where, as we find performance regressions in core, if we can create tests for the scenario that weren't tested, not only will we not break those things again, but they'll cover certain combinations of cold and warm cache that aren't already covered, or different pages. The more coverage we have based on things that have gone wrong, the fewer issues we'll introduce. The other thing with the contrib module is to write a test for the contrib module and run it on their GitLab pipeline every time anyone submits a merge request. For high-use contrib modules, that would be great, because a lot of real-site issues are core, but a lot are not: it could be Redirect module, or CTools, or something like that.

[01:14:49] It failed again, but I'm not sure why. One reason is that I left the export in, so now the output is huge, and that might be the reason. Yeah, it failed just because of the output. Oh no, there are still some variations with queries, it seems. In the cache-warming step, did you visit the front page or a different page? This page, the page we're testing, node one. It should be okay. Oh, it might need to be visited twice.

[01:15:52] I would visit it twice here. Why? Because it fixes it. It is a bit of switching it on and off again, but the reason, especially when you have multiple requests issued by one request, is cache collector. A lot of Drupal core uses a cache collector. When you're getting things from state, plugin definitions, or something, we don't load the entire huge amount of data on every request. Instead we build up a partial cache of hooks and hook implementations, or path alias prefixes, and each request builds on that.

[01:17:02] With a cold cache you start with nothing, do a request, and the big discovery caches get built. But then the next request, for the hook implementations, we only record the hooks actually invoked on a page. When you hit a different page, different hooks get invoked, and the list gets written at the end of the request, after the response is sent to the browser. So when you have image derivatives and asset aggregates, you're hitting like eight PHP requests at once, but when we write to the cache, one of them will win, the one that starts at the beginning and is the last to end.

[01:18:00] By hitting it twice, you ensure that the first request, whatever wins, could be one of those asset or image requests, but on the second request those have been done. That means the request to the node page itself, the actual PHP request that generates the HTML, will be guaranteed to be the last HTTP request to finish, so it will write cache collector items to the cache key object, and then your third request has a consistent state. That's why. So I think we should add, someone mentioned, a helper that's a closure you pass a URL, and it hits the URL twice and then clears the cache, so you can go from that step onwards. Even with warm-cache setups you still want to do that without the cache clearing. There's a performance test trait class, and if anyone wants to take that on, that's a good issue to open. It doesn't exist yet, but it would be a good one.

[01:19:39] Even after that, it seems the number of queries and the ordering is fluctuating a little, so we probably picked not the best example, since cold cache is the hardest one to do.

[01:20:15] (Question from the audience, about Drupal Test Traits.) The first answer is no, there isn't yet integration for Drupal Test Traits, but it could be added. There are two parts. Because it's a performance test trait, you could add another trait. There's a performance test trait class that has most of the work, and then there's performance test base, which extends functional JavaScript test and does a little handful of things in setup that weren't easy to put in the trait. So if you made a new base class that has the performance test trait and the Drupal Test Traits stuff together, then you've got maybe a dozen lines of glue to make it do what you can copy from performance test base.

[01:21:08] That would give you all the OpenTelemetry integration out of the box, because that's from Chrome, so everything front-end: stylesheets, script files, images when it's added, there's an issue, it's not in yet, and the time-to-first-byte stuff on Gander would all work once those steps are done. What would not work is the query collection and the cache integration. But you could change the existing test module so that, instead of creating a database, because you can't access the database as easily to pull the data out, if you got it to write JSON at the bottom of the page at the end of the request, you could read that JSON from PHPUnit and get all the same information. It just makes it slightly less realistic, because you're putting extra content at the bottom of the page, but to use it on a real site that might be the trade-off. So all the things you can do with this could be done with DTT, but there's work involved to port it over.

[01:22:43] We have just two or three minutes left. That test is still failing, but we'll try to figure it out and open a merge request on the issue for that test, so you can come back to the issue a little later and see how to make it work. As promised, this slide has all the steps to remove everything we added to your environment, so you have a clean working machine.

[01:23:29] We'd like to ask you to start using Gander. If you're a core developer, start writing more tests, as we tried to do today before we ran out of time. If you're a contrib maintainer, you can also start writing performance tests; assertions will work out of the box, and if you want your own Grafana dashboard for your module, there are options, and you can reach out to us. Same with custom projects: if you find it useful, especially if you're already doing automated testing, it's quite easy to start using Gander. And if nothing else, just tell somebody about it, so people know it exists and can start using it. Thank you for coming.