Introducing Drupal Test Traits

[00:00:05] All right, you're at "Introducing Drupal Test Traits." It's a mouthful, but it's super useful when you need it, so I'm excited to tell you more about it. A little about me: like many of you, I've been around a long time. Dries founded the project in January 2001, and I started contributing in October 2001, so I've been with Drupal since the very beginning. I started a bunch of projects you might have heard of, Drush and Devel, and down at the end, DTT, Drupal Test Traits, which I'm proud of. Mike Ryan made Migrate and I was a co-maintainer. I'm also a member of the Drupal security team.

[00:01:21] Here's the agenda. I want to talk about what DTT is, when you use it, how to install and run it, then go into some code to look at what test cases built on DTT look like. Then we'll look at one of the built-in traits, the Drupal trait, and show you what's under the hood, so you see it's nothing too complex. It implements just the code you need to do rapid testing with existing sites.

[00:02:05] If you look at DTT's readme, the slogan is that it's a set of traits useful for testing Drupal sites that have user content in them. Maybe that's too dry, because DTT doesn't click with people at the beginning, so I'm willing to take new slogans. The key difference is this: Drupal's testing suite, which is quite nice nowadays, browser test base, Nightwatch, kernel tests, generally blows away your Drupal site, reinstalls a new one, and then begins the test. That's great when testing core or your contrib module, but not when testing a client site or a real live Drupal site that has content in it.

[00:03:15] A real site will always have content, the same theme it's supposed to have, and menu links, and you really want your tests to click on those menu links, navigate to content that's always there, and verify it still works. With the Drupal core testing suite, before DTT, we had no way to test populated sites, because the expectation was that Drupal would tear down the database at the end of each test and reinstall from scratch. So Drupal Test Traits re-implements part of the scenario setup and teardown to not do that. Instead, the scenario setup just bootstraps Drupal; it doesn't reinstall it. And it turns out that's all you really want when testing populated sites.

[00:04:27] When you're testing populated sites, the traditional tool is Behat, and I have to say it's a brilliant tool. However, it's usually not the tool I need. There's a great quote from the maintainer of that project, who is quite clear about how Behat should be used, while the user community continuously misuses it. His quote: Behat is like no other test framework; instead of proving your system was built right, it helps you build the right system in the first place, by facilitating and enriching requirements communication. Behat is great when you're talking to a product owner describing how a carousel should work; they write it down as use cases in a language called Gherkin, and the developer builds it to match.

[00:06:05] But it is not a good tool for regression testing, which is making sure that carousel stays working for the next five years. That's the job of other tools; in PHP, almost everyone uses PHPUnit for this, and Drupal Test Traits plugs into PHPUnit. There's a great blog post by Fen Proxima of the Drupal community about how he used Behat for regression testing in the Lightning distro, and came to realize his folly. We could talk later about why Behat isn't the right tool for regression testing, but I'll leave it there for now.

[00:07:07] Now you know when to use Drupal Test Traits. The good news is it's super easy to get started. The only step to get the code is a composer require of the DTT package. Then you set an environment variable, either as you run the tests, or in your Docker env file, or in phpunit.xml. That variable is DTT_BASE_URL, and its value is the website's URL. To actually run a DTT test, you run the PHPUnit executable, pass a bootstrap option pointing to bootstrap-fast.php, which DTT provides in its vendor directory, and eventually you can copy and customize your own. Then, as usual with PHPUnit, pass the path you want to test.

[00:08:59] The takeaway is that running DTT tests literally is running your PHPUnit test suite, so all the usual CLI options and arguments apply. Let's look at some code. Here we have a class, SimpleExampleTest, that extends ExistingSiteBase. It has one method, testArticleContentType, with testing code. ExistingSiteBase comes with Drupal Test Traits; it's a convenience class you don't have to use. You could extend the usual PHPUnit test case class, or make your own base class, like a MassGovTestBase if you're working on the Massachusetts state website, which is where some of my examples come from. If you make your own, you'll want to do something similar to what ExistingSiteBase does: include the traits necessary so these calls work.

[00:11:22] On to the first test method, testArticleContentType. It creates a user, loads a vocabulary, creates a term in it, and creates a node with that term and author. This will look familiar if you've written tests for Drupal core or contrib: create user, create term, and create node are traits provided by the Drupal core testing system, so we're just reusing that functionality, and it's well maintained because tons of core tests use these methods. Then node set published and save: we're directly creating and using the Drupal API here. You cannot do that very easily in Behat, where the real awkwardness is that you have to write step definitions for everything, an abstraction layer that gets in the way.

[00:13:19] When you're trying to talk to a product manager and determine the feature set, Behat is great, but otherwise, no step definitions; just use the APIs you want, when creating content and when asserting. We created our node, so we can browse to pages. To make sure our article content type renders correctly, we just Drupal-get the right URL, and there's no special way to get URLs; the node already has a way, so use it. You can log in as a different user during your test, get enhanced permissions, and go to the edit form, getting its URL through the API. Everything you know about the Drupal API works.

[00:14:46] (Question from the audience.) Does this test tear down the data we scaffolded in? Yes. We keep track of the entities created using these traits, and they get automatically deleted during teardown. On BLT: if it's just PHPUnit it really should work; I've looked at it briefly but not the test-running part of BLT.

[00:15:31] The next test does something different: it also extends ExistingSiteBase, but tests the ability of your website to send emails and the correctness of those emails. These are regular PHPUnit methods, so this one implements setup and calls the startMailCollection method, which comes from the MailCollection trait we've included from DTT. That trait looks at your mail system definition and replaces the usual mail system with the test collector, a mail system provided by Drupal core that collects all your emails in the state system. There's a handy method to do that, and one to clear a queue; here we're deleting the unpublished-reminders queue, which is custom code whose job is to send a reminder that content will soon be automatically unpublished.

[00:17:26] The clearQueue method is provided by the QueueRunner trait. Moving into the test method, we create a node and set an unpublish-on field to three days ahead, so the node will be unpublished in three days. The unpublish-on field is provided by the Scheduler module. We create the node, run the queue, which sends the unpublished reminder emails, and once that line runs, the test collector should have collected several emails. Now we assert: we grab the mail collection, make sure we're only looking at mails from the mass-unpublish module, only the ones sent to the author, and that there should only be one sent to the author, and that the CC header is populated so teammates in the same organization also find out.

[00:19:04] If we expanded this test, we might look at the body of the email and assert it has the text it's supposed to. This is a very different kind of assertion, but DTT handles it easily and provides convenience traits, so we've written very little code and it's easy to read. That's what you'll find with DTT: it's developer-friendly. That's important, because we don't love writing tests, they're often slow and difficult, and making them readable and easy to write means your test coverage improves.

[00:20:09] (Question from the audience.) Are DTT tests fast? Yes, very fast, especially compared to core and contrib tests, which set up and tear down Drupal all the time. Even Behat, which doesn't do that, basically always uses web requests, and we don't have to; in DTT we can use the Drupal API to set things up and get things out, so we shave time versus Behat.

[00:20:47] A third test case does something different: it extends ExistingSiteSelenium2DriverTestBase. DTT can test not only your back-end code, like the node rendering we saw, but your front-end code too. Here we're using Selenium to drive a real web browser, doing assertions, clicking buttons, and filling fields to make sure the front end works. We go to the node-add article form, fill in the title, and start typing "term." We're filling the taxonomy tags field, and our next assertion waits for the autocomplete to be visible. This site already has term one, two, and three as testing data, so we assert that autocompletion worked by really typing keys in a browser.

[00:22:33] If we can make sure autocomplete runs on the back end and works on the front end, we're quite sure we won't break it. So front-end testing is possible with DTT; you'll want to read about how to set up Selenium 2. To finish this test, we populate the field with term one via autocomplete, save the node, and assert we have the right title and that term one was associated with the node.

[00:23:31] Here's a peek under the hood at the Drupal trait, really the first one written. Its job is to run very early during the setup method for all DTT tests. We first find a Drupal root, thanks to the drupal-finder package, in a couple of lines. Once we have the root, we get a class loader, and with a request object, we get a Drupal kernel. Then we do a quick change directory into the Drupal root, boot the kernel, and that's all there is. Each test basically boots up Drupal and then stops. That's why you can freely use the Drupal API in your test method: Drupal is fully bootstrapped, and the services you might call are all available. This is the code that doesn't reinstall Drupal all the time; we bootstrap it and that's it.

[00:25:26] (Question from the audience.) Can we run any old Drupal code in a DTT test? Yes. You can do queries, render things; that's all fair game. Moving past the Drupal trait, some more things available to you: we saw the MailCollection trait, and there's a ScreenshotTrait, useful for Selenium 2 tests, where you can take a screenshot whenever you want and save it to the file system with a captureScreenshot method. It puts it where Drupal usually puts its debug HTML files.

[00:26:44] If you're familiar with the HTML output printer that Drupal core provides, you get a new HTML file every time there's a new web request from the tests, which is handy for debugging. That printer works with DTT tests and we use it all the time; it's great for debugging in CI, where you can turn it on and declare those files as artifacts, so they get uploaded to your CI provider and are available for months to figure out why a test started failing. The LoginTrait is a contributed composer package, not part of DTT, that you'll want if you use the TFA module, because logging in is more than filling a form when you use two-factor authentication. It gets a one-time URL the way drush uli does, so it's a drop-in replacement for the login method that works for TFA sites.

[00:28:25] The QueueRunner trait we saw briefly in the mail collection test lets you clear queues or process items, such as sending email, easily from a DTT test. (Question from the audience.) On visual regression testing: it's sort of like that; if you really want visual regression testing, we use a project called BackstopJS, so check that out. (Question from the audience.) Is there an equivalent of Behat's "then I break"? That's the equivalent of PHP's sleep method, so if you want to sleep, just write sleep and put in the seconds you need and rerun your test. Notice we didn't have to go into a step definition; all it includes is the sleep method. (Question from the audience.) Expected exception will work too; you can put those annotations on your test methods, because DTT are traits that supplement your PHPUnit tests.

[00:30:37] That's the end of the slides. You can keep asking questions in the chat. (Question from the audience.) Any way of testing an API-first back end? For sure: take your favorite web client, say Guzzle, and make requests. You probably won't want to use Drupal-get; use Guzzle directly, or look at what the JSON:API module uses. You could skip the HTTP part and just make a request object, but I think it's more straightforward to make real HTTP requests using a web client.

[00:32:03] (Question from the audience.) Is it possible to parallelize the tests? That really doesn't have to do with DTT per se; remember DTT's only job is to bootstrap Drupal and then slowly back away. In general yes, it's easy to parallelize since we're not tearing down and setting up Drupal. The only caveat is you don't want one test doing something that's undone by another running at the same time, so that's about test authoring. When I write these tests I create content on the fly using new nodes, and they get torn down at the end of the test method, so as long as you use new nodes and terms, parallelization shouldn't interfere. How you parallelize uses the same techniques you'd use with PHPUnit.

[00:33:51] (Question from the audience.) On documentation or tutorials for DTT: I'd start with the GitLab project and follow the blog posts listed there. It's intentionally a thin project, so hopefully we don't need mountains of documentation; we can point to PHPUnit for setup, teardown, and assertions, and to Drupal's methods for creating content. On randomly generated parallel DTT as a way to load-test a site while testing for race conditions: I think that would work. There are load-test tools like Locust.io and its Rust cousin Goose that I've used, so I could see DTT playing a part in load testing.

[00:35:19] (Question from the audience.) Are you able to execute these tests on an external site? That's not really the design; the tests run local to the site you're working on. There are other tools for that, but that's not how these work. (Question from the audience.) Could you use DTT in a Behat feature context or plugin, so a Behat step calls DTT? Maybe, but that's so convoluted I wouldn't recommend it. DTT and Behat do the same sort of thing, so pick a side and use one.

[00:36:37] (Question from the audience.) When to use DTT and when to use Behat? Behat is great when you're writing features from scratch for the first time and need to make sure the feature is built according to the product owner's spec. In contrast, DTT, and PHPUnit in general, is about preventing regressions. Generally when people say they're writing tests, they mean regression tests, so usually what you want is not Behat. It's a super cool tool, and I'm happy someone focused on getting the initial authoring of code correct so it meets requirements; we need a tool for that. But it was built so well that it was immediately misused for regression testing.

[00:38:13] (Question from the audience.) On speed differences between DTT and Behat: I haven't measured, but I'd expect DTT to be a little faster in general, mostly because it encourages you to use the Drupal APIs directly and use fewer web requests in setup methods. On when you write regression tests: one common case is after a bug comes up and you don't want it to happen again, but you also write tests when authoring code initially because you don't want it to break in the future, which is also regression testing, so DTT is perfectly appropriate. Maybe "regression testing" isn't the most helpful phrase; "validating functionality" might be clearer.

[00:39:38] (Question from the audience.) If you're doing regression tests on an existing site, what facility stores the data? In general the test methods create the data they depend on, namely nodes, users, and terms, in the site you're testing. Remember, unlike Drupal core, there aren't two sites; there's only one, with no second one getting torn down and installed again. The new data gets deleted afterward. If you need several test scenarios that all create the same nodes, that's no problem; use a trait or a base class and create the nodes and tear them down in each scenario.

[00:40:57] (Question from the audience.) If authoring code for the first time, would you recommend writing a suite of Behat tests first and then DTT tests once you need to prevent regressions? My recommendation is to go right to DTT tests and put them in the module with the code you're authoring. Each module can carry its own folder under tests called "existing-site," where you put your PHPUnit test cases and they automatically get discovered and run. You can immediately go to DTT; I wouldn't suggest doing two sets of tests.

[00:41:56] (Question from the audience.) Am I not using existing data for testing? Sometimes. I might click on a certain navigation on my site, the sports or metro or opinion section; those are menu items, so I'm relying on that existing content to be there. But for other things I usually just create what I need: if I want to make sure article nodes render properly, I create an article node, Drupal-get it, and make sure the text that's supposed to appear does.

[00:43:26] (Discussion from the audience.) Several attendees noted that Behat has been a savior for reproducing bugs when you have no idea where they came from, letting you click through steps before you even commit. Once you've found the bug, though, you don't want to click ten times and wait for all those steps on every CI push, so converting that into a fast DTT test saves developer cycles. It's a trade-off: if a slow test prevents a really bad bug, great; if you can convert it and save cycles, that's great too. It's not either-or. Behat's visual feedback, watching Chrome fill in screens and click buttons, is powerful for front-end and content work, while DTT compresses that feedback into a green check.

[00:51:56] I think there's room for both, and if you're already invested in one, there's no need to change. But if you haven't invested in one yet, I'd check out Drupal Test Traits. There's nothing wrong with having two test commands, one for each; a composer script is an easy way to do it. If you're spending a day fighting a test system, it's frustrating, so work around it. One slow or fragile test is better than no test at all.