A robust command line tool for all Drupal sites

[00:00:00] I think we should get started. Welcome to "A robust command line tool for all Drupal sites." A little bit about me before we start: I'm old on this project, since the year it was born, 2001. I've been a core contributor, which is kind of unique. I've started a bunch of projects you might have heard of: groups.drupal.org, the Drupal code of conduct, the Drupal 7 contrib experience, and recently Drupal Spoons and Drupal Test Traits. I'm the maintainer of Drush and the Devel project, and a member of the Drupal security team.

[00:01:07] We have a short agenda, because this is intended as a conversation. I'll invite you to ask questions in chat, or just come and join via voice and screen. We're going to talk about the current CLI world as it pertains to Drupal core, and then to all the Drupal sites we build and run for clients and ourselves. We'll talk about my proposal, which is a new, composer-based approach to getting Drush into every new Drupal site: add the Drush package to the recommended project. Then we'll talk about the benefits and considerations, and open it up for discussion.

[00:02:22] Here's a quick slide of some CLI commands in Drupal core, implemented as Symfony console commands. There are just a few, and the implementations are quite basic. There's the db-dump command, which only works for MySQL and doesn't support a lot of what sql-dump does in Drush; you can't pick which tables to dump or add extra command-line options. That command basically exists to create dumps used during the test suite. The install command only works with SQLite and doesn't support multi-site, profiles, or fields, lots of what site-install supports in Drush. And there's a server command to quickly spin up a PHP-based web server, analogous to Drush's runserver. That's all we have. There's a long-standing ticket where people say we should have a command-line tool in core. I partially agree, and later I'll say how I propose to fulfill that.

[00:04:06] This is how Drupal sites get built nowadays, per our documentation. You start with composer create-project and pass the argument for the recommended project. Recommended project is a slimmed-down version of what we used to use, called the drupal-project. It gets you Drupal core, some scaffolding plugins, and some getting-started documentation. Some people are still using drupal-project, but that's not what the docs tell you to do. The next step most people take is composer require drush/drush, because they need to do a site install, enable and disable modules, log in as any user, import configuration, check their site's security for Drupal releases or PHP advisories, or deploy their site into a new environment, which is a new command that comes with Drush 10.

[00:05:47] I see Doug mentioned require-dev; that's not my recommendation, but let's not get off track. We have sql-sync, sql-query, rsync, and lots of other things people need. Clearly nobody needs everything in Drush, but enough people need a given thing that it's in there, and there's no Venn diagram that includes exactly all the things you need in a small set. That's why I don't think the core ticket, where we build a few key commands, is really going to work out. Someone asked about Drupal Console: Jesus, the maintainer, posted recently that it's on hiatus, which makes sense because it hasn't had significant changes in a long time. So there's really only one actively maintained CLI tool for Drupal.

[00:07:06] Here's the proposal from the agenda. Step one: add drush/drush to the recommended project. The effect is that everyone gets a command-line tool along with Drush at the beginning of their project. I really think that's the right way to go. Drupal has had this ticket asking for a command-line tool in core for a long time, and the recommended project gives us a way to cut this Gordian knot: some people wanted to put Drush in, some wanted to build from scratch in core, but now we can put Drush into the recommended project and not into drupal/core, which satisfies the different concerns.

[00:08:07] As for the three commands I listed earlier, db-dump, install, and server, those can stay in Drupal core as they are. My vision is that core uses Symfony console commands to the extent it needs them for testing itself, and to the extent they're useful for end users, people can use the existing Drush commands that come with the recommended project. I'm really happy the recommended project came along, and I think it's a great place to put Drush, and maybe a few other things.

[00:09:02] Let's list some benefits. First, out-of-the-box experience: we'll have achieved the title of this presentation, a command-line tool for all Drupal sites. It's pretty frustrating to install Drupal and not have this, then have to go into the documentation and figure out that this tool, sometimes mentioned and sometimes not, is what you need. The out-of-the-box experience is far improved with this small change. Once we expect Drush to come along with Drupal for new projects, we can make our documentation consistent. The install docs are pretty schizophrenic about how Drupal should be installed, and we can make better decisions when drush site-install is assumed to be available in the default case.

[00:10:23] This also gives a good signal to contrib that they're welcome and encouraged to ship Drush commands. Right now they have to decide between no commands, Symfony console commands, Drupal Console commands, or Drush commands. The security of Drupal sites will also improve. Drush ships with pm:security and pm:security-php commands, which are ways sites can verify they don't have insecure modules or insecure PHP dependencies. The insecure-modules part is also covered by the update module, but the insecure PHP dependencies part is really important, and I venture to say not many Drupal sites are monitoring their PHP dependencies. With Drush built into the recommended project, we'll have much better compliance with this practice.

[00:11:46] Drupal deployment is a completely ignored topic in the documentation. We document some of the steps, update.php, config import, cache rebuild, but the real process isn't well documented. Drush now has a deploy command that standardizes how you're supposed to do deployments, or at least gives a best practice, and if people want to do something different, that's fine. That best practice in the recommended project will help smooth out some of the problems people have adopting Drupal. And there's the common-sense point that Drush is over ten years old, quite popular, and not going anywhere, so it seems particularly foolish or stubborn to start rebuilding even part of it in core when we have this recommended project we can put useful packages in.

[00:13:33] Here are some further considerations. Whenever we depend on a new package we have considerations, and this is no different. Release management needs to be coordinated between Drupal and Drush, and as the Drush maintainer, I'm committed to following all the best practices and being as good as the best dependency Drupal has. If we like how we coordinate with PHPUnit or Symfony, I'd love to follow that model. The security team needs to coordinate with Drupal's dependencies, and as a member I have a good feel for how that works and am committed to making it happen. Drush isn't proposed to be a dependency of drupal/core, but it is proposed as a dependency of the recommended project, so these things still apply.

[00:14:51] Drupal core and Drush will have separate issue queues and repositories. I don't see that as a problem; in our modern PHP world, people are used to their application being built by a number of projects with different issue queues and repos. There would be some benefits to having everything in the same issue queue, but that's not how it's proposed. People have brought up testing, which is important. Drush is tested with Drupal really thoroughly: multiple PHP versions, multiple core versions, multiple database backends, and Drupal is never mocked out of our tests, which distinguishes us from Drupal Console. We download real Drupal with composer, so we know when our commands stop working with Drupal. Our tests are slower because we don't mock dependencies, but we've caught so many bugs this way that I'm happy we set it up like this from the beginning.

[00:16:28] We have lots of old, cranky people in the chat and in Drupal, myself included, so there's an easy way to opt out: if you use the recommended project, your next command can be composer remove drush/drush, and you're right back where you started. I see few downsides and lots of upsides, and I'm really curious to hear what you think. Let's open it up; you can request to join the voice presentation or ask questions in chat.

[00:16:56] (Question from the audience.) What's the objective here, just Drupal 9 and Drush 10, or also Drupal 8? Good question. The recommended project is literally only used once in the lifetime of a site, to launch it on its first day; it's like a birthing tool. New sites are being birthed on Drupal 9 right now, very few on Drupal 8, so it's not a priority to add it to the recommended project for Drupal 8. We can discuss it, but at first glance it's not a big priority.

[00:18:38] (Question from the audience.) Would there be any consideration to add the ability for Drush to turn features on or off based on environment, say on production make sure nobody can run db-drop? We have a mechanism for that now, called policy command files. If you google "drush policy command files" you'll find examples; there's one in the drupal-project that shows how to prevent sql-sync into production and rsyncing over your whole files directory. Take a look at that, and we can talk about additional ways to prevent mistakes, but that one has served us well for the last few years.

[00:20:23] (Question from the audience.) Given all your responsibilities, do you think this proposal to do more release coordination would increase or decrease your overall responsibilities? I don't really see it increasing. We've successfully coordinated with Drupal core for a long time now, and if we have to make new releases every six months, we will; I'm not concerned. The list of maintainers for Drush is already fairly long, so other people can step in to make releases if I become unable to, and I welcome adding to that list. Submit a few PRs and usually you'll have me knocking on your door asking if you want to maintain something. I think there are enough people to carry the load, and it doesn't feel burdensome.

[00:22:15] It does acknowledge a little increased coordination, and core maintainers could be concerned about that. One hard thing about being a core maintainer is that you have all these children you barely control, that you depend on and need to coordinate with, and I just want to commit to being not a problem child but a teacher's-pet kind of child.

[00:22:53] (Question from the audience.) I maintain some of the migrate system in core. There have been requests over the years for putting things into Drush core for migrate support. Do you think putting this into recommended would encourage that or reduce those requests? I think some form of migrate runner in Drush makes sense. On my last project I used the migrate_run module and thought it was great, so that's my current idea for what a runner in Drush core could look like, but it's not a done deal, so input from the migrate maintainers would be very valuable. Let's keep that conversation going.

[00:23:58] (Discussion from the audience.) What if it wasn't in Drush core but in Drupal core itself? How many workflow or other core modules might we want to add Drush commands to, and would this open the door to an officially supported way to do Drush things while working with core? That's totally fair game, and it would work out of the box: Drush commands in modules that core ships with. It's a matter of getting the community and core maintainers to sign off. I could see arguments for putting it inside core, or into Drush core, or into another composer package; there are lots of options, and I'd be supportive.

[00:24:24] (Question from the audience.) Can you explain a bit more how the process will go? Will you talk with the core maintainers, will there be an issue, and do you want to deprecate those three core commands? People can correct me, but I think we need to open a pull request into the recommended project and an issue in the core issue queue for discussion. As for the existing commands, I have no particular passion for changing them. The db-dump command probably stays because it's about the testing system; the server command seems like it doesn't have to stay, but I'd like to know more about how people use those, maybe they're useful for core mentoring.

[00:26:36] On whether Drupal core might need a Drush dependency: that's not something I'm advocating for, but I also wouldn't be opposed.

[00:27:12] (Question from the audience.) In my world I create and modify content types all the time, and the only thing I've ever needed Drupal Console for is exporting all the bits and pieces of a content type in the Drupal 8 world. Without it, I'd export everything and use git to hunt for the various YAML files. How would I do that with Drush? In general, yes, you export everything and use git to take what you need. There's the Features module for partial exports, and Chris will probably bring up his content-hub thing that does something similar. If Console has a feature people love that makes sense in Drush, let's do that. There are lots of ways to service that need. I'll add it to the queue and we can work on it.

[00:29:28] (Question from the audience.) This change should have happened ten or fifteen years ago. Drush moved away from Drupal because of release coordination issues, since it wasn't Drupal-release-specific, and you got frustrated. Now it needs to move back toward Drupal and toward coordination. There are long-standing people issues that prevented this fifteen or twenty years ago. Are those solvable, or is this just a technical release-management problem? It's a good point: as usual it's easy with the technology, and the people are the problem. I haven't read the back-scroll, so I don't know how much support or fury there's been, but if people are grumpy in the chat, we'll get the same grump in the issue queue. I think there may be less resistance this time, because Console isn't as big a thing anymore and we're no longer proposing to put the command-line tool into core, so a lot of the real sticking points are gone. I'm hopeful there'll be some success this time.

[00:31:46] (Discussion from the audience.) You can't actually use Console with Drupal 9 anymore; there are different versions of Symfony in use, so Console isn't even an option for Drupal 9, and the same for 8.8. That's good input and will help. Along the lines of the people thing, this only works if people are vocally in support. We really need people to jump into the issue and say they support it and help argue for it, because the status quo is very powerful and this is a change to it. I haven't created an issue yet; I wanted to see how this session would go. We probably want to start a new issue rather than go into the existing command-line-in-core issue.

[00:33:22] (Question from the audience.) I actually supported the command-line-in-core issue more than this. It seems like the model is that we do something in contrib and then move the key, API-ish parts into core. Why not move the engine into core? Because it's really big, and the core process would take years to get something that big in, whereas we have ten years working in this model successfully. There's actually very little benefit now. I think the model you're describing made more sense before composer, but now we're all getting comfortable with an application that has many dependencies, and the "just throw it into drupal/core" approach doesn't make sense. Why didn't we take HttpKernel or PHPUnit and put the best parts into core? The same thing holds for Drush.

[00:34:52] Maybe Drush has several projects: a Drush core that gets integrated, with core commands, and more optional things that aren't always there. Drush has been factoring out parts of the application into other projects, so that's a way forward, but in my mind it's a multiple-year plan. And in all those years, we have a poor out-of-the-box experience, schizophrenic documentation, and people who can't check their PHP dependencies because they don't have Drush in their project. I just think we can move faster and better with this proposal.

[00:36:10] (Question from the audience.) I'm curious, since Drupal Console has stalled, whether the commands that were useful in it could be triaged into Drush, or whether Console could be a good substitute. The big thing useful for our team is that we're new people who haven't worked with Drupal, PHP, or Symfony before, so a lot of that scaffolding and the generate commands help bootstrap and accelerate feature development, and I'm a little concerned. Console was innovative with that scaffolding, and Drush added the same functionality with the generate command. If you run drush generate, you'll see lots of things Drush knows how to generate now, so we have good parity there. I'm happy to close any discrepancies, but I think we have a good scaffolding tool now. So Drush has consumed a lot of the unique features of Console. Initially Console consumed Drush and did the same sort of thing, and then Drush consumed Console's innovations.

[00:38:03] (Question from the audience.) My main concern, and I support this idea, is whether the Drush maintainers would be willing to coordinate with core and maintain the same cadence of dependency support, so that if Drupal 9 doesn't drop support for a PHP version until 9 is gone, you keep Drush at the same cadence. That's the issue now: people can't upgrade to Drupal 9 if they need Console, because Console didn't keep up with the Symfony dependencies. I'm totally happy to coordinate in lockstep with Drupal core, so that shouldn't be a barrier. I think we've been doing that; you can still use Drush with Drupal 7. The details are to be determined, like whether there's a problem with Drush going to a major version before Drupal does. I can't think of a reason it would be, but the core maintainers have lots of experience, so I'd rely on that.

[00:38:57] (Discussion from the audience.) It's great to have a first-class, out-of-the-box command-line tool: this is what you use and here's how, versus "there's this thing you should use but it's not really part of the product." It should always be part of the product. Someone created the issue and posted the URL in the chat; someone should write a better issue summary, and everyone interested please follow the issue.

[00:41:06] (Discussion from the audience.) To get a change done to the recommended project, it's pretty straightforward. In core it's the recommended project, which is a subtree split from core, so it's a composer.json in core that gets subtree-split out into GitHub. So it's just a patch filed against core, though soon some projects have merge requests enabled on Drupal.org, so we'll have merge requests for core once everyone hammers out the workflow, and it might not even be a patch. But I wouldn't file it against core; for the whole composer initiative, the ideas queue is where it creates support from the entire core infrastructure of people saying yes, we want to do this. Then the product managers and release managers weigh in, and it's less like shouting into a well. That makes sense; we'll move that issue.

[00:42:57] (Question from the audience.) Have you run it by Dries or any of the other folks? You probably meet every day, right? Nobody meets with Dries every day. In an earlier iteration of this proposal I ran it by Alex Pott and Jess at DrupalCon Europe in Amsterdam. It still wasn't quite this proposal, and there were some questions Alex Pott had that I think are resolved by it. He gave a plus-one with reservations when I talked to him, and I don't know if that's his current position, so you'd have to ask him. I don't know how all the core maintainers will feel; we'll find out once we ask them to look at the issue.

[00:44:30] It sounds like we got our discussion done. Thank you so much for joining, and I'll see you in the issue queue. I've made you the reporter of the issue, because I don't think it should be under my name, if you don't object. I'll go take a look and edit it, and everyone's welcome to keep making it better.

[00:45:10] (Question from the audience.) Someone mentioned Robo in the chat; what is that? Robo is a command runner for PHP. It shares a lot of the same underlying packages that Drush does, but it's less Drupal-specific; it's for anyone in the PHP community who needs to run scripts in an organized way.