Following Drupal Core Development: Is it Possible to Understand Every Added Change?

[00:00:00] Welcome, everybody, and thanks for coming to the presentation. I'm going to talk about following Drupal core development, and whether it is really possible to read every commit that happens in Drupal core. I'll try to take you along a journey I've been on for a few years.

[00:00:27] Let me start with a quick introduction. My name is Marco Villegas. You can find me on drupal.org as marvil07. I've been around for a while now, almost eighteen years this year. I was part of the Migrate team a while ago, along with many other small things you can find on my profile. Right now I'm working with the great team at Tag1 Consulting.

[00:01:08] Here's what we're going to go over. I'll talk about four topics, with a couple of interludes about things I find really interesting around the subject. First, why is it a good idea to read changes? What resources do we have around them? How fast is Drupal core really moving? And some experimentation I've been doing myself.

[00:01:52] I'll give you a minute in case you want to follow along on your own device. This is the URL of the presentation. It's just an HTML page, so you can follow the links, especially the ones inside the slides.

[00:02:22] The first thing I want to say is thank you to the Drupal core development team, because this is a meta talk about the work they do. Even though I contribute from time to time, Drupal core is a huge group effort involving many parties. A small disclaimer, too: I'll be talking about core code contributions, but there are more kinds of contribution than code. This presentation is focused on the code side.

[00:03:01] So let's start with why it's a good idea to try to follow what Drupal core is doing. I'll focus on three fronts. The first is technical. You can see directly which new features are being added to core. If you maintain a large amount of custom code for different clients, that lets you react accordingly, whether a feature is being deprecated, moved into contrib, or something new arrives that lets you do things in a better way.

[00:03:46] The second is learning. As new APIs are added internally in core, following along is great exposure to them. Drupal consumes a lot of its own internal APIs, and when you see them used in new changes, you learn how to use them from your contributed modules or your custom code.

[00:04:42] The third is participation. If you follow the core logs and commits, you get information about who is doing the work on a given issue. Now that we have commit credits on drupal.org, you can also see which companies and organizations are behind the individuals doing that work. That's useful in a lot of ways. If you want to improve something specific in core, you know who to contact for that part of the problem.

[00:05:34] And I may be biased, but it's also just fun to read the commit logs of a really nice project. There are many projects with genuinely interesting commits.

[00:05:47] Let me start the first interlude, because I want to point out that I'm not the first to try this. Drupal is more than twenty years old now, and many people have done this. The first attempt I know of was Moshe Weitzman, who created drupaldig.com. This was back when we were using CVS, which had only two repositories, core and contributions. He would select a set of commits and say, this is interesting. It was a form of curation of what you should focus on.

[00:06:52] That project lasted, I think, only 2007, and died the next year. But Greg Knaddison, who I saw at this conference after a while, started a Drupal dashboard domain with very similar code and the same idea. His also added the idea of important module releases, so not only commits but releases across core and contrib that might be interesting to look at. All these snapshots are from archive.org, if you don't recognize the banner. So I wanted to say I'm not the first; many people have been doing things like this.

[00:08:28] Let's talk about what resources we have for following core. The main source is git. You can just read the changes that have been happening. Now that we use GitLab, that URL shows you the list of commits aggregated by date, and you can click on every commit to see exactly what happened. That's the simplest possible approach, because all you need is a web browser.

[00:09:13] But that's not the only resource. We also have release notes. The release notes for the first Drupal 11 release contain quite useful information. The core maintainers make a big effort to condense what happens in every release. Sometimes the set of changes is so big that it can't all fit and some information has to be excluded, but there's a whole section on update information you may need to be aware of.

[00:09:55] There are also change records. Every time you make a backwards-incompatible change or an API change, you create one. This isn't only for core, but the change records list gives you, for every issue in core that changed an API, a record of what changed. The core team does a really good job here, because an issue isn't closed until its change record is added.

[00:10:41] One example is a change to BlockContentTestBase, where the first argument of a method changed from a string into an array. Usually both are valid for a while, and then the original is deprecated. So if you were extending that test, it's important to be aware of the change.

[00:11:18] Drupal also has a lot of issues, and the issue is the first source of information about changes, because it contains the discussion of how a feature was added. It has more in-depth description and a lot of metadata: relationships to other issues, parent issues, related ones, and the credits for who worked on it. A commit message contains the issue number, the participants, and a title, but the issue itself contains far more context about what happened.

[00:12:42] Let me go to a second short interlude. I've been thinking about this topic for a while. The first time I published something was in 2008. A friend from a different open source project showed me software called code_swarm, and I posted about it on the drupal.org forums, which were actively used at the time.

[00:13:29] It's better to just see it, so let me try to play it. I'll speed it up a bit. What this tries to do is condense information about what's happening in a given source code. It takes the commit logs as input and creates a video where the gravity is wherever the most influence is. At the center here you can see Dries and webchick doing the core commits. This is 2009; the date is in the bottom right. There are also bars in the background showing the number of changes to a given file.

[00:14:55] This is one way to summarize the whole history of Drupal. Someone who has been really involved can recognize moments: oh, this is when Drupal 7 started, and when Drupal 8 started you see a constellation of things happening at once. It's really fun, and something I liked a lot at the time.

[00:15:23] Then in 2010 I started a small project to parse things out of the commits of Drupal core. Those are links to different blog posts I've made in the past, and the last project, the Drupal contribution analyzer, contains the actual source code of what I was doing.

[00:16:00] Let me summarize what that project did. I parsed the git history to extract information, because unlike other projects that put the author in the commit author field, we have it in the commit message, so I had to extract it from there. I created a small SQLite database, then generated graphs in HTML with a bit of JavaScript, some stats files, and even some tag clouds and a code_swarm video from the XML it required.

[00:16:57] This is an example tag cloud from the whole Drupal 7 history, excluding the core committers Angie and Dries, since they'd be too big compared to the rest. It gives you a close understanding of how much different people contributed. This is not just the top, it's the whole set, so it's a little blurrier because a lot more people are in it.

[00:17:38] I also added graphs showing the number of commits per month across different branches, in this case 7 and 8. It's a bit old, which is why it's only 7 and 8, but you can clearly see 2011 was the point when Drupal 8 development started. The contributors-per-month graph is a good way to track community health, because the number of contributors is a good indicator of how many people can help. If it's going down, we need to do more to reach out.

[00:18:30] So that was an attempt to summarize what's happening in development in several ways, giving you different inputs on what's going on.

[00:18:51] Now let me get you inside Drupal core and how it has changed over time. I'll analyze what's inside core and how fast it's changing, so we can see how fast we'd need to read in order to follow it.

[00:19:18] To get a good perspective, one interesting exercise is to see what was inside Drupal core when it started, when it was first published by Dries. This specific commit is the second in the whole history, and it contained the full set of initial files. That commit was only about two thousand files, which is a really small set. Compare that with today: we now have almost twenty thousand files in core. At the start it was mostly PHP, and it's still mostly PHP, but now we also have a lot more Twig, HTML, YAML, and JavaScript.

[00:20:19] By the way, all the information I'll show you, when I say 11.x, is as of March, which is the exact commit I mentioned. That's the cutoff point for this presentation. Another way to see the size of core is lines of code. In the original commit there were around four thousand lines, mainly PHP. Right now we have around three million lines, still mostly PHP, but with a lot more of the others in proportion.

[00:21:18] Now that we have an idea of the size of core, we can think about how fast commits are being added. I made a graph summarizing this by year. Across Drupal history it's around 3,600 commits per year. You can see the bump in 2011, which is Drupal 8. It goes up and down, but it's around 3,600 commits a year. The last year has just started, which is why it's low; we're only in March.

[00:22:13] Another way to look at it is commits per version, because we're not reading all commits across all branches; we likely want to focus on a few, or maybe one. This graph is around 1,500 commits per version per year, and that big peak in the middle is again Drupal 8.

[00:22:56] But we may not want to focus on the whole history, so let's focus on recent commits per version. Here are the last three branches, 9, 10, and 11, at around 1,200 commits per version per year. Now we have a more tangible number: about three or four commits per day on the most recent active version of core. That sounds doable, right?

[00:23:38] Let's examine what that means. If you think about the number of files changed in those commits, maybe they're changing a thousand files at a time and that would be hard to read. But that's not the case. For all time it's around 77 files a day, and for the last four years around 130, for other reasons. For 11, the one we're focusing on, it's around 78 files changed per day, aggregated per day rather than per commit.

[00:24:31] Another way to look at it is lines rather than files: around 800 or 900 lines changed, and around a thousand added, on a given day. Those are the changes you'd need to read. Even if that doesn't sound small, it isn't that much, because most of the changes are in parallel in different parts of Drupal that do the same thing. The core team is really good at keeping a given change focused on one thing.

[00:25:22] To summarize the previous slides: you want to focus on the latest stable branch, which is the latest branch in development, currently 11. You'll need to read around 900 to 1,000 lines added or deleted over three or four commits a day. That's probably the effort needed, and based on the data it's mostly PHP you'll be reading. So, do you think that's doable?

[00:26:04] I'd like to start talking about some experiments. Just before that, I want to focus on the simplest possible path. I remember Angie Byron and webchick mentioning that the first thing you do every day is go to the computer, do a git fetch, and see what has happened in Drupal core over the last day. That's the simplest way to see what's happening, because you literally have all the data you need.

[00:27:01] But as I mentioned, there are many resources around those changes. At some point it was decided that core commits would just reference an issue number rather than all the contextual information. The context of why a decision was made lives in the issue, along with the change records and related issues.

[00:27:36] So I was thinking there should be a better way to do this, because I had to go to different places. A couple of years ago I started another project. All these projects are on drupal.org, so you'll find the links there; corelogs is one. This time I was intentionally trying to be really focused. I didn't want to start doing many things, so I discarded the tag clouds and the code_swarm videos and focused on information that might be highly valuable.

[00:28:39] The scripts in that repository consume the data inside the git commit history and create a set of plain files with different information. This is inspired by the git model: if you look inside a git commit you see different objects in different directories, so it adopts that architecture. But I'm not only trusting the git history. I use the issue numbers in the commit messages to fetch more information about each commit through the drupal.org API, using a small client.

[00:29:43] I use that to see what changed, who contributed, and which organizations are around a specific change. I also rely on manual input, because sometimes things are better explained directly, so I write an explanation of a given commit as a separate piece of text that can be reused. At the end I produce a journal for every day, a summary of what happened inside Drupal core that day.

[00:30:32] It's important to mention that all this information is after the fact; it's already in core. Many things are in development at any point, and sometimes they take a week and sometimes ten years, but this is about what's already landed. So let me show you some of that output.

[00:31:19] This is the project I'm mentioning, corelogs. It has a link, and the code is available there. It's a set of directories that contain the data, one for scripts, and some extra information. Let's focus on something more tangible.

[00:31:53] This is the page generated with GitLab Pages, which GitLab provides and the Drupal GitLab maintains, so we can use it. I use markdown files, then convert them with pandoc, which can convert many formats, to produce a set of HTML files.

[00:32:26] Here's a sample output for one day. The last one here is from March 24. Part of the manual input is this overview of what happened that day. Sometimes you're not interested in the whole set, just one part, so it also provides relevant links to everything that happened, including the commit message as the title, a link to the issue, and a link to the commit itself in GitLab, plus an explanation for every commit.

[00:33:20] Let's start with one. This is a simple link to the GitLab issue and the GitLab commit, so you can explore what happened in your browser. Sometimes changes are really small and not scary at all, so you can just read them. You can also see the information summarized in a more specific way if you need to.

[00:33:54] At the end I add some extra information I find interesting. For this day there were seven commits. At this point we're almost no longer using patches; there was one patch and six merge requests. Something really interesting: the shortest-running issue was one hour from start to close, which is great, because issues have historically not been that fast to close. The longest was three years.

[00:34:41] The summary shows that twelve people were involved that day, two of them volunteers, and seven organizations were among the contributors. You can see who is in which organization, because all of that is inside the commit credits on drupal.org; I'm just parsing it and putting it there. You can also see the maintainers accepting the changes into core and doing the commits that close the issues.

[00:35:28] I'm also adding parent issues. That was a request I got last time I talked about this, from a contributor, and it's really useful, because it shows what we're working on. Usually you have a parent and a lot of small issues with atomic changes needed to close that big meta-issue. I'm also adding tags, both the ones on drupal.org and a set I add manually, which helps categorize things over time.

[00:36:19] One thing I wanted to mention: the review queue initiative. I keep seeing it day after day, and they're great, so thank you, because they're really pushing things to actually close. That's the main explanation for that shortest-running issue. That initiative is really good at getting things done.

[00:37:02] This is minimal information about the actual git logs in a short format, so they can be inspected more if you want. I'm using UTC for the dates, because we're a global project, so the date needs to be the global date. That's an example of how this small set of scripts, plus some extra work to produce the metadata, can be useful. Even just for me, this is really useful, because I get to know in more detail what's happening in Drupal core.

[00:37:36] Last time I talked about this, I didn't mention that I'm really open to contributions. If someone wants to, the project is open; you just open a merge request in GitLab and I'll be glad to include your changes. It seems like we have some time, so maybe it's a good idea to do a small demo of how I actually produce this. But first, let's go over some closing words.

[00:38:48] The first thing I'd like to say is thank you to the Drupal contributors and the community at large, because this analysis of data about Drupal core development isn't possible without the development happening. I appreciate the effort of the whole community, especially the maintainers who go over the changes every day.

[00:39:18] For following core specifically, I think it's doable. You probably want to focus on the latest development branch, currently 11. That's three or four commits a day and around a thousand lines of added or deleted code, which sounds doable, and mostly PHP changes. And maybe you want to follow corelogs. I'm also producing a journal RSS feed, so if you still have an RSS reader, you can follow the changes there.

[00:40:14] In the few minutes we have, I'd like to do a small demo of how I do this, in case anyone is interested in contributing. Please also start thinking about any questions you have. Feel free to let me know, otherwise I'll start with the demo.

[00:40:45] (Question from the audience.) I do it for myself, in public, so it's all there. It's around half an hour, so it's manageable. I don't do it every day, and life happens, so sometimes you need to do other things, but it's a habit I have to close this gap. It's around half an hour to read three or four commits and understand what's happening. It can vary; sometimes there are really big commits.

[00:44:01] (Continuing an answer.) The main resources I find useful are technical: I see what new features are being added, so I can be aware of how I should do things differently, and if I see a new module going into core I can start poking at it. One really good use case, maybe the main one for people working in professional environments, is that sometimes you maintain custom code that isn't trivial, or contributed modules, and all of those are affected by what core does. Being aware of the changes is an excellent way to be prepared and make changes in advance. And I really like being aware of who is doing what, though I admit that's arbitrary.

[00:45:01] Let me move my terminal over, and make it a bit bigger. That's the project, a set of scripts; it's not that big. If you look at the scripts directory you'll find a couple of bash scripts, dcl and drupal.org. The main CLI I use, dcl, is less than 800 lines of bash, and drupal.org is just a client for the API, also reading with curl and shell script, so it's really simple.

[00:46:15] Let me run "missing journals" so you can see some output. Sometimes I have a hard time following myself, and this shows the entries I haven't created yet. Let's choose something recent. I was working on something, so let me put that away for now.

[00:47:17] What I do is use dcl, which is a wrapper. A lot of the lines in that shell script are documentation. You can provide a set of commands: draft journal, read day, which gets the information from drupal.org, and others like field commit, field day, and open day issues. I'm quite influenced by the git idea of porcelain and plumbing, high-level and low-level commands.

[00:48:30] There are examples of how to use it. If you want to start reading one commit, this processes what's inside that commit. The day-to-day is something like: read these dates, then field day, which is an interface to go over them. Let me run it so you can see it.

[00:49:35] Let me choose this date. Before I forget, this small wrapper passes another directory with the actual core commits, the source git repository, and this one with a browser is because I open a window with the issue number when I want the information. So, dcl read commit with the date. What's happening is it's getting information from git itself and from the drupal.org API. If the network is fine and there weren't many commits that day, this takes less than ten seconds. In the meantime, feel free to ask questions.

[00:50:48] (Answering a question.) As I mentioned in the diagram, I really wanted to make this easy, and I didn't want to deal with a database, so I'm doing it in plain files in the same repository. For instance, this is metadata I produce for a given commit: one file for the git committer, another called git numstat with the output of that command so we know how many changes happened over which files, another with the subject of the message, another with the related issue. Very few commits don't have issues, but they exist. And the approach, the issue contributors, all of this is generated by that script I left running.

[00:52:04] It has finished. If I do a git status, you can see a lot of files were added to the metadata. Let me add those. You can see similar output to what I showed before, and then I just add a commit with that.

[00:52:42] We were on this date, so the same way, we want to do field day. We probably won't finish this, because we don't want to spend half an hour on it, but what it does is open a window with the issue related to the first commit that day. I can see, oh, this is about JavaScript errors on Drupal.behaviors active links, and it also opens the diff of what changed. It's a change to the active links JavaScript file, and the change in the test, and that's it. It's not that much.

[00:53:35] Then I produce this window, the output of one of the commands in that dcl script, on the left: the title, the URL, the commit, the tags used, the people involved, the approach, and the numstat we created. On the right I have an "about" file, one of the files I fill manually. I try to explain what happened in this commit that's shown in the diff. Then I fill another one called extra tags, which adds tags to the journal output.

[00:54:33] After going over everyone, I create a draft journal, and it's markdown at the end. Let's open one so you don't have to take my word for it. The one I'm showing is March 24. This is the markdown I end up editing: a title, an overview of what I think summarizes the day, and details about each of the things I added in that "about" file, for each specific commit. That's basically what I've been trying to do, and it's the same summary you saw in the HTML page.

[00:56:01] Any more questions? I know we're approaching the end of the session.

[00:56:10] (Question about a recent example.) That's a good question. Recently I've not been that active doing this, so the last example I find really memorable was the day something related to external logging was added to Drupal core. That was when the API related to performance testing was added, along with the underlying vendor library. I was completely unaware it was happening, and it was during DrupalCon Barcelona. I wondered when this started, went to the issue, and realized there was a performance initiative doing this testing. That's how I started my first issue related to performance testing in Drupal core, because I felt drawn to it.

[00:57:36] (Question about automation.) I was expecting that question. I've thought about it, and I intentionally don't want to add any automated input for the explanations. I really want to do it manually. I know it's possible to automate, because summarizing is one of the best things these tools can do, but I find value in it: since I'm already doing a lot of scripting to collect data, the synthesis is the best way to provide value to anyone reading this. It's also for me, because if I do the synthesis I understand it better. So even though it's possible to automate, I still prefer to do it manually.

[00:58:33] (Closing.) I hope you liked the presentation. All the resources are available in the presentation itself, with links to every place, including the corelogs project on GitLab Pages. You're welcome to open pull requests with more of these if you're inclined. Thank you very much for being here.