Following Drupal Core Development: Is it possible to understand every added change?

[00:00:01] Okay, welcome everybody to the session. We're going to explore a bit how to follow Drupal core development. A bit about me: I'm Marco Villegas, you can find me as marvil07 on drupal.org. I work with the fine team at Tag1 Consulting. I've been in Drupal for a while, you can see the number of years on my profile that I'm not willing to say. I was part of the Git team for the migration to Git, I like to work on migrations, and recently I was helping with single sign-on on drupal.org for the Drupal Association, and that went well.

[00:00:56] We'll talk about why we want to read Drupal core changes, how fast it is, and what resources are around that. Just in case you want it, this is a link to the presentation. There are many links I'll be adding, so I can come back to this at the end, but it's a resource.

[00:01:30] The first thing I want to do is thank the Drupal development team and the community at large, because none of this is possible if Drupal isn't developed and doesn't maintain itself. So, thanks a lot to the maintainers and everyone in the community who works on the software. Second, even if this presentation is highly focused on code, code is not all the contributions to the Drupal ecosystem, but that's what this presentation is focusing on.

[00:02:13] So, why do we want to read changes? On one side there's the technical aspect. You always want to know early about new features, and that's something you can do directly by reading the changes. You can also think about possible impact: if you're maintaining a project over time, you can foresee if a module will no longer be part of Drupal core, and know that before it happens in a release. There's also the learning opportunity, to see what's happening there and what new APIs are being added.

[00:03:00] On the other side, there's participation. You can see who's doing the development on Drupal core and which organizations are actively pushing it. Those are all great reasons. And I may be biased, but it's also just fun to read the logs and know what's happening.

[00:03:25] In parallel to the presentation I'll do a couple of interludes, because I'm not the first to try this. The first attempt I can find was in 2006, by Moshe Weitzman. I'm probably pronouncing his last name wrong, but Moshe created a Drupal website that contained a curated set of CVS commits. At that point we were using CVS, and it looked like that: you could see a given commit in both Drupal and contributed modules. So here's one commit from a module and one from Drupal core itself.

[00:04:07] Later this was also adopted by Greg Knaddison in 2017, to continue the initiative under a different domain, but the same idea: a curated set of commits from Drupal core, and also a part for important module releases happening at the time. All these captures are, as you can see, from the Wayback Machine at archive.org, but I could find one that actually had the styling from that team in Drupal.

[00:04:52] Now let's talk about how fast Drupal is really being developed. First, I want to look at what Drupal was when it started: only about 2,500 files, and mostly PHP. If we jump to now, we have 18,000 files, 54% of them PHP, and many other formats. It's a really big code base, so in one go it's really hard to try to read all of Drupal. There's software still in that category, but Drupal is definitely not.

[00:05:39] Another way to look at this is lines of code. In the original commit we had around 4,000 lines of PHP; now it's more like 3 million, and again mostly PHP. But at least PHP is what most of us can read. Another way is to see how fast it's moving, commit by commit. This is a graph over time of all the Drupal commits since the repository started, and it's around 3,600 commits per year right now.

[00:06:34] We can also look at commits per version. This is based on trying to subtract the commits of different versions from each other, and it's around 1,500 commits per version per year, except for Drupal 8, that big peak in the middle. That sounds like a lot. But if we look at recent commits, the last three versions, 9, 10, and 11, over the last few years, we arrive at more than 1,200 commits per version per year, and that's around three or four commits a day. Remember that number.

[00:07:34] Another thing to look at is how many changes come in those commits. Let's stay with the latest figure, 11. On 11 since it started, we have on average around 77 changed files per day in those three or four commits. We can also look at changed lines per day, and that's around 900 lines added per day. So if you want to look at the changes, you need to follow the current development version, right now 11, and it's around three or four commits and around 900 lines of code changes, and it's mostly PHP still.

[00:08:35] Let's talk about resources around the development. We've mainly been talking about the code base, which is managed in git, so you can see it in GitLab, and the interface is just the set of commits grouped per day. We also have release notes for every release. This is just one example from 11, where you'll find information about the important changes on that version, with maintainers trying to put the most useful and summarized information there.

[00:09:22] Those aren't the only sources. There are also change records, a way to capture changes, especially API changes or things that affect any part of the code base. This is an example from BlockContentTestBase, which changed how one of its parameters should behave. But our community mainly works on issues. Every issue has a lot of metadata and information, so if you really want to know what's happening around a change, the issue referenced in the commit message is the best resource to understand what happened, why decisions were taken, and to get more in-depth knowledge. You'll also see interrelations with other issues, so you get a big picture, and you'll see the credits.

[00:10:24] One example would be this one from a few days ago: you'll see the issue number, and then you can go to drupal.org, find that issue, and find a lot of metadata that helps you understand what's happening.

[00:10:47] Let's take a second small break. In 2008 I found this software called code_swarm, a really special way to look at how things are being done. I posted about it on the Drupal forums and didn't really get any feedback, but I published a blog post about it the next year. It's something really special: you can see the whole history of development summarized as a video, where you can see the nicknames of the people behind the actual development. There's a lot of concept behind it, and I can come back to it at the end if we have time. This video is a bit old, but it starts with the development since the origin of Drupal.

[00:12:05] Naturally you want to make it more consistent, so I started working on an analyzer, a base of scripts to get something more interesting and consistent, instead of a one-off video. I started reading the git history and, with some custom PHP, converting it into a SQLite database, then creating several assets. For instance, the tag clouds looked like this: Drupal 7 contributors in different sizes depending on their amount of contribution. I think this one was based on the number of file changes over the whole Drupal 7 development.

[00:13:13] I also started with some graphs about how many commits per month we have. This is the comparison between 7 and 8, from the time this was created. And I collected this interesting one, the number of different contributors per month, which is also really interesting information we may want to look at.

[00:13:43] Now let's talk about how we can actually do this reading of Drupal core. On one hand you have the manual approach: you can always just do git fetch and git log and see what has changed. Maybe that's enough for a lot of people. I clearly remember Angie mentioning doing something like this every morning. But as I mentioned, there are many resources around, so you don't need to do that. You could start following release notes, expanding the information you're interested in and going to the relevant links. There are also change records you can expand on. And since most of the metadata about Drupal development is inside the issue, if something looks really interesting, the best idea is to go there and find out the history.

[00:14:59] The current effort I've been trying is corelogs, which is a different thing from the analyzer I mentioned. Now I've taken a really simplified approach: I'm reading the git history, but since we have the drupal.org API, I'm also retrieving information from the issues and adding some manual input. I'm actually typing things about what I think a commit does, along with the idea, and I generate several metadata files and then produce a journal, a manual markdown file that contains the main things that happened over that day. It's something I plan to continue maintaining. I'm just around two weeks in, but it's something I've tried in the past, and this time I plan to stick to it, which is why I want to simplify the approach.

[00:16:16] Let's start closing these ideas. Again, thank you very much to all Drupal contributors, because without that we don't have anything else to talk about. If you leave with one thought today, it's this one: you just want to read the active development branch, which is around three or four commits a day and around 900 lines of code you need to read, and it's mostly PHP that has been changed. And if you want, you can follow this set of corelogs I'm mentioning.

[00:17:19] Let me try to open that. So this is how it looks: a simple page generated based on a small shell script that creates one entry per day. This week has been really easy, because most maintainers have been really busy with DrupalCon, so you'll find several days with no commits. This one contains, oops, that's one that didn't contain any. Let's take this one, and you'll have a small overview and the set of changes, with the title of every commit along with a small comment about what I think it is.

[00:18:14] There's a small annex with the set of things I think may be relevant, including the organizations. This is based on drupal.org credit metadata, so it's basically who is working over this set of commits on this day, plus some small statistics I find interesting. In this case the shortest-running issue is 26 or 29 days, but I've seen days where the longest-running issue is nine years, so it changes over time. And if you're really interested, I also have an almost manually created script that produces an RSS feed, so you can follow it if you still use an RSS reader like I do.

[00:19:15] That's the main set of things I wanted to convey. Thank you very much for being here, and if you have any questions, this is a nice moment to ask them. This is the contribution opportunities we have during the week: Friday is the main contribution day.

[00:19:52] (Question from the audience.) First of all, really awesome to see some analysis on the commits and the things we're doing for core. Something that might be interesting: a while back, when I started working on JSON:API, I did a database with all related issues to find where the hotspots are and what's good to work on. A lot of issues unblock other issues or are part of a larger meta-issue. It could be really interesting to see, on what base are we working, what's the most parent issue of the things being done, to see what goals we're actually working toward. That could add a lot of value to the release log for you. Are you planning to extend this and look for opportunities to make it even more interesting, and where can we help? For better or worse, I keep coming back to this topic over the years, as you've seen, so I'll probably be around doing things like that. That's an interesting idea, and patches are welcome. All the code base is public; if you go to corelogs there's a link with the project, another Drupal project, and all the code I'm using and the actual source of the analysis is public, so it's all there, and I'm happy to expand the scripts to do more.