Upgrading to Drupal 10 using the Migrate API

[00:00:00] Okay, it's 3 p.m. Thank you everybody for coming. This is "Upgrading to Drupal 10 using the Migrate API." My name is Mauricio Dinarte. The slide deck and everything I'm going to present today is already available, so you can go to the URL and get more content. My username is dinarcon, and that's also my email. My pronouns are he and him. I'm from Nicaragua. Nicaragua is very beautiful, very hot, and there are a lot of lakes and volcanoes. I'm a software engineer at Agaric, and I'm also an instructor at Understand Drupal, understanddrupal.com. I enjoy reading, traveling, and learning languages, both human and computer languages. If you speak English, Spanish, French, or Portuguese, feel free to talk to me; if you speak another language, I might not be able to reply.

[00:00:51] As I said, I collaborate with Agaric. We're a cooperative based in Boston, but we have people all over the place: myself in Nicaragua, partners in Germany, and here in the States in different cities. I'm very passionate about teaching. I've written a lot about Drupal migrations, and I plan to keep writing about other topics like site building, module development, and theme development on this website, understanddrupal.com. For the most part the content is in English at the moment, but I'm working on translating it to Spanish and French. I'd also like to thank Pantheon for helping me get here to Pittsburgh to be present at the conference through their Pantheon Heroes program.

[00:01:37] This is what we'll be covering today: triggering the upgrade process from the UI, from the command line, and how we can use the migrations that are automatically generated as a base for a custom migration. If you want to follow along, as I said, the slide deck is available, and there are also companion modules, because this session is sometimes presented as a full-day workshop. The code for the workshop and for setting up the whole project is open source and freely available, so you can get it from there. And by the end of the session I'd like you to provide some feedback; that's the feedback link.

[00:02:17] So let's get started. Before going too far, I want to point out that the Migrate API in Drupal core is an implementation of a design pattern called extract, transform, and load. This is not exclusive to Drupal, not exclusive to PHP. There's a lot of documentation; I recommend the book on the left, The Data Warehouse ETL Toolkit, and the one on the right is a book I wrote a few years ago about Drupal migration. If you don't know anything about migrations, with this book it's like one short article for 31 days, and you'll learn how to migrate content from different sources to different destination entities, plus general tips for working with migrations and debugging them.

[00:03:05] As I said, this is sometimes presented as a workshop and there are recordings available, so if you want to see a hands-on demo, those are the links. Today we're going to talk about the process and give some general recommendations, but if you actually want to see how to run a migration visually, that's where you need to go.

[00:03:28] So let's start by preparing for an upgrade. Before going any further: upgrading in this context is if you have a Drupal 6 or 7 site and you want to move it to Drupal 9 or Drupal 10. I recommend that you create an inventory of the modules used in the source and destination sites, audit the content and configuration in the source site, understand the assumptions the Migrate API is making, be aware of some issues and limitations, and know what the scope of the system is. We're going to go over these one by one.

[00:04:13] For the source site module inventory, I highly recommend you use the Upgrade Status module. You install it in Drupal 7, for example, and it checks all the modules enabled on the Drupal 7 site to see if they're available or if there's a replacement for Drupal 10, and you get different recommendations. In this screenshot the addressfield module was a module in Drupal 7 that didn't exist in Drupal 9; it was replaced with another one called Address, so you get the recommendation that this is no longer available but you can use this module instead. In the case of the Date module, part of it was incorporated into Drupal core but some elements were not, and you'll get a notice about that. And in the case of the Entity API, that's basically everything available in Drupal core, so you get a notice about that one too.

[00:05:12] One thing to be aware of is that the Upgrade Status module checks that a module is available; it doesn't mean it will have an automatic upgrade path. That's something different. Just be mindful that being available doesn't guarantee your data is going to be moved over for free or automatically.

[00:05:36] In addition to getting the list of modules, I also recommend that you audit content types, vocabularies, menus, because many times, I'd say 99 percent of the projects I've worked on, there is something to change, either improve or adopt new features of newer modules available in newer versions of Drupal, or they just have some legacy content they no longer need. It is very likely that by inspecting what you have today you are going to find out things that are no longer necessary or that you want to change.

[00:06:16] So from a configuration standpoint, but also from a content standpoint, do a basic review of what you already have. I have a template I use in some of the projects at understanddrupal.com. It's high level: this is the list of content types, this is the list of nodes per content type, what do I want to do, drop it, replace it, keep it. For different entity types I'll do this analysis. Something that's not necessarily an entity but is worth checking is roles, text formats, image styles, basically anything that triggers a warning like "oh, I might need this module." Document it in one place and have that as a source of truth to be used later.

[00:07:24] In terms of assumptions of the API itself, for the sake of this presentation, so I don't have to repeat a lot of version numbers, let's say we're migrating from Drupal 7 to Drupal 10. Your Drupal 7 site needs to be in the latest stable release, and Drupal 10 should be in the latest stable release. In Drupal 10 there should be no content or configuration; you basically install the site using the minimal installation profile and start from there. Then, if you want to get automatic migrations, you need to enable the modules on both the source side, Drupal 7, and the destination side, Drupal 10.

[00:08:07] Just be mindful that this is the assumption and the recommendation, but sometimes life is not perfect and you need to deviate from the norm. It is possible, but this is the base the Migrate API is going to use.

[00:08:31] In terms of scope, you're only going to be able to migrate content and configuration automatically. If you have custom modules, that's not covered by the Migrate API; you can use Upgrade Status or Upgrade Rector to help with upgrading custom code, but that's not the Migrate API's responsibility. And if you have a Drupal 7 theme, you basically need to recreate it from scratch, because the template used for writing themes in Drupal 7 is completely different in Drupal 10, and as far as I know there are no automatic translation tools.

[00:09:10] In terms of issues and limitations, this comes with a caveat: everything I say, by the next time I present, something has changed. One that changed for the better is views migrations. In the past it used to be "we don't support views migrations automatically, you need to recreate them all." I've worked on sites with literally hundreds of views, and that was very painful to do manually. Now we have the Views Migration module. It's, I'd say, 80 percent; it will get you 80 percent of the way there. Because views is so popular and there are so many plugins that might be available in Drupal 7 that don't have a counterpart in Drupal 10, you may not get everything perfect, but for the most part it does a really good job.

[00:10:04] In terms of filter formats, if there is one that's not recognized in the new site, by default it will return an empty string; the content will be present in the database, and you might need to do some cleanup after the migration, but the data is going to be there. If you used the PHP module in Drupal 6 or 7, that is no longer supported for many reasons including security ones, so we don't support that. And if you already have content in Drupal 10, it is possible to use part of what we'll be talking about today, but be mindful that if you're not careful you might be overwriting content and relationships. Let's say in my Drupal 10 site I create node one, authored by user 5, and then I run an upgrade procedure and on the source site node one was created by user 3; you'll lose who the author was. So be mindful of that. There are a lot of ways to work around it, but that's what the API assumes by default.

[00:11:20] Now, how do you actually perform the upgrade? There are at least two options: through the user interface, or through the command line. If you do it from the user interface, it works out of the box, you don't need any contributed module, but it is not customizable; you basically get a one-to-one copy of the previous site, as much as Drupal can do. There are things that might not be automatically migrated, but Drupal will do its best effort, and you cannot change much of the process itself. If you go with the command line route, you'll need Drush and a couple of contributed modules, but it is very customizable, and I think every migration I've worked on I end up doing on the command line because it's more flexible. I want to highlight that this allows for content model changes, so most of the time I need to do that.

[00:12:20] Let's see how we do it from the user interface. You install Drupal 10 using the minimal installation profile, and in Drupal 10 you enable Migrate, Migrate Drupal, and Migrate Drupal UI. Migrate is the core of the API, Migrate Drupal gives you a lot of source and process plugins specifically for moving data from a Drupal 6 or 7 site, and Migrate Drupal UI is the interface. You enable the modules that are going to be automatically migrated on both Drupal 7 and Drupal 10, go to your domain slash upgrade, and follow the wizard. In the wizard you'll be asked to enter the database credentials, and if you want to migrate files you can provide either a fully qualified domain name or a path on the same server to pull the files from.

[00:13:14] After providing the credentials, you'll be presented with a summary page: there are 31 modules that can be upgraded automatically and 11 that will not be. Take this screen with a grain of salt, because it's not always 100 percent accurate. For example, views: when I took this screenshot the module I mentioned before wasn't around, but according to this, views appeared twice, in both the modules that cannot be upgraded and the modules that could be. Another one, address field: if you can see, in Drupal 9 I have the Address module, which provides an automatic upgrade path from addressfield in Drupal 7, and it's enabled on both sides, yet it still appears as part of the modules that will not be upgraded automatically. So, a little confusing.

[00:14:22] Block in Drupal 7 was broken down into two modules in Drupal 10, so you need Block and Custom Block; if you only enable one, the block migration will not be complete. Some of these come with trial and error, or coming to sessions like this that give you a high-level overview. The point is, treat this page as a reference, and for the most part you might need to run the process at least twice: see how it goes the first time, see what worked and what didn't, and then do it again. After you review it, it's going to run the process again; it's not customizable, so it does as much as possible.

[00:15:03] Everything that happens is going to be logged in the database, so after the process is complete I recommend you go to the logs, review the messages, review the general configuration and content of the site. As I said, very likely you'll need to do it again, because the first time you find some things that worked or didn't, but after two or three runs you get a site upgraded with relatively little effort.

[00:15:44] Now, if you want to go the more flexible route, the command line: again, the last time I presented this slide was a bit different. The one thing to remember is that Drush, just like Drupal, is evolving, and the modules around Migrate are also evolving, and they don't necessarily evolve at the same pace, so at some point there might be incompatibilities between Drush versions and Migrate Plus, Migrate Tools, or Migrate Upgrade. As of today, the latest stable releases of all the modules needed and Drush itself are compatible, but sometimes that's not the case and you need to pin down specific versions.

[00:16:43] What do you do? You install Drupal using the minimal installation profile, enable the Migrate module, Migrate Drupal to get the source and process plugins, Migrate Plus, Migrate Tools, and Migrate Upgrade. You enable the Drupal 7 and Drupal 10 modules that are going to be automatically migrated, and this is where things start to change. Before, you had a form in the interface to provide the database credentials; when you do it from the command line, you actually modify your settings.php file and add a new database connection, basically the same information.

[00:17:30] Then, via the command line, you run the migrate upgrade command. In the database array, the first key, which says "legacy," is what you're going to use in the migrate upgrade command where it says legacy DB key. You can name it whatever you want, but in this case it's legacy. The important part of running this command is passing the configure-only flag. If you don't pass it, it produces exactly the same result as running it from the user interface, but we want to be able to customize the migration, so we pass the configure-only flag.

[00:18:17] When you run that, a lot of migrations are going to be generated for you, but where are they? The migrations are in Drupal's active configuration, and by default the active configuration is stored in the database. So how do you get the files to modify them? You need to export the configuration; you can do that using Drush, so you do a config export and all the migration files that were generated are exported to the default config directory. This is a trimmed-down version, but you can see core.extension is at the same level as the rest of my migration files.

[00:19:04] Now that you have the files, you again have multiple options to run them. You can run everything using the group flag, which produces the same result as the UI, or you can say I only want to run configuration migrations, or only content migrations, using the tag flag. But what I recommend is actually reviewing the migrations that were generated and cherry-picking only what you need, because, as I said, there's a high chance something is no longer needed or has to be changed.

[00:19:43] In this example I'm going to show how to convert nodes to paragraph entities in the new site. In this case I created a custom module called UD Drupal Upgrade, and in my config I created a config install folder and moved the migration I care about into it, so I can start modifying those files. This is an example of a content model change. The migration for the UD book content type used to be nodes in Drupal 7. In annotation number one you can see I'm using a source plugin for nodes; annotation number two is mapping the fields and base properties of nodes in Drupal 7 to fields in a paragraph in Drupal 10; and annotation number three says I want to migrate this into a paragraph. So this is a very simple example of how you can change from one entity to another. Another example would be changing nodes to user entities.

[00:21:11] By default, if you run the migrations today, you get what's called a node complete migration, meaning you get the latest revision, all the history, and all the translations in one go. If you don't care about the history or translations, or if you want more control over how those elements are migrated, you can fall back to what's called a node classic migration. You do that by updating settings.php, setting the migrate node migrate type to classic, and instead of getting one migration per content type that does everything, you get three: one for the primary active revision, one for the previous revisions, and one for the translations.

[00:22:07] Another thing to highlight: when you use the Migrate Upgrade module, by default the migrations you get are managed via configuration, as configuration entities, provided by the Migrate Plus module. It's not the only way to work with migrations, but it's the default behavior, which is why I show it. When I was cherry-picking the migrations, I had to put them in config install and treat them as configuration entities. If you make changes to the files, for the changes to be detected you need to import the configuration again, sync the configuration. There's a Drush config import partial command you can use to trigger those file changes. When you create migrations like this, you can run them either from the command line or from the user interface using the Migrate Tools module.

[00:23:08] Lately what I've been doing more is just using migration plugins. The two key differences are that instead of creating a config install directory, you create a migrations directory within your custom module, and the file pattern is a bit different, but otherwise they work the same way. The reason I like this approach is that if I want to make changes, I only need to clear the cache, which is faster than importing configuration, because sometimes you're working on the configuration of the site at the same time as the migration, and if you're not careful you might overwrite your own work. You need to run this migration from the command line.

[00:24:03] This is something, well, not new, but something I want to share. Two hours ago I released a new contributed module related to migrations, for something I've found a lot in the projects I work on. When making content model changes, many times you want to drop a field. To give a complete example, you were using Radioactivity in Drupal 7 and you don't want it in Drupal 10 for whatever reason. When you run the automatic migration, it tries to upgrade everything, but when it gets to that field it breaks, because it doesn't find a suitable Drupal 10 alternative.

[00:24:54] So many times this is the type of error you'll get and need to manually review. Because I find this error in every project, I created a module and made it available to the community. With it you can skip fields based on different criteria: by entity type, say I don't want to migrate any field attached to nodes in Drupal 7; by bundle, I don't want any field attached to the article content type; by specific field name; or by field type, like video or Radioactivity. This is probably the most common error I find, so to make it easier for myself and share it, I released this module earlier today, and I invite you to try it out and provide feedback.

[00:25:53] Now some general tips and recommendations. If there is one thing I'd like you to take home, it's this, from Lea Verou, author of CSS Secrets: understanding the process of finding a solution is far more valuable than the solution itself. I bring this up because I used to provide a lot of support in the migration Slack channel, and many times people go there and ask very specific questions, but they just want a snippet of code to copy-paste. Sometimes we can provide those snippets and they work, but sometimes that's not the case, and spending time learning how the Migrate API works, outside of an upgrade project, will help you a lot when you actually need to perform an upgrade.

[00:26:51] The book I referred to before, 31 Days of Migrations, you can read online for free, and if you don't know anything about the Migrate API, that could be the start. When you face more complicated challenges as part of the upgrade, you'll be able to ask more specific questions or understand the recommendations we provide better. In general, it's just a good idea to understand the tools you're using.

[00:27:27] Going back to tips: be mindful of the concept of Drupal entities, and particularly what is a content entity and what is a configuration entity. For example, Drupal comes with two content types by default; the content types belong to the node type entity, and there are no nodes created, which would be content. So as long as you understand the difference between configuration and content, it's going to be very helpful.

[00:28:03] One thing to note: 90 percent of the projects I've worked on over the past five years doing migrations pretty much full time only do content migrations. The reason is they want to use a completely different content model, or they just want to benefit from modules that weren't available before, and they want to start from scratch. Normally they build the site from scratch, just the structure, the shell of the site, and once that's created they move over the content. So if you understand the difference between content and configuration, you'll be able to more easily cherry-pick the migrations to select only the ones related to content and make adjustments as needed.

[00:29:05] In terms of content entities, it's good to know the different entity properties, also called base field definitions. For example, nodes have a node ID, a vid, the langcode, the type, the status, uid, title, when it was created, and so on. It's also good to know if they're fieldable or not and if they have bundles or not. For example, users can have fields but the user entity doesn't have bundles; files cannot have fields nor bundles; media can have both. This is useful because there will be different entities connected to one another, and if you understand how they're related, you'll be able to make content model changes more easily. For example, I have nodes connected to files directly, but let's say in my Drupal 10 site I want to use the media suite of modules, so I want a media entity in between: from node I make a relationship to media, and from media to files. As long as you understand the in-between entities that need to be created, you'll be able to do it more easily.

[00:30:35] Other general recommendations: measure twice, cut once. The API is not only for upgrading from Drupal 6 or 7; there's a bigger use case for it, so if you spend time learning the core of the API, it's going to be very useful, and you have the book I talked about. If you have the time and technical knowledge, another very useful thing is understanding how the modules in Drupal 7 and Drupal 10 store data, because ultimately what you're doing is moving content and data from one database to another. If you know this field type has three columns and that's the data I need to move over, that's useful when you need very custom migrations, for example if you have fields you want to break down into multiple ones or things you want to combine into one.

[00:31:48] I have another presentation where I give more concrete examples of how to look at the data model; there's a website called drupal.tv, and if you look for dinarcon or Mauricio Dinarte you'll find recordings of those more technical sessions. The one caveat is that if you're going to be creating very custom migrations, be mindful of revisions and translations, because those are usually stored in separate tables or separate columns. Also, create custom plugins as needed; source and process plugins are the ones you'll create most often.

[00:32:36] As I said, it's okay to deviate from the norm. The Migrate API has assumptions and expects you to work in one specific way, but real life is different sometimes and you cannot abide by the rules all the time. It is possible to migrate a site that already has content and configuration, and most of the projects I work on are like that; the only thing to be mindful of is entity IDs, to avoid collisions. There are different strategies for that. Another thing: I've worked on projects that are 9, 12, or 15 months long, with a very long lifespan, and it would be impractical to build everything and only start the migration at the very end. So it's possible to have site builders and developers build part of the site, then someone working on migrations creates the migration for what's already built, and so on.

[00:33:54] The only thing is, I've worked on projects where something was changed three months later, and you need to be very mindful about the migrations. If you change something but don't change the migration, things break, so you can have test suites and CI as part of your process to run a partial migration, and if something fails you get an error and know there might be a configuration change that broke an already-implemented migration. Just to give an example, the Migrate API can read from multiple sources. Two years ago I worked on a Drupal project upgrading from Drupal 6; it was long-running, and they needed to do testing with the content before launching. For that project we had three sources for the final site: Drupal 6, which they were using until pretty much the last minute; a Drupal environment where they were entering content for practice, part of which was supposed to go live at the very end; and CSV files. So we were getting content from three different sources into one final site, and in the end it worked out pretty well.

[00:35:19] More tips: start from an existing migration. As I said, this presentation is really a full-day workshop, and there are real and complete examples of how to perform a migration, so if you have the time, look at the resources, set up the local environment, run the migrations, and if you have something that works you can start tweaking that to practice. Read the official documentation; the Migrate API is, in my opinion, one of the best-documented APIs in the whole of Drupal. The one that lists the process plugins in Drupal core is very well documented; it's one of my bookmarks that I visit every day.

[00:36:13] The files you create use YAML, and YAML is very sensitive to whitespace, so an extra or missing whitespace can break the file, and it's not that your migration is wrong, it's the file syntax, and it will produce a fatal error. So be mindful of how you write the files. If possible, divide the migration work into small chunks. Sometimes I do one field at a time; I just want to make sure this field is working before I move to the next. Depending on the complexity, sometimes I focus on migrating every field of one type, and after that type is complete I go to the next. This comes with practice; find something that works for you, but you don't want to be in a position where you migrate the whole site, something is breaking, and you don't know what, because there were no stop points to use as a reference.

[00:37:19] Ultimately the migration definition files are text, in general format, so commit to your repository often; this is one way you can recover if you hit something that doesn't work, you can go back in git history to a state you knew was working. Another thing: don't force the API to do all the work. Sometimes it's easier to do some cleanup either in Drupal 7 before you start or in Drupal 10 after the upgrade is complete. You can come up with very clever, complicated process pipelines that give you a perfect migration in one go, but then you spend three days versus spending one hour on a manual content update, so try to balance the options.

[00:38:22] Analyze the errors and learn how to debug them; this is very important. Debugging migrations is a very useful skill. The reason I created the module is because I was spending too much time debugging errors about fields, so I said, that's enough, I'm going to create a module to automate this. But there are other errors I find only once, and as long as I can pinpoint the cause, I can provide a fix or see if there's a patch available, and the rest of the migration works fine. As I said, the migrate community is very active in the migrate Slack channel; Benji is one of the maintainers and provides a lot of support. Seek help from the community; the maintainers of the Migrate API are distributed across the globe, so you can almost ask 24/7 and get an answer from a maintainer or a volunteer.

[00:39:28] Some examples of content model changes I've done over the years, just for reference: if you were using files in Drupal 7 and want to use media entities in Drupal 10, there are modules that help. Maybe you had the Location module in Drupal 7 and want to use the Address field in Drupal 9. You had a blob of text in the body field and want to put that into Layout Builder. You were using Organic Groups before and want to use the Group module now. If you have inline resources in your body field that you want to convert to media entities. If you want to convert nodes to a different entity type like users, groups, or paragraphs. If there are fields or full entities that need to be renamed or dropped, which is part of what the module does. And if you have plain text you want to move into structured data. I remember a project a few years ago where they were storing width, height, and depth as a plain text field, and now they're using specific field types for that, so we used regular expressions to convert what they had into the new field types.

[00:41:07] That is the end of the presentation, but maybe it's the start of your own journey. I'd like to thank all the maintainers of the Migrate API, past and present; here in the room we have Benji, and everybody who has contributed to the API has done awesome work. I was talking to someone before the presentation and said the Migrate API is very, very stable; I could have given this session two years ago and probably only two or three slides would have been different. So it's a solid piece of work, and over time it gets better, because there are new modules that we identify a need for and provide, but the API itself is quite solid. Thank you. I'll be taking questions, and thanks for attending.

[00:42:04] Are there any questions?

[00:42:13] (Question from the audience about images in HTML with missing alt text.) So for the recording I'll repeat the question: if I have any recommendations for massaging HTML data, and the concrete example was images in the body field with a missing alt attribute. For images specifically there are two modules, one called Migrate Media Handler and the other called Media Migration. They're both about moving Drupal 7 files into Drupal 10 media, but they provide a lot of extra functionality to help with that. If you want to process HTML generally speaking, the migrate modules have plugins for manipulating the DOM, so you can use them directly.

[00:43:44] (Question from the audience about entity IDs.) The question is, in terms of entity IDs, whether it's possible to know the ID that's going to be used ahead of running the migration, and the answer is no. Part of the ETL, the load step, is the one saving the node or user or entity itself, and until it's saved you cannot get the number. One thing is that the Migrate API allows you to specify the ID you want to use, and that's actually the default behavior: when you run an upgrade, it fetches the same ID from Drupal 7 and passes that as the ID in the new site, and if in Drupal 10 you don't have any content, it's recommended to keep it like that. The alternative is to modify the generated migration file, remove the mapping for the ID, and then any related migration will need to use the migrate lookup process plugin to map the old ID to the new one. But in advance you cannot know the ID that will be used.

[00:45:44] (Question from the audience about example code.) The question is whether there's example code for migrations. Yes. The Migrate Plus module has a lot of examples within it. The book I wrote, 31 Days of Migrations, comes with a GitHub repository with 20 or so examples. If you go to the slides I pointed to the full workshop, which has the example of a full upgrade. If you mean a centralized place with all the examples, I don't think anybody has compiled them into one resource, but Migrate Tools, the 31 Days book, and the workshop all have a lot of examples; at least those are the ones I can point out off the top of my head.

[00:47:19] (Question from the audience about reading the variable table.) Let me see if I get the question right: you want a way to read data from the variable table in Drupal 7 and create multiple configuration entities in Drupal 10. Yes, that is possible, and it's actually heavily used as part of the automatic upgrade process. There are source plugins to read from the variable table in Drupal 7, and depending on the destination process you use in your custom module files, you can create different configuration entities in Drupal 10. I can't give you a concrete example, but if you run this process, reading from the variables table and creating configuration entities in Drupal 10 is supported out of the box.

[00:48:28] Are there any other questions? No? Okay, well, thank you very much for coming.