Cloud Native Computing Foundation BackstageCon NA 2022, 26 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: The Evolution of Backstage Backends - Patrik Oldsberg & Johan Haals, Spotify

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

The Evolution of Backstage Backends - Patrik Oldsberg & Johan Haals, Spotify

Join the maintainers of Backstage on a deep dive into Backstage backend's past, present, and future. In particular it will explore the evolution that was first presented in RFC #11611, which aims to improve many aspects of developing and deploying Backstage backends. The talk will explore the existing system and what the maintainer team has learnt from it, and then move on to introduce the evolved system and how it aims to improve the lives of anyone that touches Backstage. As a by-product of these discussions, listeners may also learn a thing or two about building scalable augmentable systems and TypeScript API design.

A

Did we? Oh okay, so hello, my name is Patrick on GitHub. You know me as rugby I'm, one of the maintainers of the backstage project and.

B

For like three years, um nearly four actually working on a lot of Backstage yeah and we.

A

Started the same day, yes, foreign So, today we're going to talk about how we are evolving, the backstage back-end system. So we're going to look at what it looks like today for building back-ends we're going to look at where we're headed and why we're headed in that direction and we'll also dive a bit.

A

That's better. That is much nicer, we'll also dive a bit deeper into how the new system works and and with a bit of a comparison.

A

So first I want to give you a bit of history of the current backend system and how it came about. So when open sourcing backstage, we had a pretty well built out plugin system for the front end, but on the back end, there really wasn't much at all for for plugins, and we had also taken the decision to rewrite all of our backend systems into node.js. So we basically started from a completely clean slate and we started very simple just with a pattern for how to set up an HTTP router to serve the backend apis.

A

Then, over time we added more plugins things grew. We added more utility services for things like logging and configuration database access and as back-end plugins got even more elaborate. We started seeing extension patterns for plugins, so you can now add things like entity providers in your catalog or custom actions in the scaffolder.

A

Now all of these new things grew out. While we maintain this we're kind of frantically trying to keep things a bit consistent across the project. So that's led us to where we are today, where quite powerful tools for building elaborate backend plugins.

A

So what doesn't backend plugin look like today at the heart width is the create router, every plugin exports, this function or something like it, and it's essentially a anything that gives you or a plug-in is anything that gives you an HTTP router to serve external traffic and then, through the options for the create router function, the plugin can accept any kind of options or dependencies that it needs to get started.

A

Now this design is extremely flexible, because it's it's just a function. You can do anything with a function really, but with this flexibility, there's a lot of cost. So when you have something that, is this flexible, it's very hard to provide any utility.

A

So from from the Frameworks point of view, we can't really hook in and help out in any way, because there's no points for us to do that.

A

So, let's zoom out and see how this create router method is used.

A

So the this code handles the setup of the plugin right, where you supply your dependencies from the environment and any other extra options and it lives in a separate file in each back-end installation and to be clear, there's a copy of this file in every single backstage instance that uses this plugin. So so this file is just copied over and over again.

A

Well, at least it's pretty simple right. So what's the problem, it's kind of what we thought in the beginning, too,.

B

So yeah as a patch is leading on to here, it does bring a couple of headaches.

B

So let's look at the installation of the an existing back-end plugin today, so these instructions seem to be very plug-in specific and in most cases, they're always available in the readme I'm sure this looks pretty familiar to people today, as you might be able to see from the instructions we first need to install the package from npm. Then we need to create a file which is named after the plugin and then import that into the back end into the backend package and then wire it all together in the HTTP router.

B

Now we're about evolving an existing plugin. So let's say I'm a developer and I want to add Access Control to my plugin. Now, luckily, we've got a permission service. That's super simple!

B

So to use it I need to update my plugin to accept it as an option to the create router now requiring permissions is a breaking change and that line that you can see on the slide needs to be added for every installation of this plugin. This will continue to happen for every capability. We add that requires a new service we passed from the outside.

B

Well, plugging authors ended up. Doing is basically just making this optional now that's kind of problematic because it means we need to deal with optional code from like as a producer of a plug-in as a like as a developer and also as a client, that's going to use the plugin itself.

B

um So let's look at the catalog installation, so this is code that you will recognize I think today, because this is basically adding the scaffolder processor to the catalog, so the process we're adding here is what we now call a module which extends the catalog of new features. It's extends. It's essentially a plug-in for plugins.

B

The way that modules and plugins interact is completely up to each plugin and there's no real standard pattern for it. So, let's imagine we want to evolve any part of this setup. So, let's look at the API surface, so changes to how the Builder is Constructor can be breaking changes to out, add processor method and the processor interface that can also be broken changes how the entity processor is constructed. That can also be broken. You get the point like there's a lot of breaking changes there that we could bring.

B

The API surface is huge, so we're always in mind. We tried to make the situation better for both end users and plug-in Developers So, based on learnings, like what we've just seen. We wanted. We basically sat out with the design for these higher level uh like lists what we wanted to do so, first and foremost, we wanted to simplify the installation of a plugin so having to create multiple files, and managing Imports is just too much of a hurdle and it's a hassle for people trying to integrate with backstage.

B

We want a much simpler process, both for plugins and modules, so. Secondly, we also want a fewer breaking changes. So, by reducing the API surface exposed by these plugins, we want to get rid of the situation where everyone needs to update their backend code. When we introduce new features or new services for plugins, we want less code in the backend setup and move all of this back-end wiring into the backend system itself, like we're going to build a framework. So let's hide away all the hurdles that people have got to jump over.

B

Let's move a lot of that stuff to the framework and we want the system to feel recognizable so to anybody. That's been working with front-end plugins. It should kind of just be a really simple switch to right and back-end plugins right. We've got a lot of that framework already built in the front end. So let's take advantage of that, and we also wanted to make it easy to evolve and extend so users shouldn't have to invent or learn new pattern for every plugin.

A

Now, how did we go about building this new system, so we started sessions and Proto design sessions and prototyping at the beginning of this year, using a top-down approach and focus really heavily on the backend setup and installing new plugin and slimming that down to what you need to get up and running, and we set a lot and founded the sign that we were happy with and published it as an RFC.

A

It was sometime in the middle of May um and you all seemed pretty happy with our approach, so we got straight into building the new system, and that was around summer time and then we've been shipping early versions of the systems for the last couple of months.

A

Now, let's explore what this system looks like with a high level overview.

A

The base of the new system is the back end instance. You can think of this as the unit of deployment. The backend itself really has no functionality at all. It just is responsible for wiring things together.

A

Next, of course, we had plugins which housed actual features just like in the existing system, so plugins operate completely independently if they want to communicate with each other. They have to do so over the wire, so there can't be any direct access between plugins. Now, because of this constraint, a plugin is very much like a microservice.

A

Now it's up to you to decide the way you structure your deployment in your backend installation, so you can take all of your plugins and put it in a single backend or you can go the other way and Other Extreme and split every single plugin into its own backend or something in between all, depending on your need to scale and isolate individual plugins.

A

Next up are services, so we don't want plug and style to implement everything from scratch right. It wouldn't be a very good plugin system, so to make it simpler to write plugins, we provide access to services like the current ones, the database as task scheduling and configuration.

A

Now those are the built-in services and there are many more, but you can also import existing services and you can easily create your own custom Services as well.

A

Now, like the other building blocks, we've seen so far here, um Services is nothing new. What is different, though, is that we now provide the back they're now provided from the backend through dependency injection.

A

So this means you no longer manually need to wire up your backend to pause services to each plugin and the shout out to you see from Rhode for helping out with that earlier.

A

We also wanted services to be a customization point, so just like in the front end where you can replace services, so you can completely replace a service or you can also take a existing service and customize it like changing the middlewares of the HTTP service.

A

Now, we've also seen this plugins for extensions or extending plugins themselves, like the processors in the catalog and the actions for the scaffolder.

A

So we have now created a common pattern for these types of extensions that we call extension points. So they function a little bit like Services, where you depend on them and they're registered into the backend, but rather than being provided by the pack backend. It's the plugins that provide the extension points based on both each individual plugin wants to expose what kind of customization they want to expose and extension points are also decoupled from the plugins themselves and exported separately.

A

So it's much easier to evolve and deprecate individual extension points rather than dealing with one single huge API surface.

A

Now the last piece of The Puzzler modules, which use the plugin extension points to add the actual features, so a module might, for example, add a catalog processor or a scaffolder action and modules are quite similar to plugins. They also have access to services, but one key difference is that they must be deployed together with the plug-in they extend and each module can only extend a single plugin.

A

So in many ways a module is a plug-in for plugins.

B

All right, so, let's take a look at what this new backend system actually looks like in practice. So all of that set of code that you used to have in the back end has now been distilled down to a single function, call create backend by moving all of that setup code into the system itself. It's a lot easier for all of you to maintain the back end and it's also a lot easier to evolve and extend over time.

B

There's plenty happening under the hood of course, like wiring up all the built-in services that everyone wants to manually configure before. We also choose to split out the startup into a separate function, call which makes the API a little bit more flexible, and it also makes it easier to add custom logic. This code, obviously, is not going to get you much more than just like your bare bones back end.

B

So, let's add the catalog plugin so, rather than having to manually, create and copy the contents from the readme that we saw before we now have everything in one single call so note that there's no arguments being passed into this catalog plugin and all of its required Services have been provided into the back end provided from the back end and through dependency injection wow mouthful.

B

This means there's much less code needed if you stick to the defaults, but it's also possible to make the customizations.

B

So you might notice that the catalog plugin is a function. Call that's because we give you the ability to pass in additional options for making lightweight customizations.

B

So now, let's look at an example of installing a module, so this is probably conceptually the biggest change that we've made from that from the previous system. As we mentioned, modules now are first-class citizens that you can create and install rather than some loose pattern invented by plugins themselves.

B

So in this case you can see that the scaffolder module extends the catalog by adding a processor for software templates. So, let's see how this thing works under the hood, so I just want to make this clear that this is not code that you have to write to integrate with plugins. This is something that you would make as a plug-in author, uh so to create modules, for instance.

B

So this is how it's implemented in the scaffolder, for instance, so we'll start with the scaffolder module, because it's a bit simpler, so you can see that we can create a new module. We give a plugin ID and a module ID and as implied that only means that a module can extend only extend a single plugin so in register we add the init function for this module. We declare a dependency on the logger service and the process and extension point from the catalog if you're familiar with the front-end plugins are backstage.

B

This is the same system that we use for utility apis as all of our declared dependencies are, then all of our declared dependencies then get passed to the init callback. Finally, in that callback, you can see how we use the catalog extension point to add. A new processor modules are always initialized before plugins. So therefore, there's no race condition between each other.

B

Now, looking at the catalog plugin, you can see there's a lot of similarities from the previous module example. Again, this code is housed in the catalog plugin itself, so the main difference between the main difference compared to the scaffold scaffolder module, is that we register an extension point to be used by the modules.

B

The reason we settled for this particular sort of nesting callbacks is so the extension Point implementations can be accessed directly from the closure and the init closure itself. This makes it much easier to build extension points, since you don't need to juggle internal and external services and dependencies and interfaces.

B

This is a slimmed down version of catalog implementation, but in reality the catalog plugin is configured exactly like this today. The upside is that this code can evolve without having to get users to update their code themselves.

B

And lastly, for people that want to go a little bit more extreme and do a little bit more customizations that we talked about, you can see how we can replace uh service apis just like we do in the front end so, for instance, here we're replacing the logo with a cloud logger. Instead, we also added the ability to supply default implementations for new custom Services, which means that you'll be able to import and use them right away without having to wire them up beforehand.

A

All right, let's look at some comparisons between the current system and this new one. So in this example, I have a kubernetes client service in an external package that I want the import and use in my plugin.

A

So if you want to do this today in the current system, there are a lot of Hoops that you need to jump through. So obviously you need to import it into your own, plugin and wire it up and use it, but there are also many changes that you need to ask users of your plugins to make to their backend installations.

A

So, first off, you need to ask them to do what you see on this slide and add the packages are dependency.

A

Then they also need to update the plugin environment type to include this new service, and they also need to update the backend wiring to create a manager and a plugin specific instance of this service and forward it to the environment. And, lastly, they also need to forward the service from the environment to your plugin. Now that's a lot of wiring. Not everybody has to do for something that really should be simple.

A

So in the new record system, this is a lot simpler. uh Thanks to the introduction of default service factors, all you need to do is import and declare dependency on the service. As long as the service has a default Factory. That factory will be used to create an instance of the implementation for the service at runtime.

A

Now this means that users of your plugin won't need to make any changes to their backends at all, unless they explicitly want to override that service implementation.

A

One of the more visible changes between the new system and the old one is the reduction in code size of a typical backend installation.

A

So comparing the setup of a newly created project, there's a quite massive difference so, rather than nine separate files with 348 lines of code, we now just have a single file with 26 lines of code. That's more than order of magnitude difference. So, overall, this gives an indication of how much easier it will be to manage a back-end installation. Once we've rolled this new system out I'm sure something a lot of you are thinking is: oh crap, another migration and um yeah.

A

There will be some some paint upgrade, uh but, as we saw in the last slide, it's mostly about deleting a bunch of code, especially if you have a somewhat standard setup, so you delete the plugin specific uh file and you replace it with a one-liner in the index file now any customizations to the backend wiring, uh where, if you've done custom services that would need to be replaced with a custom service implementation and any plug-in customizations would be either options or migrated over to be modules instead, like be able to find both General and plug-in specific guides for this in our upcoming migration documentation.

A

So, overall, this is going to lead to much Slimmer code base and going to be a lot that is going to be a lot easier to keep up to date.

A

So what about migrating? The plugins themselves to the new system might get quite complicated right. Oh so, we've we've done a bunch of these already and usually what you need to do is you take the create router and you wrap it up in the create backend plugin instance. You depend on whatever Services the plugin needed before, and you also depend on the new HTTP service, where you register the router and then you're good to go now.

A

The plugin can accept options for any of the more lightweight customizations, but if you do have a plugin with more complex patterns that probably needs to be migrated over to the extension points. Instead,.

A

So, in summary, the backend Sim this system simplifies the installation of new plugins. It will be a lot less code for you to manage and there's going to be way, fewer breaking changes if you're used to the front-end apis and even the current backend system. These patterns are going to be recognizable and it is going to be a lot easier to evolve and extend this new system compared to the current one, all right. So what we've shown today is available for testing.

A

So, if you like what you've seen you can give it a spin we'd love for you to share any findings or feedback either directly with us here or in the backend system channel on Discord, which I just created an hour ago.

A

Now you can also read more in-depth documentation that will be evolving as we build the system out at the link at the top, uh and we don't have a specific date in mind for a stable release. We really want to give this time to settle and respond to feedback, but we're hoping for some time early next year.

A

So uh hopefully we have time for some questions, but you can also find us here, of course, and later on at Cube. Thank you.

C

Martina will run around so if there's any questions raise your hand, objections about migrating again.

D

Available! For us to experiment.

A

Available I can't promise anything so we're hoping next year, but it really takes time, especially from just our experience, with the front end, um building out the front-end apis as well.

A

We discover things that we didn't think about that need to be addressed, so it does take time to to build this out, but the more feedback we get and the more people try it out. The quicker quicker it's going to be, and also the question was: when do we expect this to be available.

B

I think there's one here running this system.

A

So question is what about running this system in parallel with our current one, and this is the current one going to be deprecated I hope so.

A

Is basically the answer I, so we haven't talked a lot about running them in parallel in the same back end so as mentioned plugins, they they are quite a lot like microservices. You can't really split them apart and move them over one by one from two different installations. If you have a big and complex setup that you need to migrate, um yeah.

E

Sort of a related question, if you're running multiple replicas in this for the same uh deployment for pods, for, for example, um how how friendly is this migration going to be for not killing your three other pods is one of them updates.

B

E

So how much testing of that has gone on.

B

So um maybe struggling for the answer here so I! Guess, if you mean well, you have you want to replace the exit like the migration for ruining the new system. Yeah I mean I. Guess the apis have the contract right so like the apis of your public facing apis like the catalog API, whatever you're using that's the contract that that's not going to change right.

B

It's just the wiring together of the back-end system that surfaces these plugins that's going to change right, so I guess in terms of downtime I, don't think you should have any I, don't think, there's any migration from an end user point of view. It's just integration. Point of view of plug-in author of the point of view. Does that answer your question so I kind of unsure, if that was the.

E

Answer, I guess for clarifications per sake: um database, migrations, I'm, assuming this comes with database changes.

B

No I don't think it does.

E

No! No! No! No so.

B

This is basically just all typescript code. What does all the wiring of the back end together.

E

B

Yeah see the plugins are all the same. Still it's just the actual exported things from the plug-in, so the database migrations are contract somewhere else like this is just what the plugin exports and how we wire everything together. Thank you all right cool. We got there in the end.

F

Thanks I'm curious, if that's the right Forum, uh what's a roadmap of data model Evolution or for uh backstage.

A

I think the catalog Sig is the best place to show up and discuss that yeah.

B

I would say so too I mean in terms of Road mark on what we have planned um kind of hard to say. Don't really know it yet I'm going to defer this to products somewhere at some point, um but yeah, not sure I, think Sig is probably the best place to start. Then we can start uh driving that road map and driving our next um priorities. I guess is a catalog second, so.

A

You can find information about the catalog Sig in backstage Community I, believe yeah, good yeah.

B

Charles has a question.

F

This is for Charles, so this.

C

Is Charles question.

F

um What are your thoughts on being able to unmount a module once it's been activated like at runtime.

B

A

Plugins and a member of that discussion, yeah we're looking at um most likely introducing some form of life cycle service to kind of hook into events of startup and teardown, and just especially for starter startup, where you want to get http HTTP serving listening kind of at some specific point when the backend is ready, um no concrete plans yet work with us to implement. It sounds.

F

C

Okay, it seems that we have no more question well, one.

D

Guides how to guides.

F

D

F

D

Show because, if I'm kind of little bit looking at how much effort we need to put in I'm migrating the current one to the new yep.

G

Automated tools, or maybe you can build some automated clock images. You know this is the things that you have to change.

G

Sometimes yeah.

B

I guess I guess maybe it's possible I think like the pattern of like code, mods I guess is what you're essentially talking about. So the question is, is: can we make uh migration easier, um I? Think it's hard right, because there's quite a lot of handwritten code already, that's kind of very specific. It doesn't really when we mentioned in the presentation that there's no real standard for it already it's kind of hard to detect these patterns and make it easier it's going to be.

B

You know, there's going to be some effort, but it mostly hopefully going to be deleting code like Patrick said, which is always good. We always want to do that, um but yeah I mean, of course, without documentation. Of course um we do. We have some documentation already.

A

Some documentation is a migration documentation. It's not that yeah, so I, also where the backend system is kind of coming from, is the pain to migrate or upgrade. So that's kind of the thing we want to address as well. With this new system.

C

Great, so give it up.

C

Thank you thank.

A