Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: TechDocs: Unlocking the Potential of Engineers' Collective Knowledge - Emma Indal, Spotify

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

TechDocs: Unlocking the Potential of Engineers' Collective Knowledge - Emma Indal, Spotify

Emma is an engineer with a big goal: her new service should provide the best developer experience, including well-maintained, discoverable and usable documentation. So she thinks, “Which tool should I use to write my docs?” followed by “How will other engineers find it?” What if she could instead, in one click, get docs generated together with her service when it's created? Documentation that lives in her repository, together with her code. Documentation that becomes discoverable automatically, to all engineers within the organization. Documentation in one single place: Backstage. This talk will guide you through how we fulfilled Emma’s big goal by adopting the “docs like code” approach and developing TechDocs, one of the core features of Backstage. It will also walk you through why we decided to open source this successful internal product to remove the friction of writing and discovering technical documentation for all engineers.

A

Hello, everyone and welcome to my talk today. I really appreciate you all taking the time to be here with me, and I just wanted to start this talk by thanking all the organizers of this conference and especially for one thing and for that's for encouraging, first-time speakers. This is my first ever conference talk and it's also my first ever conference talk proposal that I did send in, but it's also actually my first ever com tech conference that I attend.

A

So I can tell you that I am really really excited to be here with all of you today and the goal here for me is really to talk about how we have been unlocking the potential of engineers, collective knowledge and how you can do the same within your organizations.

A

My name is emma my pronouns. Is she her hers?

A

You can find me under twitter and on twitter, github or other platforms as well under emma indal, I'm based in gothenburg, sweden- and I am a web engineer for spotify and I've been there for around two years now and I work specifically in a team who are focused on technical documentation um but also discoverability in a on backstage, and so when I'm not working on that. I'm also super super passionate about teaching kids to code. So I'm a volunteer. Imagine mentor for a startup called imagilabs.

A

But let's dive into spotify's engineering organization a little bit, we are around 1 600 engineers distributed in autonomous um cross-functional teams. We have around 14 000, plus software components, and we have around 1400 microservices in production, and these numbers are great. But you know the more. These numbers are growing, the more the need for to unlock the potential of engineers. Collective knowledge are growing as well and when I say collective knowledge, what do I really mean by that?

A

um So what I I presented in the previous slide, some numbers around, that we are 1 600 engineers internally and we have around 4 1 400 micro services, services in production and so, as you might have been able to figure out already. All of these engineers are not working on the same micro services um all the time. So that means really that different engineers and every individual working for spotify knows a little bit about a very small part.

A

But together we know a lot. So if we can avoid siloed knowledge and instead one person can contribute with what she knows about a service or another person about what he knows about a website and a third person about what they know about their data pipelines. You know we're starting to build up this space of collective knowledge, and so why is this so important? Then?

A

um The simple answer is really speed, because if we can speed up our engineers and make sure that they go from stock to unstuck faster, they can focus on their products and their end users. Instead of like issues they're running into daily um and we're now starting to see a more distributed workspace as well, so we will continue to do after the pandemic as well, and this means that you're not able to like just run by your team member's desk and ask them a question that you have on top of your head.

A

So the the need for collective knowledge is growing there as well, but to be able to capture this collective knowledge. We need to remove the friction for the producers and with producers. Here I mean the people who are writing the content, and it's really about so that the engineering knowledge can get shared within organizations and I'm going to tell you how we did that internally at spotify and how we are now making it possible for you to do the same in your organizations and first thing. First, I wanted to guide you through a story.

A

um It's it's not really a story, it's more about me talking about myself in third person, uh but it's really about so that we all can get on the same picture and understand and dive into the problems and friction.

A

And that comes up when, when trying to contribute to collective knowledge- and this is specifically in the format of technical documentation,.

A

So emma has a new feature she has built and she goes through her everyday workflow looks something like this. She has been writing her code, committing her changes, creating a new pull request and the already configured continuous integration environment gets triggered test, hopefully passes and she's emerging the man, the branch into master and her code gets deployed into production.

A

She feels super confident in shipping changes and new features this way, because they're not much friction for her to do so, but she also feels that she cares about this new service and this new feature she's built, and she wants to contribute to this collective knowledge by sharing what she knows because she's she is the expert right.

A

She has been working for this for a long time and she wants to contribute back so, and she also knows that um someone else in her team might work on this in the upcoming weeks or months, and if she can get that person up to speed faster, it's going to be worth it right and so she's, starting to think a little bit about.

A

How can I produce this documentation for this new service, and where can I start like what tool should I use and she gets to a point where she feel like like no matter what path I take, I need to move out from my everyday workflow to do this.

A

So she starts writing the documentation in a new workflow. Of course she has been. She had left her everyday workflow into a new one and she starts writing the documentation, some tools that she took a decision around. She store it somewhere, maybe in the confluence.

A

If if she decided to use that or maybe in google drive, if that was what she she decided to write the documentation in and then she wanted to just share the the documentation with some team members, and so she just copied the link to the the documentation and and then just share it with them.

A

And then, six months later, when you know the documentation has been around for a while, emma's team members gets back to her and asking where, where she did put the documentation that she wrote and if she can share it again and emma first needs to think about. Where did I actually put the documentation? Because she didn't really remember herself, maybe- and um you know she needs to go and find it, and then she can get the link again and you know share it to to our team members again.

A

For the second time and this time around emma did actually produce documentation, even though she had to switch context. Go take a lot of decisions about tools to use where to put it where to store it um in the meanwhile, and leave her everyday workflow to actually produce the documentation.

A

But you know all these extra steps that she had to take will probably stop her from doing it. Next time, and especially since, like her team members couldn't even find the documentation without asking her a second time where she did put it, and so she spent time on time, writing documentation that no one could find, and the end result of this is that the knowledge emma sits on will not get shared within the organization, and if we take take just one second and think about cncf, you know cmcf technologies.

A

Let organizations and individuals develop software more effectively and removing friction of writing. Documentation contributes to this too. So what if we can just merge these two different workflows into one by tightly coupling the process of writing documentation with the developer workflow, so that it becomes as frictionless and natural as shipping new features.

A

So let's go back to the first workflow, because this was the workflow that emma felt confident in there were not much friction for her to ship changes this way. So let's go back to that, but this time emma can actually write documentation in markdown files.

A

At the same time, she's writing her code, so she can commit her documentation changes at the same time, she's committing her code changes and when the already configured continuous integration environment gets triggered, the new documentation is generated using from markdown to html, using some static site generator, as well as publish to some storage in the cloud, and the only thing emma needs to focus on here is to stay in her everyday workflow and all other additional steps are abstracted away for her and then one month later, sorry, six months later, when the documentation has been around for a while and emma's fellow team members doesn't even need to go back to her asking where the documentation can be found, because it's so well known within the organization that all internal technical documentation lives in one single place and can be discoverable there, and that place is something we call tech talks and we will go into that a little bit later.

A

In the talk and the end result of this, then amazing work in tech talks. It was so easy to set up. I will give you a kitten spotify engineer.

A

We did not get a real kitten or we haven't got it yet, at least, but we did end up with a happy spotify engineer who did contribute to this collective knowledge, because it was so easy to set up and get started, and some of you have maybe already heard of this as something called docslike code.

A

That was something that we adopted and built out within spotify. But it's really just one half of solving the problem. It did remove the friction uh to produce and share knowledge in the format of technical documentation in the same workflow as you were, shipping new features, but backstage solves the other half making it collective and discoverable in one place.

A

But what is backstage, then, if you have never heard about backstage before it internal developer portal, a developer portal that enables engineers to create, manage and explore software components, and the idea is really to centralize and simply simplify end-to-end software development, with an abstraction layer on top and that sits on top of all your infrastructure and developer, tooling, and so backstage comes with specifically three core features of which plays a super important role into solving this problem.

A

So I'm just gonna dive into the details of these three core features a little bit more and what role each of them have in this experience.

A

So the first core feature out is the scaffolder and the scaffolder lets you create new software components based on standard templates, and these standard templates can be configurable with tech talks already from before, so that your engineers who are creating a new software component out of these templates, gets a documentation site up and running automatically, instead of having to configure it all themselves, and so who are responsible for putting these templates together.

A

Some of them, you see here in the picture, are actually available for you, but you're, of course, free to contribute with what you, your organizations, what what your organization need uh with your own templates really, and so let's say that we want to create a standalone documentation component, because maybe it's not necessary doesn't need to be attached to a specific service. Maybe it's meant to be a onboarding guide or a or documentation that spans over multiple services.

A

So you just start with filling out some information information such as name description owner of the software com of this documentation component, as well as where it should live, what location, meaning that where the repository should live really and this process just creates a new repository for you with all configuration needed, as well as some default documentation in the docs folder, and so now we have the content. We have some default documentation already um because we created a documentation only component through the scaffolder.

A

But where can I actually read the documentation then? Well? The second core feature is tech talks, and I've mentioned it briefly briefly before, but tech talks is really the place in backstage where documentation can be discoverable and consumed, and it's really one place for all internal technical documentation.

A

So, for example, here we can see a collection of all tech talk sites within the organization and right there we can find the documentation component that we just created in a previous step and by navigating into it we can start read the documentation that has already been generated for us and, last but not least, another important core feature of backstage when it comes to unlocking the potential of engineers.

A

Collective knowledge is the software catalog, and the software catalog in backstage is a way for you to discover all services within the organization that either you own or another team, owns, uh of course, also the documentation that is attached to it.

A

So, let's take a look at what that looks like as, let's select the service, for example, we can just navigate to the docs tab and right there close to the service that we own or another team own. We can quickly just start reading the documentation to it, and so we didn't just wanted to unlock the potential engineers collective knowledge within spotify, but we also wanted to unlock this for other organizations as well.

A

Therefore, we started to open source tech talks as one of these three core features of backstage, but you know we quickly realized that, just because we use a specific set of infrastructure behind tech talks, it doesn't really mean that other or engineering organizations are using the same.

A

So we came to the conclusion after some discussions in the team, but also with the community, that we wanted to focus on making tech talks for all so that, no matter what technologies you are using from the cloud native landscape tech, talk should be configurable to your needs and solve this problem for you and let's take a look at a few examples of that, for example, with less than 10 lines of configuration in yaml you as an app integrator of backstage, can configure the techdocs plugin to publish generated documentation to the cloud storage of your choice, and here are three examples, such as google cloud, storage, amazon, web services, s3 and azure blob storage.

A

Another example of that is that you, as an app integrator, can set up templates available through the scaffolder that I showed before, and you know that core feature where engineers can create a software component out of already existing templates.

A

If you pre-configure your templates with the source code provider of your choice, you are abstracting that configuration away from the engineer so that again they can focus on contributing contributing to this collective knowledge. Instead of the configuration behind.

A

And this slide is just something that I want to show you to list all our currently supported as source code hosting providers, as well as file storage providers, um and if your underlying infrastructure and tools are not listed here, we are more than happy to collaborate with you to get it on the list, and so, if we just take a step back and look at this picture again that I showed in the beginning of the talk, we can see that this light bulb in the middle can be replaced with backstage backstage makes it possible to have one centralized place for collective knowledge and again, if we think about tech talks, which is the key part into producing and consuming internal technical documentation, knowledge is more than just documentation, maybe in the format of shorter questions and answers.

A

Maybe you have a stack overflow instance put together internally and that you're collecting questions and answers within the organization.

A

So therefore we develop features on top of tech docs, for example, to the left. Here you can see the stack overflow widget nic that you have next to your documentation so that you can get exposed to questions and answers connected to the documentation you are reading and remember you know. Collective knowledge only happens when more than one person contributes.

A

Therefore we have developed.

A

um We have focused on like closing the feedback loop so that the consumer of the documentation quickly can just highlight a piece of text and open a new issue and contribute with what they know and keep the documentation up to date, and these are by the way, features that we have not open source yet, but are definitely on our roadmap, and maybe now you all sit here and think collective knowledge is not possible without great discoverability, and I told you before about emma's team members who were not able to find emma's documentation just and just because we have one place for this collective knowledge.

A

It doesn't mean it it. It doesn't mean it is easy to find, especially not when you're about to browse around 4 000 tech talk sites that we currently have internally. So I agree something is really missing here.

A

So we have not been unlocking the potential of engineers, collective knowledge, without leveraging a good search. So that's why we are currently open sourcing and building out the search plugin so that one, for example, from the tech talks, plugin can can index the documentation and make all documentation discoverable through backstage search and with that added.

A

On top of these three core features of backstage, we have been unlocking the potential of engineers collective knowledge, both by removing the friction of contributing to collective knowledge, as well as making it collective and discoverable through one single place, and you know you can do the same, because it's all open source or in the progress of open sourcing, at least so.

A

Thank you all for listening to my talk today and if you are interested in the project head over to backstage.io, where you can hopefully find all information, you need to get started and you know don't hesitate to contact me if you want to like. You would like to hear more about this, or maybe you want to tell me about your stories, but before we end this session, I wanted to give you uh some time for any questions that are on top of your mind and hope that I can answer as many of them as possible.

A