Cloud Native Computing Foundation EnvoyCon 2020 - Virtual, 12 Nov 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Envoy on Kittens: Improving Developer and Maintainer Velocity - Itay Donanhirsh

Description

Envoy on Kittens: Improving Developer and Maintainer Velocity - Itay Donanhirsh

In this talk Itay will present RepoKitteh (https://repokitteh.io), a lightweight, rapid and low cost approach for GitHub automation. The presenter will explain the necessity for GitHub automation, how RepoKitteh approaches the problem and how it compares to the alternatives, such hand crafted custom integrations and GitHub actions. The integration with Envoy will be detailed, and a demonstration of implementing new automations will be given. At the end of this talk, Envoy contributors will be empowered and inspired to add their own RepoKitteh integrations.

A

Hi, my name is itai. Welcome to my talk and kittens improving, developer and maintenance, velocity, which I am going to describe repo kita.

A

First of all, let me assert that cats are a very important part of computer science, as you can see, google's artificial brain learns to find cat videos that was done some time ago. If I remember correctly, also 50 of the traffic on the web is calculated, which is quite significant and also your favorite, open source project envoy has a lot of cat related tags on it and obviously, let's say not forget about octocat, which you can see here and we actually deal with it a lot.

A

First of all, let's define the need for automation when a project have a growing number of contributors like envoy. There are procedures that need to be set in place. This procedure need to be enforced and better. Yet automated, if possible, we don't know want any human doing all the bureaucracy by by themselves, because then you know that just doesn't scale.

A

So there is a need for something to be able to do it for contributors, some operations need to be performed. Other customers constrained, for instance, every project have different needs, and this needs to be customized customized appropriately.

A

For instance, people assign people on issues pr, but the only specific people can do that or inform inform people about the other procedures like enforcing the pr body structure, for example, for maintainers automation can help with issue tracking specialized approval policies and enforcing various rules, which can be done using some kind of scripting that I'm going to show here when approaching this.

A

I've seen in the past, in big companies like lyft and twitter, when interacting with some kind of source control repository, is to build some kind of custom, github, app or other app that consume events, for example from github, and codifies the required behavior that enforced and supported inside the company in code in this app- and it's usually not really reusable and a lot of containers are also grad and bitrate, and that causes a lot of pain when the company goes further on. Also.

A

This is not really appropriate for a lot of projects that are open source that not all the time have the correct the appropriate resources to deploy and develop. Such a thing, there are some custom applications that are out in the wild, but combining them all together is a bit problematic when doing such an approach.

A

There's a lot of things to take care of like maintaining the actual service, authentication, monitoring, high availability, secret management and adapting to github api changes that happen once in a while, while they are much better at it right now, dealing with github api audits, which mostly contain concern ordering of events and also something that often often neglected, which is present preserving issue context, which means sometimes you want to have some kind of store that contains some kind of of some kind of state regarding your pr that you cannot maintain the set elsewhere.

A

So this is a traditional pro approach, and what actually I'm proposing here is repo kit. Repocator is something that does actually most of the work for you. It's replacing your internal internal github app or some other integration that you're using it with one product that take care of most of the let's say: non-business related issues, for example. All of this is now taken care of. If previously, you needed to maintain all this and care about all this stuff, now you need only to care about only the behavior that you you you need.

A

This is actually done by formulating that abstraction to have your business logic, live in and and operate towards the other integration.

A

If you create the right abstraction in place, there is no need to worry about intricate, not gitabate events related things like ordering or other audits that are in the api. You only need to deal with the required business logic that you care about for that the right application is required.

A

I found that starlock, which is part of bazel, is actually perfect for this stuff. Stalac is a language internet for use as a configuration language. It was designed for bazel build system, but may be useful for other projects as well, which I'm very happy to use. Salax allows for deterministic evaluation, hermetic execution and parallel evaluation, which is exactly what you need, because this allows for a jade, low-cost, low overhead, serverless and serverless architecture, where only starla code is executed.

A

No containers required here, as we can actually take a script that has all the the appropriate constraint that we need on it and just run it. It can do only. What we allow it to do.

A

Stalactite is very easy: it's actually a diet of python with a very few differences, and these differences are actually important because they allow all the properties that we like aka, deterministic, hermetic and concurrent repocket is using the actual star, go module by alan donovan, which is amazing.

A

This is how the repo kit architecture actually looks like there is a repocator github app that you install on your repo. You don't need to write any new github. You are just using the github app that actually developed and given to your repo.

A

The repo keta engine actually takes consumer event from the github app, and they can also call a various api functions on github. It executes a startup streak on demand which will expand where they reside and how they are written very soon, and it's variable facilities that we can use in order to actually make it easier to do the stuff that we want. For instance, we got tracing. We can actually have a very nice ui. That shows exactly what the repo takita script actually doing or done any persistent, and you can introspect it in any.

A

Given time we have um secrets that we can be supplies to the script and there's also per issue per pr context, which we'll elaborate on later. On the other side, there is also a ui, currently quite simple, but it's going to be improved later, on which the nice cat lady here can interact with, with all the components using it demo time. I will demonstrate now how you can enforce a bug notation in a pr.

B

B

B

A

A

There's a lot of stuff that actually included in repo kit that gives us in order to focus on what we actually need to write and not turning various other services. For example, repo kit includes tooling and apis for frequently less capability. As I said before, secret management, digital debug output, using tracing github api access function, permission model, which you github does not give you, and we can actually use the gripo kitty to actually enforce it models with version piping and much more. So how does it all work in the root of the repo?

A

There is a file called replicated.star. This file actually is the root module and whenever there is some kind of event that needs to be exited upon repertoire, node how to open this file and evaluate it. This is a very simple example of how to write a very simple repocket script. In this instance, we are registering a slash command when you type, let's say in the comment, slash backboard. What will happen?

A

It will write the underscore backboard backboard script that will issue a label that will put a label on the issue of the pr that says, backport review, I included references to documentation of repocite in each slide, which I hope I can send later and you can just click on it and see how it works.

A

Handles command, actually register a command header that is executed on slash commands. This is a simple demonstration from a pr on envoy someone do it slash backward and you can see that repokete added required label on the pr. This is an example of documentation from the reference manual that you can actually look up online. Even now, if you want it shows you the github module that has an issue label function, which is the description of each argument and also it can point you to the appropriate github api that it actually accesses.

A

This is an excerpt from a module that supplies by repocator, which you can check out later. In this case, we register a pull request event. When we get a pull request event from github, we can actually perform stuff on it if you're not really familiar with polar cost event. Just go to this link and read about it. It's not that difficult.

A

In this case, we handle the synchronized action in the pull request, and you can actually see that we're doing some high-level operations here, according to some criteria that we saw in this module handlers receive context when executed. As you can see in the definition, depth, pull request, action and labels. These are actually populated. Dynamically depends on what parameter you specify to the pull request you can see where these parameters come from a bit later.

A

This is an example of the use statement. The use statement is used to load a module that is defined in repocate, or you can write a model yourself. First of all, you specify the path for the model that you want to load. It needs to be residing in the private or public repository.

A

As long as the application has permission to access, it depends on what you install it. Configuration can be supplied to the module once being load. The module will actually every event on it and event handler will accept this configuration and be able to parse it.

A

Some generic modules are supplied for you as part of repo keter, and they are open source and they are in this bus a it's actually being supplied to the reconcile method which is registered using handler's command. So when there is slash check owners being typed into a comment, the reconcile is being called with the config argument and the config argument is supplied to get specs which can load the passes parameter from there. So there is another way to load modules, which is using the starloc load statement load.

A

Actually, instead of registering handler on modules and stuff, like that, what he does it brings a function it defined in another module may be an internal repo kit, a module like test here or lie on another sl party module like utils and circle c high ear and bring the function that is defined there inside the context that's being evaluated. For instance, the text.match here is being called after reloaded.

A

So what is the difference between loaded use? Load is star lock, built in which brings in a function from other models and can be called from. Any module.

A

Use is a repockety function that registers handle from other models and can be called only from the root module note that use does not bring any new variables into the local context. It just tells tala a replicator to load that module and register its handles.

A

Repo gita allows for a state to be stored for an issue or npr.

A

This is useful when a state need to be stored for use later, for example, in this case, in the excel form repo kit owner check module, we actually store who approved, which passes in pr this allow later on, to see if all relevant passes were approved and by whom you can supply secrets to module. In this case, we are using the get second function to fetch a registered a branch registered, a secret that was supplied using the replicator ui to a module. We are using the getsecret function.

A

The get secret can be called only in the root model, meaning only replicated dot star. In this case, we actually can also specify secret url to http when any parameter begins with the word secret underscore, it will not be seen in the tracers later on and it will not leak and replicated debug information. This will be a little more clear when I explain.

A

Traces here, I'm explaining traces, so let's say that repocket issued some kind of a comment to in your pr. In this case, this is an example for owner checks, uh customers, owner check for envoy, and you can see that there's a little smiling cat in the bottom. When you expand it, what happens? You can actually see a little bit of debug information about how this event will process.

A

You can actually press the trace link which will bring you the tracing information page that you can see exactly what happened with this event. First of all, you can see the event payload from github, which is long, but it can be very useful for a debugging.

A

You can press the evaluation tab and then you have a lot of information about how the event was actually evaluated. First thing that it's interesting is the context. The context is what the parameters that are given to the handlers can take the information form. This is an example of a context. Every filter can be consumed inside an event handler. All you need to see to do is just specify its name as a parameter for the event handler and it will populate for you when it's called another interesting stuff. Is that calls tracing?

A

If you expand the course, you can see all the methods that were called from your script. The dollar sign be before the use say that this function can be used only in the root module. You can see the all the models that were used, along with the configuration sense, the secrets, because it begins with secret underscore, and you can also see an example, github call and exactly what the github call returned.

A

Let's see now pin draft and modules pin draft has actually all the references that will pin in case on alvo. You will always pin the repo kit the modules to a specific shot that case. If the mode was getting updated, we can actually wrap in it in envoy to adopt it, and these are all the models that were loaded. The laws, along with the exact shot that were taken from this, is very useful for debugging.

A

Obviously so, let's talk a bit about repo key test, specifically in envoy, you can see the communication from every command that envoy is using in the link, above also the root modules. Obviously it's in the root of the envoy repository.

A

Although specific models will live in unvoiced eye report models, these are models that were written by android people, not me or anyone else on the reporter side and are just being used available, commands that are ready for you, which you can see again in the replicated.md on the top, is slash assignment, slash, review to assign specific people or assignments reviewer specific paper to a pr, retest and retest circle, which are relaunching circle, ci or azp, which is very useful, wait and wait any which is mostly for the maintainers.

A

When they're expecting some kind of response from someone, they do slash weight and it will label the issue as waiting and whenever someone else or even them write a new comment. It will just unlabel it and then they know there's a reply there and another command backboard, which adds a backport a label to apr.

A

Also, how we talk made an awesome custom made ownership for envoy, which you, if you contributed to amber, you, probably know about that alerts. When specific owners need to review specific passes in the pm.

A

So github actions you probably under asking okay. So what's wrong with github actions, nothing is wrong with the github actions. Github actions essentially supply the same functionality. The difference is that actions are more optimized for long-running processes. Like crm deployment, you can do shorter stuff, but will better slightly higher latency for the stuff, which is not very comfortable.

A

They are more resource, intensive intensive and are generally more cumbersome to implement, as they require document containerization, which has more levels of moving parts. That's resulting in a slow turnaround. Time and time to you know to take you to actually get the action to run and to be developed. Ripper kitty is optimized for short running actions enables much lower latency, faster turnaround time and lower cost, because we are actually using something that is very, very low over at last starlock in in the process or in a separate process for a higher security.

A

There is no containerization waiting for kubernetes to spin up another port or container and stuff like that. You just get the event and once as fast as possible, we actually have a very few instances of gpc in gcp for android and I think the highest cpu that I ever got was about two or three percent. So if you need something that is very low cost and very lean and mean this is for you.

A

So what's in the future of if you're replicated, there's still a lot to be done, it's working very nicely for about two years for a envoy now, but I would really like to integrate it into gitlab, which will be awesome which I'm working on right now. Also it shouldn't be hard to take it and put it on premise.

A

If you need which I was paying a lot of money, let's say for to github improved ui, currently device pretty minimal, it's very effective, but it's very minimal, also script testing and some kind of github fake to enable the script. Testing is very high in the priority priority list and also something is very high, even more documentation that I have now in order to import more people.

A

And so my experience showed me that people actually getting on board and on boarded on re-pocketing, very, very fast, which makes me happy, I'm looking for more projects so go and go to a repo kitty. Dot io stretch wait list, and if you want sound yourself there and I'll get to you the moment that I can onboard more, which is pretty fast, I just want people to come for more information. There is repocite.io.

A

You can have links to the recommendation: a link to open support tickets.

A

Please sign on the waitlist if you're interested and also I want to do some kind of hands-on lab session when enough people are going to sign in just go to repo dot, io lab sign there specify your time zone, because this is important and will organize some kind of lab session, or maybe a few of them to teach more people about repo kitten.

A

Thank you very much questions hi. I hope you can hear me if you have any questions. I'd love to answer them.

B

All right, hey good good afternoon. Everybody welcome to.

B

A

Feel free to ask questions in the chat, yay, ryan uh yeah, I'm considering it. I need to think about it. Some more. I don't think the code is really ready to be open-sourced.

A

It needs to be, I think, maybe better documented, because I'm the one who worked on this uh but yeah, I would consider it depending on the interest and if you, if you want it open source, just let me know whether it's in.

B

A

I'd like to say again that if you're interested in a testing, repo kitty or interesting in some kind of lab go to replicated.io, you should have links there. I think both to the lab and the waitlist. In any case, it's replicated.lab or slash waitlist.

B

B

A

Also, if you have any questions, you can email me I'll post in the chat, the email.

A

Or for further questions and.

B

A

A

A

A

If any other time might be more demos but yeah, maybe some other.

A

Time design who just entered the session here, I did some interesting uh plugins for on for repo, also how we did the owners check, which was uh really cool based on my own check. He made it more specific to envoy and.

A

I can't say enough how much um starlock is awesome, especially the implementation that alan donovan and yeah. I think that there's really no need to create a container for anything just go, and you know if you have some kind of general environment like lua or on star luck, you you can actually use this. Maybe I'll talk in the future, some other place about the benefits of such environment. I think it could be very.

A

A

I'll give another minute to open the question: if there's nobody I'll just leave.

A

All right, if you do the lab, I will make you a replicator certified engineer. This is cool and make a.

A

A

Okay, if you have any questions, feel free to email me and also register on repokito.org and yeah I'll. Give you back the time bye.