Kong User Calls, 12 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: [User Call] Controlling your Kong Gateway with decK and CI/CD

Description

In this session, DevRel Director Micheal Heap will show you Kong’s declarative configuration capabilities and how to use your CI system to lint and apply these configurations in a variety of environments. Test your changes in a staging environment, then apply exactly the same config to production with a click of a button.

Kong’s User Calls are a place to learn about technologies within the Kong open source ecosystem. This interactive forum will give you the chance to ask our engineers questions and get ramped up on information relevant to your Kong journey.

#KongGateway #decK #CICD #Kong

A

Thanks to those who joined us today, I'm taryn jones part of the developer marketing team here at kong, and I want to welcome you to today's online meetup.

A

We're going to be talking about controlling your kong gateway with dec and cicd and presenting today we have our director of developer relations michael heap um at the end of michael's presentation, we'll open it up for q, a and discussion um you will be able to unmute yourself um and turn on your video if you'd like, but you're, also welcome to just post your questions in the chat at the bottom of your screen and we'll make sure and get to those.

A

um So with that I'll go ahead and pass it over to michael to kick us off thanks, michael.

B

Thank you, taran, oh welcome everyone uh good morning afternoon or evening, depending on wherever you may be joining us from now. It's getting a little bit dark over here in england, um but I'm really looking forward to our time together this evening today we're going to talk about configuring, your congate we're using deck, which is kong's, declarative configuration tool, and you might be wondering why you'd actually want to do this.

B

After all, you've been using the api or the ui just configure services, routes, plugins and that's worked just fine in the past, but how many times have you heard this? What do you mean productions down? It was only a conflict change, it's easily done and it's not just small companies. Some of you may have seen the facebook dropped off the face of the earth the other week. That was just a conflict change.

B

Hi, I'm michael I'm, the director of developer relations at kong. I've been here since april this year, but my background is in engineering, including sometimes an sre.

B

A few years ago I wrote a book on configuration change management uh using ansible with from red hat and despite that, I've still broken production, with a configuration change recently like the last four weeks. So.

B

Today, we're going to take a look at how deck and github actions can save you from breaking production, we'll introduce deck use it to back up our gateway instance configuration make a change to that config file and then apply that back to our gateway using deck first manually and then through a github actions, workflow and then once that's done, we'll get on some of the more advanced stuff.

B

Now things like manual approvals for deploying to production and how to manage distributed, service configuration and, finally, how to go from zero to running gateway in under three minutes using declarative configuration she takes about 20 minutes, you ready all right.

B

Let's start by introducing deck, which is a configuration management and drift detection tool for kong. Well, that's a bit of a mouthful! So what does it actually mean?

B

Well that can both export and import your service configuration providing backups and a way to see your states as a text file. It can run a diff between your proposed configuration and your actual config. It can validate a declarative configuration file and it can automate distributed configuration changes in the cicd platform.

B

It's also important to know that the deck is 100 open source. It uses the same apis that are available to everyone.

B

You can see it's source code on github, it's written in, go that also means that you can contribute if you have any issues or feature requests, we'd love to hear them and if you've not got yourself and want to contribute, we'd love to help you on your journey. You can learn more about slash github.com.

B

Now it is possible to start from scratch with deck, but I'm guessing that most of you already have a running gateway, instance somewhere.

B

So let's set up a new instance with a single service and a single plugin just so that we have a shared starting point. It won't be a complex one, just single route single plugin.

B

So I'm going to configure a service that points at a demo that we built for kong summit that ran two weeks ago and I'm going to use the kong manager ui. So, let's swipe over to the.

B

B

Load up the kong manager ui, which is behind the firewall. So let me just fix that.

B

um Edit inbound rules, so you don't want to be exposing your kong manager, admin port to the whole internet, which is why I've got it um behind the firewall. So I'm just going to add a quick rule to open it up to the entire internet for demo purposes. Only just so we can see what's going on.

B

B

That and then try again here we are now it says here: kong is running in free mode. That's true, and this is important to note, uh because it means that you can use it too. You don't have to have an enterprise license to use kung manager, it's not in the open source version, but it is in the the free version.

B

So if the important thing to you is that kong is free, not kong is open source. uh You can get this for free and try it out. It's really quite nice.

B

So let's go in.

B

And it won't, let me create a new workspace.

B

Live demos. Hey give me a second. I am going to apply a license um just to get this working very quickly.

B

um Eight thousand one and then so you can't see this, I'm doing it just off to the side of the screen.

B

Hey because I don't want you all to steal my license data there, we are that's what I was looking for so now. I can go in and create that service that I was talking about.

B

So the service name will be summit demo.

B

This running on summitdemo.com.

B

We want to add a single route, um we'll call it sessions, because what this api does is. It returns the sessions that were happening at summit. So it's a get request to slash sessions and we don't want to strip the path and let's just check that that works.

B

So if I go back here actually I need to grab the url just quickly.

B

And go back, I 8 000, not rematched by call sessions. We get all of the session data from the api that shows that our upstream is working as intended.

B

Now, if I go ahead, let's add the red limiting plugin.

B

So we want to limit by ipa, address, let's use the local policy and give them three requests per minute and create that if I go back here, one two three: the limit exceeded excellent. So now we've got a non-starting point: I'm going to show you how to use deck to make a backup.

B

So if I run deck, dump deck will download and serialize the configuration file um to a file named kong.yaml. So if I go ahead and run deck dump, I can now see that we have kong.yaml.

B

If I open that up in my editor, I made that a little bit bigger.

B

We can see all the things that we just configured in the ui, the hosts, the name, the limiting plugin, I'm just going to slim this down a little bit because we don't need all of these values and we don't need the redis host to password using the local policy.

B

And we only want it to apply to http.

B

We've got our route, it's a get, its name is sessions. The path is slash sessions.

B

The strip path is set to false. This is everything that we just configured.

B

At this point, this is an awesome backup for of api gateway configuration.

B

If I go back into this and say actually I want to delete that.

B

Oh, I need to delete the routes first.

B

And then delete the service, like all of that configuration is now lost, but bring it back it's as easy as running deck. Sync, we can see it created a service, a route and a plugin. If I go back and refresh everything is back.

B

Now, usually, when I run deck dump or deck, sync I'd have to specify the admin api address too, but in order to simplify things for this demo, I've added the con address to my deck configuration file which lives at um dot deck.yaml in your home folder and that's what it looks like the alternative would be to run deck dump.

B

Cong address then the url I don't want, so I have to do that every time, so I've added it to my deck config file.

B

So we had a service definition with an upstream and a plugin configuration.

B

So we ran deck dump to download the config indexing to push it back up. But let's make a change to the red, limiting imagine that we we deployed it with three minutes and decided that's too low.

B

Instead, we want that to be 10., so we can change it to 10 in our configuration file, jump back and run deck sync, and this time we can see that one thing was updated because our service already existed, it's already existed and it's just the looking config that's been updated.

B

So if I click in to our service and into our route up not into the route into the plug-in edit plugin, we can see that that config.minutevalue is now 10..

B

We didn't use the api well, we did indirectly through deck, but we didn't use the ui we're now starting to manage our gateways. Configuration into collaborative config.

B

Now being able to manage that config declaratively is great, but now you're the only person that can make changes unless you're editing that config file on your machine there's no guarantees that you have the most recent configuration, and this is where using deck with ci cd comes in, because we can treat a github repo as our source of truth for the gateway configuration.

B

So let's go ahead and do that github.com, mp, deck ci.

B

Cicd, so I've got this repo here, it's currently empty, and this is what I'm going to be using as the the source of truth. So if I jump back to my terminal.

B

um I'm going to add.

B

There we are, I've, got this folder and I'm just going to copy.

B

Kong.Yaml into this folder.

B

B

Initial commit and push that up.

B

If I go back to the repo con.yaml is now there- and this is great- we've now got a shared source of truth, but people still need to pull it down to the local machines, make changes and commit back up. So, let's add a github action. That applies the changes in that file. Every time we commit to men.

B

So, to do that, we create a new file in the github, workflows, folder and I'll just call that push.yaml, because that's the event that we're going to be triggering on and say on push.

B

We need to run some jobs and call it run deck and it runs on ubuntu. Latest um deck will run on windows, ubuntu and mac, uh but for this demo I'm gonna use the ubuntu runner.

B

I have some steps and the first thing you've got to do is set up deck.

B

So deck isn't pre-installed on the the runners. Thankfully kong offers a github action to set this up, for you so just say: uses kong set up deck at v1.

B

B

um Deck version and we use 1.8.0.

B

At this point, the machine has deck installed, which means our next step is run deck.

B

So we run deck sync and we've got to pass in the cong address this time, because we don't have that config file on that we have my local machine on github actions, so I'm gonna jump back across here and just get that address there. It is.

B

And that should be enough to apply on push.

B

B

Just a new folder.

B

B

And push that up now, if we go back to the browser.

B

We can see it's there, it's not red, which is a good start.

B

But it is going to go red because I forgot a step.

A

Michael while we have just a moment here, mert has a question: um should you ha um should have we multiple repositories for multiple applications? I imagine each application has different config files.

B

It's a very good question and I will cover something about that in just a little while.

A

Excellent. Thank you.

B

So I've set up deck and I wanted to run deck, but I realized I haven't actually checked out the files in the repo, which means that when dexsync runs, it won't be able to find kong.yaml.

B

So just name check out and I need to use actions check out at v2.

B

And push that up.

B

And then we'll go back to the browser, see that it failed.

B

Oh, we can't see the logs because I'm not signed in.

B

If you give me a second, I will sign in on this browser.

B

So I'm using a special presentation browser so that we don't have my entire history in here and I forgot to log in to github. So let me just do that now.

B

Okay, I'm logged in which means I should now be able to refresh there. We are, we can see the logs now say.

B

Oh no shorthand flag. I have also mr dash on cong address.

B

So we go back in again.

B

You have to apologize for my terrible, commit messages.

B

And let's watch it run this time.

B

Here we are so check out and then it ran deck and if I go back, nothing should have happened because it's exactly the same config file yep there. It is still 10 minutes.

B

If I go back to the chord and edit that config file and say actually we went from three to ten. That was a bit too much. Let's give them five five minutes.

B

Change rate limit to 5 minutes for session push that up.

B

And go back to actions again.

B

It should just take a second or two.

B

That old van fighting, we can see that we got rid of minutes, 10 and added minute. Five one thing was updated and if I go back and refresh here.

B

Config.Minute is now five and now we've got that single source of truth on github, and not only that we've got a record of when things changed, who changed it like configuration, changes are no longer invisible. We've got that history in github.

B

So we're halfway through we've introduced deck configured a new instance: we've used deck dump indexing and we applied changes to the github actions. So now we should get onto the more complex usage things like reviewing changes, because perhaps you don't want configuration changes applying directly to production?

B

After all, that's no safer than people making changes on their local machine. So we're going to add an approval step to our actions. Workflow that uses deck diff to show what would change.

B

And to do this, we need to create a github environment. You can call it whatever you want, I'm going to call this one production and I'm going to configure it that it requires review. I'm going to say that I have to approve anything that goes to production.

B

That's saved, which means we go back to our our workflow and the way that we're going to set this up is we're actually going to run two jobs.

B

I'm going to run deck diff.

B

And what deck diff do is it will show you the output of what was created deleted changed without actually doing anything.

B

And then we've got run deck which needs.

B

Deck diff so we're adding some ordering here, so this deck diff job will run before even deck and also this depends on the environment.

B

The production environment.

B

Environment., if I push that up.

B

And then go back to the browser.

B

We can see now that there's two jobs we've got deck diff, which is new and run deck, which is what we had before, but that now is gated on that production environment. So if I click into deck, diff.

B

We can see the changes that were going to be made.

B

But if I click into run deck, it says it needs approval to start deploying the changes, and I can review that want to go to production. Yes looks good to me proven deploy so now. We've got that governance step. Not only are people not making changes on the through the ui through the api it's going into github, but they can't just make changes themselves.

B

It's now got to go through that approval step from someone to say. Yes, this is okay to go to production.

B

Now we've been working with a small configuration file so far, um but it's unlikely to be this straightforward.

B

Let's take a look at a configuration file. That's a little bit more representative of the real world.

B

So, let's close those down.

B

So here's a config file with uh I can't even remember how many services one two three four fives, six services. There are only six services in here and they're, not even big. Some of the services don't even have roots or plugins, with six we're already at 250 files and it's easier for these configuration files to get to a thousand two thousand five thousand files, long five thousand lines long.

B

It gets unmanageable, and this is still a small configuration file to help with this deck supports, distributed configuration where you can manage each service as its own file, and it will merge them all together when applying changes. So this is where we get back to motor's question about. Should we have these configurations in different vpus?

B

If you're using a single configuration file, then you have to have it all in one repo, but if you're using distributed configuration, which is what I'm about to show you, you can have them in individual repos and then build them all in your cid cicd platform um into a single thing to send off.

B

So imagine that we've got the sessions team, and this is just what we had before.

B

Get sessions looking config, but we also have this products endpoint and what products gives you? Let me show you actually.

B

B

ah That won't work because I don't have anything created. Yet uh that's fine, let's go ahead and apply this distributed. Configuration I'll show you how to do that. Then I'll show you what the end point returns.

B

So the way that it works with deck is you run deck sync and by default it will just look for kong.yaml. But if you pass in the dash s flag, you can pass in multiple files.

B

B

I should be able to yeah call that slash products endpoint and we can see con cars. The gateway has insomnia, has developer portal and some of which are open source, some of which are enterprise. And if I go back to the ui, I can see multiple services now and there's the product route, with the rare, limiting plugin.

B

Finally, I want to show you just how easy it is to get started with conch. So what I'm going to do is provision a brand new ec2 instance in aws, using polumi, install kong with ansible and then configure those services with deck, and it's going to take less than four minutes.

B

B

um Blew me up stack um community.

B

So plume is now going ahead and it's creating me an aws.

B

It will be if I could actually press up on my keyboard and it's going to create us a security group and an ec2 instance, and let me show you what that looks like in code whilst that's running so pollumi is a infrastructure as code tool and if you've ever used, terraform or something like that, it might feel familiar.

B

um Bloomie is called rather than declarative, config and I've written this one in typescript, but they offer lots of different languages, so we're going to create an ubuntu instance.

B

So we need to get the ami id we're going to create a new security group uh ports 8000 uh for hosting the block set in 8001 to be able to run deck against it again.

B

You wouldn't have this publicly accessible on the internet, usually, but for this demo, I'm just going to do it for ease of use want to create a new ec2 instance, put it in the security grid that we just created and it's accessible using the key name of key name, which is m heap gateway testing, which is just a key that I've got on. My ec2 instance um on my aws account. Sorry now, if I jump back to the console, that's created, so that's we've got an issue to instance. Next thing is to install kong.

B

I say I'm going to use ansible for that.

B

So I just need to update the hostname.

B

Ansible playbook point it to that inventory: file: ask fault password and playbook.yaml.

B

And I'll show you what this is doing so ansible is a yaml-based configuration management system. We've just got a set of tasks that it runs in order. It adds to the kong app to repo because we're using ubuntu it installs the required packages, a python 3 and acl for ansible and then postgres and kong enterprise edition.

B

We create the postgres database, create the kong user copy. The config file, which is just here, it's a minimal one, listed on all ports on all ip addresses. All interfaces using postgres. My password is new password.

B

um And then we want to run migrations, so kong, migrations, bootstrap and then restart cong. Those are all the steps you need to deploy. Kong so add. The report install the package create the database, create the user copy, the config file run migrations and then restart cong.

B

And what I was showing you that ansible did all of those jobs. So now, if I go back to the browser.

B

And call port 8000. We can see that kong is running but not root, matched with those values. If I go to slash sessions, we get exactly the same thing.

B

So we've got just one thing left to do, which is take that.

B

And takes the icd run deck sync cong address.

B

And it's http port 8001.

B

And now, if I go back and refresh, we get all of our session information now this machine didn't exist. Less than five minutes ago, we've just provisioned a new machine, provisioned kong install and restored all of our service route plug-in configuration using deck. If I just refresh a few times the limit exceeded just what we were expecting.

B

One last thing: dec also works with conconnect or hosted control, plane solution. All the demos we've seen today use the on-prem gateway, but that can run deck connect sync to to manage services within connect.

B

In fact, if you want to migrate from on-prem to connect that can be done in just 60 seconds using deck, which is pretty awesome, at least I think so,.

B

That takes us to the end of the demo, thanks for your attention- and I am here for questions, if you have any.

A

uh So hey this is siddharth here, uh so have a quick question, so uh the deck that you, the address that you provide from a tech perspective, is that the admin uh instance admin port and the host.

A

It is yes, okay, so whatever you um go and deploy, uh it would be there from an admin perspective, but suppose I deploy everything through uh kubernetes using the kic ingress controller. uh I wouldn't be able to view that from a tech perspective right is that a right statement.

B

um It is if you've deployed using the ingress controller. You probably wouldn't use declarative config to configure things. Instead, you will be using crds, so your crds um ideally would also be managed through a github stack type. Workflow, so you'll be committing them to github, and then your icd platform will be running. Cue. Control apply on.

A

B

A

Okay: okay, sure that helps. Thank you.

A

And looks like we had a couple of questions in the chat um some of these kind of came in towards the middle, so I'm not sure um regan did you still? um Did you get your answer to that question? He has uh yeah go ahead. It's the same question.

A

I think that's answered no.

B

And another one from uh did I integrate github in aws earlier? No, I didn't. I did something very naughty and opened up my admin port to the entire world and that's how github could reach my cloud provider. I just gave the address um and anyone could have called that admin port.

B

If I were to do this properly, um I should be limiting it to the github actions runners ip addresses only um which you can get at api.github.com forward, slash meta. They return all their ip addresses, though, um alternatively, um you can add a necessary step to your workflow, so that your github action actually logs in to your kong machine and then uses the admin api locally on localhost, rather than connecting to it over the internet, which is a much more secure way to do things.

B

But it would have made this demo a little bit too complicated and I was just really trying to show off um the deck utility.

A

And um the con manager that you showed right um so can that be I mean? Is it a pre free version? Can we use like open source or should be, should we have a license for that.

B

Nope, so deck is entirely open source. um It works with con gulp and source. It works with kong 3. It works with con connect and it works with all of our gateway. Runtimes and deck itself is fully open source.

B

It's available at github.com forward, slash deck.

B

Okay, thank you thanks, simon.

A

All right any other questions.

A

Can be general questions as well? If you have something unrelated, perhaps like you can ask.

B

Happy to answer anything kong related not just about dick.

B

All right, in which case thank you for your attention, everyone, and we will hopefully see you next month.

A

Yes, our next call is on november 9th and we hope to see you there. Thank you michael have a good day. Everyone thank you team that.

B

Was helpful, thank you, cheers bye. Thank you.