Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2019 (Barcelona), 23 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: A Kubernetes Application End-to-End: DevStats - Łukasz Gryglicki, CNCF & Josh Berkus, Red Hat

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

A Kubernetes Application End-to-End: DevStats - Łukasz Gryglicki, CNCF & Josh Berkus, Red Hat

You may know DevStats as a repository of contributor statistics for Kubernetes (devstats.cncf.io), but you're about to find out that it is also a great demonstration case of migrating a complete application to orchestrated microservices. Our community has lacked solid, production-quality, end-to-end, all-open-source application examples, so we decided to make DevStats into one.

https://sched.co/MPZG

A

Hello, everyone welcome to this session of the community track.

A

My name is Leticia D I'm, a software engineer with planet-scale and I'm, also one of the maintainer zuv wit s, which is a CNC F open source project.

A

Our next talk is going to be on a kubernetes application end to end del stats, and let me introduce our speakers.

A

This is Lukas Greg, latch, key of CN CF, he's a senior software developer with CNC F and the primary developer on dev stats, and we also have Josh burkas who's, the kubernetes community manager at Red Hat, and he is active in the release and.

B

Introverts contributed.

A

Experience six for kubernetes.

B

Okay, well, thank you. Well welcome. We've already been introduced, so you don't need that. But let me talk a little bit about why we're here. So a while ago now, we've actually had an issue open for a long time, better kubernetes, eye, dev stats, but then a while ago is actually looking. I was teaching some new people how to use kubernetes properly and I was like hey. Let me give you a good.

B

Let me find a good example for you of a production application in kubernetes right what that would look like right, something that's got deployment, information and configuration, and it's set up to use some kind of CI CD and has like a database and a UI and connection routing, and maybe some other things like jobs and config maps, etc, and some way to deploy it right.

B

All these things and I looked and I looked, and it's Google searches and I looked in all of the kubernetes and CN CF, stuff and I looked in people's medium blogs and what I found was guestbook. How many people interview is the guestbook sample app right off of the cou Bernays sample UPS right great? So this has got. It's got a couple replica sets. It has secret and it has a couple services and that's pretty much. It doesn't have ingress.

B

Doesn't the database is non-persistent, there's no operator or helm or any other way that you would actually deploy this as an app there aren't any jobs? Cron jobs is no CIC D, it's not current for 113, which is when we started working on this, certainly not current for one 14 or 15. Now nothing is SSL.

B

It's definitely miles away from being an example, production application designed to be as simple as possible. So when we actually started work on making dev stats completely a kubernetes application, we're like, let's not just make this a kubernetes application. Let's make this an example: kubernetes application, because it's open source from top to bottom, so we don't have any limitations for what we can put out there and github for people to look at so before we actually go into the details of what that looks like what's. Actually explaining with dev stats.

B

Is so you'll understand what the parts of this application do.

C

That's not the best stats is a C&C have created tool. That is that's main reason, for it is to analyze contributions and development process across the cnc of projects, so it uses three different data sources. One of them is github are chips which are which are just our way of store our github activity using bigquery and using al-kitab al-kitab events. We also use git for a storage to analyze files in git, and finally, we use github API to to see if our data is correct, it uses graph. Anna is a UI and also well.

C

You can just take a look at great.

B

It's like yeah, so I mean it's basically it's here. This is the thing that I and some of the other community managers use all the time to find out how the kubernetes community is doing. It's also available for other communities, and so we have a whole large set of different dashboards for community information like, for example, an important one to us is one called peer time to engage, which is that if somebody submits a brand-new PR, how long does it take to get the first response from a human?

B

How long does it take before it actually gets reviewed and how long does it take until it gets merged or rejected right? Because if that starts going, if that amount of time starts going up, then we have a problem so and lots and lots of other charts like that.

C

And so across the two years old def starts a project.

C

It initially was created to track CN CF project, but after some time some other foundations also joined some like, for example, continuous delivery and so on, and every single foundation can have its own deaths, as instance, and in half it can have multiple projects tracking across multiple get reports on things like that. So so.

B

We get into that so so then so we had, but basically you know so then we went through an experience that I think a lot of you either have gone through or going through. How many people here are looking at moving a monolithic application to a bunch of kubernetes micro services and how many people have already done it yeah. So this is a familiar experience right and again with the opportunity here to do this in sort of an example way right. So our old set up was a couple packet servers, so our.

C

Own set up that is currently still running: it's using two powerful packet net servers, which are quite quite powerful, as I said so, there's 38 core processors. There are 20 20, 200 gigabytes of memory. There are six mirror SSDs disks and so on so on, we are using Apache proxy. We are using, let's at Crypt and basically it's starting to became a problem, because we need to update all these tools when, for example, new version of Ubuntu comes, then we need to just install everything from scratch, so yeah and so.

B

Then, a few weeks ago, a friend of mine, Michael Rogers, who I know from the calculated A's our ping mean, was like hey. We actually want to run dev stats for our open source projects. How do I install it? And my answer for that is well that's kind of complicated right now, and it would be really nice to make it a little bit less complicated, on top of which yeah.

C

So we created this example for purpose of this cube, don't talk but do during doing so another foundation, graph QL wanted their own deficit instance and basically, while developing instance for this talk, we used exactly the same approach and actually it's already running in production on kubernetes.

C

B

The graph QL and also.

C

In the meantime, many other projects asked us for creating their own deficit since our, for instance, a cube flow HDL and Linux. Foundation is also considering running this internally as a data source. So it's it's a lot of other projects that need to be ported, yeah.

B

And then, of course, we do work either with or for the CNC F, and so you know, kind of was kind of need to be not running a model with the gap. You know we need to run this on kubernetes anyway. It was built before that was really reasonable thing to do, but now need to actually do that. So there were a few steps involved. Yeah.

C

So basically, we needed to the biggest problem to solve was how to store our data, so we needed to switch from local storage to some cloud provider. In our case, we chose AWS just just for this example, but we're trying to avoid any cloud lock. So basically, it can be used from with Google cloud with anything. So we use EBS volumes. We dock arised every part of deaths that so we have deaths, that's image. We have graph on an image. We have cron job image.

C

We have image for test, we replace local Postgres with our solution, Patroni, which is a high availability. Today's cluster on three nodes. Currently we created a separate services for read-only access and for a read/write access to take advantage of read replication. And finally, we used help for template everything and to allow adding any new projects in a very easy way in just a single diamo file, and it will just generate templates.

C

Kubernetes manifests for every project.

B

Needed were into a couple of problems in the course of making this work. Yeah.

C

Yeah so basically, the biggest problem was with docker, which was limiting a memorial short memory available for pot to 64 megabytes. So this is obviously a very low value for for a database. So we did a trick. That's that's also used by OpenShift. We mounted a memory volume under the def s hm directory, which is Mac which basically gives port access to entire Ram.

C

Of course, we are using limits and requests limitation kubernetes on every pot to not allow, for example, nodes to be killed by too much around usage now, but basically, another problem was with permissions related to storage, because cloud provider, in this case AWS most our disk with root access, while the Postgres database runs as a Postgres user. So we needed to handle that by patching Patroni installation and also by running the container and special effects group mode for for the possible user and.

B

Also so one of the other things is, we just have two big packet net servers and we actually need to split that out into more separate nodes in order to have say three nodes of redundancy for Postgres other redundancy for other things, the and the final thing was for both of us.

B

This was actually our first production application in eks and there was a little bit of a learning curve there there just some things that amazon assumes that you know about the AWS environment, that you don't necessarily know if you haven't done it before or even in my case, visibly haven't deployed a production application in a TBS in a couple of years, so we ran into and had to solve those. So it's now ported over so and because one to show this example app.

B

Let's talk a little bit about what we have in there right so, like I said, we wanted to give examples of most of the things that you would need for a production kubernetes application, because we actually need most these things right. So that includes ingress for access to things. Multiple services, a stateful, set, a replica set actually multiple replica sets in a production deployment volumes, cron jobs, secrets, SSL Certificates helm, which is what we used for deployment. I are back for setting some special permissions and also tests for the whole platform.

B

The technologies involved include ingress, EOB and then in ngns ingress for the ingress solution using the AWS cloud provider in this installation, of course, because we're on on eks griffon, a--, Postgres, Travis CI for CI update, github, docker, hub AWS and then the wonderful, let's encrypt certificate manager, we'll talk about more in a minute. This is a very simple version of what the general application structure is right. So the idea here is on the Left. We have contributions coming in to projects via get and github. We then take a whole bunch of contribution information.

B

We put it into Postgres database, where we massage it into in analytics format the that is then displayed via graph fauna, which people access through in nginx ingress access in the front-end. Also in the backend changes to the data structure and the generated reports, I come from github and Travis CI tests them, and then we can deploy them to Postgres sets sort of our basic application structure. There's a bunch of refinement in this again we'll get more into that. This lives in three different repositories. Yes,.

C

That starts initially was our one big repository, but after time we think that there are many different deployment modes like bare metal, kubernetes, hands and so on. So we divided it into containers report. It contains docker images for every single part like provisioning image that is needed. That is started only once to provisional data, our listing cron job image, tests, image that build other image and execute test on get ops using Travis, CI graph owner image and finally, Patroni database image.

C

Another another is a charts, repo charts repo is, where the hand chart lives when child lives in it defines all the loads to to deploy all projects define it here and finally, we have a destiny report. This is the old repo where all the code was used to live. It's split in two parts right now one is just a source code needed to build and another one is deploying deployment related stuff that can be used for bare-metal deployments and other things like that.

B

Ok, so, like I, said we're using help for deployment right now, one of the things that are warn you is when you look at the actual helmet arts you're going to find that they are a little bit complicated. Let me show you. The post-credits portion of the example is both there's a bunch of exception code in there to allow you to swap out specific components, because, like one of the things, one of the ability to do is decide between hosting our own Postgres and using something like RDS, because we were on the fence about that.

B

The hosting on post quiz worked great and we decided to stick with that, but we wanted to be able to swap back out, there's also a bunch of loop code in there for the Gravano builders, because we have a single graphic. We have a separate group on a pod for each different project who stats get displayed and for that reason we generate a whole bunch of them and that all is done through Lupin code in the helm templates.

B

So, just in the case of when you actually look at the helm source code itself, it's gonna be a little hard to read. For those reasons.

B

To do so deployment platform that were on top of right now, kubernetes 112, because one eks and that's what eks has is goober Nettie's 112. I it's designed to be fully portable, though um you know I, because we don't necessarily want to be tied to a specific cloud. The main reason for doing eks were you know, familiar with AWS. Otherwise, the we had I went to free eks credits did come into that and then, finally, in the case of why we would want a major cloud provider to begin with, is just that.

B

It was a much easier way for us to handle storage and load balancing right. It's because the cloud provider provide those, whereas otherwise you have to do something like you know, sand or Seth, plus IP vs. In order to have those, however, we are looking at potentially moving back to a more better metal engine.

C

Yes, yes, exactly, yes, exactly where we just need at an example for cube contact, so we don't have enough time to create a full bare metal deployment, but our aim is to, for example, we can even reuse our current bare metal servers that are serving deaths. That's right now we can make them two nodes, for example, and just take another few packet net servers to create a fully bare metal deployment. Of course, the problem will be with is grace and with the storage, but we would probably use local storage from this case yep. So.

B

Let's talk about some components here and I'm gonna be flipping back and forth because we have a whole lot of llamó here. So here is the PV setup. This is actually the helm. Template I'll be showing you some helm templates and some generated files depending on which component is but these this is the helm template for the various TVs in here the because they're, pretty generic since we're using cloud storage, yeah.

C

So this is a every single project. You need to storage for get clones because, in addition to get copy events, we also analyzing particular commits feel files. So you can so every single project need its own persistent volume crime for good storage. So this is why we have it. We also have free provision, free, persistent volumes for Patroni database, one for master and two for slave nodes. Yeah.

B

So speaking, the database first of you who aren't familiar with Patroni in the setup that sort of thing is this- isn't kubernetes based fully automated h, a setup for Postgres that, where you have one master, node and any number of reed replicas at a given time with automated leader election, so that you can fail over and then that attaches to one pv per each individual Patroni node.

B

We have three in the sort of default setup the so this includes a stateful set for postcards, and let me actually just show you that instead, there's the generate is the these stateful set and basically it's you're, pretty straightforward general stateful set the that pulls in there. The one thing the aux we see in the generated files is there's a whole bunch of command arguments which are all Patroni and postcards configuration variables that all go in there in order to configure it via helm variables.

B

But the only sort of weird thing we had to do here was both file system configuration group access and then setting up shared memories of volume. Like we mentioned earlier,.

B

There are a couple of dependencies in this which actually I think I can show you in here yeah. This is why I'm using the template circular to show you the dependencies, because the generated files, it's one per object, so we've got a couple of services.

B

Obviously, we've got the master service and then we have the we'd only replicas service that we use for the public portion, the application and then there's also some are back stuff because in order to control failover automatically Patroni manipulates kubernetes endpoints, which is an object type that you don't often talk about, because it's generally implicitly created by kubernetes service.

B

But you can't actually explicitly manipulate it as we do for Patroni in order to make sure that the connection to the master always goes to, whoever is the current master and thus prevents us from having split-brain and so you'll see in there granting Patroni the ability to create and manipulate both config Maps and endpoints.

B

Now the data actually comes from a variety of sources as.

C

I mentioned at the beginning, there are good github, arches and github API. All this data is is checked and update every hour by the kubernetes cron job, and it then run a bunch of metric time series generated program that also saved this data into possible database because we are using Postgres as our train service database. And finally, this data got displayed from graph on a-- using real applications. There are more than one replicas of graph Anna even for single project, and they just can take advantage of our high everbilly availability of the database yeah and.

B

So that's called in a kubernetes cron job, so we've got a cron job object in there that pulls the hourly sync image with again a whole bunch of variables in there. In order to do it.

B

Yeah, where you talked about that.

C

Ok, so the graph owner, we were not able to use the standard graph owner installation because we have a lot of custom changes to graph Anna. So we decided to create graph on our justice graph on an image that provision itself from a set of template variables like database access like dashboards Jason's. That needs to be displayed, and we are also customizing some artwork like icons and links and things so on.

C

So this is this is how graph owner is created and if you want to add another dashboard, you are just update the image and the kubernetes will fire. The rolling update and nuclear fauna will be available once it's finished.

B

Yeah and as I mentioned, we get a separate brief on a pod for each hosted project for each project whose community stats are going to be displayed, and for that reason, if you actually look at the generated files from helm, what you'll see is you'll see a whole series of deployments here like in this case. This is for the Prometheus project.

B

Do we're doing more than one here? Is it just Prometheus? Oh, you just did Prometheus yes, but.

C

B

Right, it's labeled Prometheus in there, but if you were actually hosting five or six different projects, you would see five or six different replica sets. There.

C

Yeah, so the secret files we are, we can easy with that one. We don't have any fancy security management right now. We just have a bunch of files ending with dot secret that are not checked in into the repo, and we are. We have corresponding files, ending with example. So before you actually can install the home chart, you need to take the example files and for each of them you need to provide your own values. They are just.

C

They are used internal by the cluster, but you need, for example, provide github, appliques and passwords for graph ana for Postma's and so on. Yeah.

B

And then the and then it'll generate the actual secrets calls for populating those.

C

Engine controller we are using, we are using nginx to road traffic to a specific graph. On our instance, we are, we are registering a domain with wildcard domains. So, for example, we have def starts demo dotnet, and we every single sub domain from this is routed to our AWS ELB endpoint and data data. Endpoint is used by by the nginx and nginx takes a look. What is your host name? What is your sub domain name and based of that?

C

It directs the the traffic to a specific projects graph, and we are also using certain manager, which is a very good program it once you have a domain once you have a domain name, you can use it to automatically use, let's say script and automatically renewal certificates.

C

Yeah, so also testing, so testing is not yet finished, but the CI part is ready. So we have Travis a job that is a favorite on github and on every key tops. We are building images and building test image that is executing the test and returning the final state. We are missing the continuous deployment part there was.

C

It was implemented on the bare metal solution, but right now, but our solution was not good enough for kubernetes, so we are considering to switch possibly to some other CI C D and V Ron environment, but it's not ready yet yeah.

B

So in case, any of you actually want to deploy dev sets itself, which you might, if you're involved with other business projects, that sort of thing just a few notes on actually running it in production. What you would actually need to do to take the code that we have in order to run in production, yeah.

C

So first things you need to do is to have some domain registry. We used AWS wrote 53:4 for our example, but it can be anything. Then you, then you need to have some ingress controller. In our case it was lv from AWS and finally, you need something that to handle certificates. In our case, we used open source programs, certain manager that uses let's encrypt under the hood yeah.

B

Now one of the things that it is set up for and like I said, the reason for some of those complicated home charts was. It was designed to be a somewhat flexible deployment. I mean honestly. It was because we wanted to be able to change our minds in the future about exactly how this was installed and where it was installed. So including things like.

B

If you don't feel comfortable running a self-hosted database, then you might want to swap it out for RDS or digitalocean Postgres or whatever else is available on your platform, and so you can do that. You might not want to use graph on if your goal is to just generate analytics data that is going to be consumed by some platform other than Griffon a--. It's also set up to do that, including output to elastic search as another option. Yeah.

C

So we had some requests to create a special instance for death cells and that instance was, for example, using Aurora DB. So this is a managed possible solution. It wasn't using graph owner at all and it was using elastic search as a final store for data from this stuff. Yep.

B

So still work-in-progress stuff, that you are welcome to submit stuff to contribute to obviously number one thing is documentation. You know there's some degree. These charts and stuff are supposed to be self documenting, but nothing is really self documenting. So we need to actually write up the purposes of all of the components and how they work internally, etc. We need to register the helm chart on Quade, I/o or another service so that you can install it without a git.

B

Clone doesn't currently have any monitoring built in that's another major sort of you know: production application example thing, and so we need at least have some basic monitoring and there, via Prometheus or otherwise, because I work for Red Hat I'm, also going to create an example operator out of this using the operator development kit talking about doing improve, CI CD in here and then maybe look at having a more sophisticated solution for secrets management. You know vault or otherwise, and one.

C

Thing that is sure for now we are moving our current deployment for our CN CF projects from bare metal to kubernetes. So it's not yet decide if we will use AWS or we will go just bare metal I think we will go with the latter with the latter, but it will. We will see yep.

B

And, of course, the big thing is we're really interested in people who are interested in contributing to this. We're interested in using dev stats are interesting in tributing to it, either as dev stats, because you want the community stats and the development stats or, as an example, application right, if you want to say, hey I, don't actually care about dev stats, but do care about sequence management. So I want to have this robust implementation of secrets management as an example to new kubernetes users.

B

We would love to have that because there's some things we probably won't get around to doing, because we don't actually need them in production. So, given that we actually have you know seven eight minutes for some questions, here's links to all of the repos and everything else. So do you have questions.

B

Yeah, so his question was the experience of decomposing the monolithic application into a bunch of services and talking specifically about having a lot a lot like hundreds of services, I I, don't think the original application was that complicated. Now.

C

So the funny part is that this application was created by CN, CF and CN. Cf is a home for kubernetes. So since the real beginning, I was considering kubernetes as a final platform, so we of course needed to split a bit of parts, but basically the main dev start was built with kubernetes in mind since the rear begin. So, for example, every single Benari, like bernard for data sources like bernard that is calculating matrix and all other stuff. They were all separate and they were communicating through the database service.

C

That was also configurable from the real beginning by the environment variables. So it wasn't that hard, because that was the final target from the scenes beginning.

B

So question was how we assigned specific contribution, both in terms of how do we know who contributed something and then asking about the company attribution of contributions. So.

C

We have our separate ACF project, which is CN CF, get DM or get data mine, and this one is trying to get a big list of people and their companions. They were working for so, for example, for single github user. We can have a list of companies and the data ranges they were working for a given company. For this is all the manual process.

C

We will train automatic tools, there are a lot of them and our outcome from this is that they help a bit, but we also get a lot of noise and trash that we need to manually filter. Currently, there is a contractor that is working only on this part, so we are constantly researching using LinkedIn, Facebook, Google Twitter wherever and trying to find affiliations. And finally, we are importing the big JSON file with those affiliations about twice per month and updating data.

B

Okay, there was another question.

B

B

Okay question was how we deal with gdpr and whether or not somebody wants to be delisted that that has not been our problem so far in terms with first part of GDP, our right is not. Privacy has not been our problem, because the data sources we draw from are all already public right, we're drawing from github, where the information that we're drawing is already public. The second part of it, which is the right of removal theoretically, were prepared to effectively anonymize somebody in the database.

B

So far we haven't had to do it as far as I know we haven't had any, but we did.

C

Yes, so, okay, there was a problem with gdpr, but our solution was exactly like. Josh said this data is public, so we don't need to hide this, but some people may want their data not to show, and we have a separate section in documentation. How can you create a pull request that requests removal of your data and- and we have a special, a binary program that will scan our data and remove that replacing this with some some special marker that we don't have that data?

C

So we actually have one or two cases of people ask us for remove their affiliation data from CN CF gdm, and we just did that and that's it.

B

Now, what was the hardest part in moving everything to kubernetes? Well,.

C

That that will that will again be a strange answer. I, don't think we have a hard part I. It was a big fun for me. I am actually so amazed. How kubernetes make everything easier for me, for example, I, don't need to care for an operating system anymore, I, don't need to care about, in this case, storage and many other things.

C

So, for example, when deploying new dashboards I, don't need to stop gravano I am just creating a rolling update which creates another one and because it was written since, since the real beginning in kubernetes in mind, I didn't really had biggest problem other than this, too is small issues with short memory and permissions. Every everything color was just good for me.

B

Any more questions and.

C

The hardest part, the hardest part of all of this I think is a database management and persistent volumes for for stateful application, but it wasn't hard at all for me, but if I need to choose the hardest part, this was it.

B

What so we one there and one here and then I think we'll be done yeah so one there.

B

So how big of a box do you need to run a test deployment of dev stats? But if you don't want how much memory would we need memory, yeah I? Don't it wouldn't be? It wouldn't be big I mean I! Think honestly, something like in a in M 3, extra, larger and 4 extra larges. Actually.

C

Yes, actually, it depends on the project that that's the the best answer, I think, because if you want to deploy kubernetes, which has 300 triples and millions of events, you will need a lot of memory. But if you want to deploy something small, you will just be fine with a single machine. Ec2 default instance. I. Think and then.

B

Last question: a.

C

Connection pool for database I mean.

B

We rate we're not actually doing connection pooling, because we have a simple graph on a prod for every project- the oh right, because it could pooling those connections for that right. That hasn't been an issue. I would inject a PG bouncer pod in there if it became an issue, but so far we never actually have more than 100. Simultaneous connections which is easily handled just through the post was default. Yes,.

C

And our bare-metal deployment actually I needed to tweak few Postgres variables like maximum connections short buffers and things like that. But our bare metal server is a beast beefy, so powerful that even Q Burnett says it's not a problem for again. So.

B

Thank you very much.