Cloud Native Computing Foundation EnvoyCon 2018 (Seattle), 16 Jan 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Building + operating a service mesh at a mid-size company - Taiki Ono, Cookpad Inc.

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Building + operating a service mesh at a mid-size company - Taiki Ono, Cookpad Inc.

Service meshes are not only for big companies. It works for intermediate-sized companies as well. We, Cookpad, are a mid-sized company having 200+ developers, 10+ teams, 90M monthly users.

We are building our own service mesh using Envoy proxy for our microservices. And, we are getting the benefit of service mesh. The biggest benefit is observability: we can easily know how our microservices are working by looking into fine-grained metrics per upstream service. The second benefit is process model independency: we now have the outbound proxy for every app which controls retries, timeouts, circuit breaking and load balancing.

I will share the knowledge and techniques through our case: how did we build, and our architecture and toolchains. We are using AWS ECS as a container orchestration, and our service mesh is built on it.

A

Okay, hi everyone, I'm Tai Chi from Tokyo, Japan and I'm, working, quick part and mu story. I am well and where users and sometimes I, send patches or documentation fixes. So you might have seen me on github and I'm glad to share our case study of an boy in which we build simple sub smash with em boy. So please enjoy our story. Okay,.

A

This is our agenda for this session. I think the most interesting part is architecture, operation spot, so Remi quickly explained background and programs then go to the architecture and operations part so background. Let me explain about our business to tree. We are doing in cooking domain and the most famous product is sharing service to koat.com. We originally studied in japan and started to grow apart home a few years ago, so we have separated proton in JP and Grover right now.

A

But let me skip the detail in this session for simplicity, so our scale and structure we have around 200 product developers and around 100 production services, and my team is proteome development team and it has three members. According me, I'm the only person who developed our service initiative and regarding our structure, we had some all production and one center operations team and the center operation team still has all of our operations. But we are changing this now so so that why we are building up to achieve service operations.

A

An odd technology stack is almost Ruby and Ruby on Rails, most of our application, written with Ruby Ruby on Rails for both front-end and back-end apps.

A

But recently we have some variety in application, run gauge right person in machine running and go on Russ in performance required domain, and the right diagram is a typical architecture before most of our applications are running on Amazon ECS. So we have some internet-facing load balancers, which was associated to front end ups and front end ups communicated back end up sphere. Internal rod versus the oral internal communication were praying adjacent over h1. So it's very simple: okay, see our programs.

A

Those are typical programs around micro services, so you know them system, reliability, program and observability program. So we studied a library approach at first made. History strike reverie in Ruby and may distribute tracing library. But yes, as you know, the library approach has a limitation and it won't work for us. So we plan to develop a proxy like as a proxy because we work every using as a proxy at that time and pronto the proxies with consultant parade or something like that.

A

Then we met em boy. It works right what we wanted to build, so we stopped investigation and sought making plans to build our subs mesh with em boy.

A

So this is a timeline we start making plan on designing architecture in re, 2017 and I was back to selfish in rate 2020, then start breeding MVP and using the MVP in production rooms of time and finish, making our provinces mesh in re, 2018.

A

Which is a house controlled brain approach? It's because the timing- and we are using Amazon's ECS, so I looked up some product at the time, but it seems no product with our environment and requirements.

A

I, don't have suspect information at that time, but I solved. There will be some management services in the future. So yeah I choose our small shot with in-house one, and there are some considerations on building our service mesh.

A

We wanted to merge, sub image configuration century and wanted to do get ops with code review an X because we are operating primitive service or matrix from my should go into primitive servers and we have to keep the operation cost lower level as possible. So we pre harass components and use of managed services as its back-end.

A

Before interest is before introducing the architecture, let me explain some key component. The first one is the kimonos. It is a tool to generate. We want exodus responses from our abstract sub image configuration.

A

This is improvement as a revision on the next we developed SDS application. It may are confusing, but yes, it's based on Leafs discovery application, but we added some futures on it. This is implemented as a rust web app and the rust the attach all to this is a basically our Syria RC a right to to generate with Exodus responses, but it also has the future to translate with the rest, HTTP POST request to HTTP GET request.

A

The translation protocol is our original funa, so I'm going to work for that protocol in the github issue. If you have any comments about this, let's discuss okay. So, let's start the architecture at first resort with above in this diagram, there is a central github repo for abstract service, mesh configuration, a product, developer, can add or change or remove the configuration and push them to github repo.

A

Of course we can the code rebels before pushing here and when we push the configuration, a CI task runs and it generates women, Sidious and RDS responses, then uploads to them to our s3 bucket.

A

Then Sadko em boys are running in each or ECS task, which is right. Cooper, net sports and the invoice are configured to see or a silly bucket. We see red arrows, so invoice can fetch the across those routes configuration from the s3 and in the city s response. We press dns name of our intern year, V the cross. Remember so amboy's, just rhetoric to the entire internal Rhodope answer.

A

Here's rotor bouncers does rotor, balancing and and 0.2 discovery and her shaking and so on. We simply change application configuration to refer from internal ear B to Sudoku m'boy. We don't use IP terrorists here, so the traffic from application container was the Sudoku m'boy en route to entirely Ruby. Then upstream, engine X. In this way our friend Jason over each one service communications were working in our women subs mesh.

A

This is given and so red C is a configuration file. It obstructs and voice configuration. We write good configuration and Casta configuration and it's a JSON file and single JSON that fire represents in results configuration as hobbies can have an upstream crosses in upstream dependencies.

A

It's almost same as my configuration, but we do some additional work to inject rhetoric on relation to get request and inject some host earlier I created computation to fit our info infrastructure.

A

So this is our service configuration and they will be converted to exist decimal seeds, which are two and next. This is V. One point: one: we studied using Bremen service discovery service, API to build infrastructure for the RPC applications at first, let's see the SDS, it's our customized. Women assist API server and its back. When is denim. Db then see the blue arrows at right of this diagram.

A

We run sidecar distrito, which which do health checking of the application register and the point information to the SDS sour, then Sanka, Amboy fetches, and the points from v1 STDs API from the sdes app and the traffic are routine directed up streams with invoice nor the warrants in future. There are no so beside Rosa burns out here, but, as you know, we now have a diverse crowd map, so we we don't have to do this money registration now, so yeah ideally wanted Kodama before okay.

A

This is our latest version, which is using which XDS please look the top of this diagram to the trans rate, which will exist. Hp post request to HTTP GET request for our s3 bucket. We run translation web server between em boy and s3.

A

It's a attach of Saba. The translation server is running as our normal ECS service, so it's hard to use with the RPC X years, so we are using which the rest exid petrest XE is here. We can simply use other ways, not talk lot of answer because of our least cost BBC issues for brittle ideas. We implemented brittle EDS api to our custom, SDS abba.

A

So this relation the registration are same before.

A

Okay, so this is our subs mesh architecture. Then, let's see matrix side.

A

This is how we are sending matrix. The primitive servers we are using dog starts decent to send our matrix in dr. D format, which is an extension of study format and then aromatisse study exporter are running on each horse. So once that d1 starts the exporter process, one ECS host instance, then the stasi exporter export our receives invoice matrix from site oka invoice.

A

Our premier servers are configured to discovered those study, exporters and parametric's from such the exporter.

A

Regarding the metric stock, we, as for specific tags to all of our matrix to identity, identify each network.

A

Okay, the initial operation side to make dashboard. We are using graph honor. We have two different dashboard for product developers, pathologies and / service to service for the was generated using grahas temperate feature, and we have one dedicated dashboard to Malta our amber instances, and we have a visitor dashboard to the quick work, our networked, 4G and service status.

A

We are using a problem from this and from this phone.

A

This is a pacifist dashboard and we can check upstream RPS responses Ritter eyes here, and this is also a possible dashboard at this time.

A

The upstream was broken, so we can see my sweetie right over her counter was charged for the specific upstream- and this is my Grove our dashboard, and we can check x, vs update status, mainly here- and this is our visitor dashboard in this dashboard- a single white dot, which is very small in this picture, but the white dot represents our request and it moves from a node to another node and if the request ends with errors, that dot becomes a red dot.

A

Seeing this dashboard is our little bit fun, but so I'm, sorry for not preparing a movie for this. It's a yes dots are moving and it's very fun.

A

This is our passives view and down streams and up streams and dots are yeah, of course, moving. We are not heavily using this business. Our dashboard, but sometimes the dash was, gives us a quick inside of our entire service topology, and we have in-house proteome console which shows everything about a service in a single view, so the river are developed, parts can be serviced at aspire just looking at this page. So we add some information links to this page.

A

Let's mention about how we use amboy on ec2 instances, a few sub is still running on ec2, so we run my processes on using instances. We build em boy as in-house david package disturb you distributed it with configuration management. We manage my process are the system of the service. We are using our invoice. Hot wizard got pie, script.

A

This is a one person or program we faced. This is waiting issues for ensure XTS set up in the background jobs. We can't wait service in with harris checking, so sometimes application container runs before insured XDS settle to handle this issue. We develop rapper, sierra to which raged insured Axios, set up with her shaking em boy and after waiting, succeed, exact application process.

A

There are some hard points, but we finally found hundreds, and so it's just for information.

A

Ok, so let me show some key result from our services quickly. The first one is observability and the second one is pure recovery. We have significant benefits in these areas and because we are now free managing managing our libraries. A proton team have more control of a proton component, a proton component deployments, so it increases our team thirsty.

A

Ok, next challenges we studied building iv injection proton to test or experiment of applications on service mesh configurations. We are planning to migrate, managed services because some money services some manage the services mesh services are released. Now so it's time to consider the migration, then. Yes, thank you for listening and I'm. Also reading for to hearing are your cases and stories so I'm glad if I can talk with you at reception order in cubicle. Thank you again. I'm done.

A

ha Go ahead, yeah.

B

Pretty piece a sorority and cranny.

A

A

Oh yeah, it's a pouring. Yes, we won't exceed yes and with the rest, Axios is pouring. So it's pouring yeah.

A

Yes, it's it's not necessary to separate, but yeah, it's just a historical reason. So yeah we we first develop STS as web application, Russell web application, and so yes generation, 2 improvement. Us go run serious, so yeah.

A

Okay, thank you.