Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2016 (Seattle), 16 Nov 2016

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Migration of NCSOFT Game Servers to Kubernetes by Kang Jeongsik

Description

Migration of NCSOFT Game Servers to Kubernetes - Kang Jeongsik, Kim Junghyun, & Son Junho, NCSOFT

" NCSOFT, established in 1997 and headquartered in South Korea, is a key leader in online games with its flagship product, Lineage. Today, NCSOFT’s reach has expanded worldwide including locations in Korea, China, Japan, the UK, and the US. The company operates many of the most successful MMO games that have been enjoyed by tens of millions of players around the world.
NCSOFT has been successfully operating multiple game services with thousands of physical and virtual servers for 20 years. But inefficient use of servers grew gradually as time went by, and there was also the need to respond quickly to temporary server increments for marketing purposes. Moreover, fast provisioning and deployment systems were required to prepare for mobile games.
For this reason, we have prepared for the introduction of Container technology from late last year. As our first target, we chose a number of stateless web servers and API servers that offer common functions. The Infrastructure configuration consists of OpenStack, Kubernetes and Docker; Kubernetes and Docker were built upon OpenStack. We changed the existing deployment system to be compatible with Docker/Kubernetes, and under careful consideration, we underwent tests and gradually applied the changes. And that brought us to a successful migration, with the results showing no problems at all.
The results show that existing VM amount decreased rapidly from 52 to 8, and unnecessary processes like creating VMs and installing software one by one on each VM were also removed. Furthermore, it enabled doing Rolling Updates dozens of times a day without any service interruption.
At the moment, we are preparing a new mobile game to be run on Kubernetes. Regarding continuous deployment on hybrid environments such as OpenStack, AWS and GCP, we are internally developing APIs for our hybrid environments. "

About Kang Jeongsik
I joined NCSOFT in 2009, and I have been working as a software engineer/architect since then. My interests lie in the application of Open Source concepts and methodologies in corporate environments. Over the past years I have been doing work related to the Hadoop ecosystem, and at the moment I am working on the Container Infra as a part of DevOps.

About Kim Junghyun
Director, Infrastructure Division, NCSOFT

About Son Junho
Software Engineer, NCSOFT
Hello~!
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Okay, I'm chun-hyang came from and she stopped gaming company in Korea today, I'm together with my colleagues Jung's again, you know here and we work at infrastructure infrastructure division today, I'm going to introduce our first experience of adopting container and kubernetes technology for some of our game services.

A

Okay, I, don't know how much information you have about Korean games, but Korea has the fourth largest gaming market in the world, and we and she stopped- is providing very popular MMORPG PC games, mostly for about twenty years now, and also providing game services in the US, UK and other countries to.

A

Our infrastructure are mostly based on on-premise physical machines and virtual machines, and we, our server applications, have been optimized for many years. We infrastructure team also optimized our servers and network for the game services too, and we also know the traffic patterns weekly daily our lives, so in short, I can say that our infrastructure and operation has been quite stable, but we've got a challenge that started from a few years ago. Sorry, it's from mobile games. The mobile game market is growing very fast, so ncsoft is also beginning mobile game business.

A

But mobile games are quite different from PC games. The life cycle is very shorter than PC games. The second chart shows that half of mobile games or mobile apps lose half of their peak losers just within three months. Three months and workload of mobile game are much more dynamic so for mobile games we impress the rocket team should be ready for more game, more new game service launches and more frequent service updates or release, and we should be also more ready for more dynamic Walker. So it's a I think it's a big challenge for us.

A

So our approach to handle DS is adopting hybrid cloud and container technology with hybrid cloud.

A

We are going to provide infrastructure for mobile game services for the static portion of the workload we prefer using private cloud because we believe it is more cost efficient in the long time and further a dynamic portion of the work out. We use AWS and Google cloud.

A

Like many other companies, our choice for the private cloud ease opens there.

A

But hybrid cloud is nice, but there is one problem: the operation can be very complex because there are no various operation truths from multiple providers and service configuration and deployment should be customized for each environment, and here we believe container and kubernetes can deduce the complexity.

A

If you containerize your own application, you can deploy it anywhere with kubernetes. You can manage the container ID the services more consistently over multiple kinds of infrastructure.

A

There are a few available orchestration tools, but we prepare coronaries because it is very active, open source project evolving very fast, and we especially like the way of service configuration of management of kubernetes. It's very well organized and easy to manage I think so we set up our first criminals over opens there, and many people ask why kubernetes over opens. Then my boss asked if it is and I have a I, have two answers for this.

A

One OpenStack is just one of them because you are going to provide container clusters of multiple infrastructure, so we will expand it over Google, AWS or bare mirrors later secondary. We think the combination of kubernetes and OpenStack is nice, with open. Still you can provision compute, storage and network with kubernetes. You can provision services. So again, the combination is very nice for the whole operation.

A

And here I'd like to share some our technical discussions we had during the configuration of our culinary server opens there I'm not sure how much it would be interesting to you for still I'm sure. If you setup crew Vanaras over Philco machines, each node is actually a critical load right. So it's node will have guaranteed capacity as the capacity of hardware.

A

But if you set up kubernetes over options there, it's not each bottle machine and a potent emotion may have less capacity if there is very busy noisy neighbor together with over same physical machine, but kubernetes scheduler does not consider the real capacity of each node when it does the pod scheduling. So that means some part may be deployed through kubernetes node, which does not have best capacity at that time.

A

So question is: we need to handle this situation or not so one way is that just keeping it that it is, and then you monitor the usage of each node and the scaling of the nose as needed. Actually we are doing like this. We are actually implementing now auto scaling using open, stable time or if you want guaranteed capacity which North cubanelles node, you may do some dedications. You can dedicate physical machines only for cubanelles nose or you can dedicate same number of physical cores which grenades node.

A

And second second thing is: was about networking option? Actually, at first we photo kubernetes. We just used the deport networking option planet. We excellent for the kubernetes networking, but we found that the vendor ease was not so good in our environment and we got a good tip from Google engineer that we can use another option host the Gateway and we found that the bandwidth is nice, like other cases, so we are using it.

A

But later we also found that we can enhance the values of app react land by changing the value of MTU maximum times a unit like this. So the learning we got here is that we need tests like this when you set up kubernetes over cloud infrastructure.

A

A

So we set up our kubernetes like that and it's time to migrate, some existing services.

A

Usually, a game service consists of much smaller services, including login service, large service world service, like that- and there are also clapham services which provide common functionality for all the game services, but it is not practically possible to containerize all of them. In our case, some services are running in Windows OS and some services are stateful and non-scalable by design, for example, game world each world had just its unique concept and different Terra unique Terra. So it's an all, so it's non scheduled by design.

A

So we try to find state list and innovation services for our migration and we found 16 such services for our first migration and we walked with the service development team and gave some cards so that they can adjust the period and deployment system according to crew, Benares, environment.

A

And we, and also we wanted to finish our first migration without any problem so that we can invite more customer of our new environment later so we just kept initially, we just kept the previous old virtual machines that they were and we added each kubernetes node to the existing load balancer using North, auto ship, and then we monitored them to verify that the new kubernetes side handlers services as expected, then when then, we removed the old virtual machines, so the migration was smooth, no service, stop.

A

So rigid its or saving the total volume was not be no, but I. Think the the gap is quite big here and I think this sort saving is not from the technology change, actually I. Think it's from the change of concept or or work process before kubernetes environment.

A

Each service team requested some number of virtual machines to impress rejecting, but the average usage is not so high right, but in kubernetes environment there is no concept of motion in service level. There is just service impart and it is very easy to scare out they are doing scare out by them service very easily, so they not be cast as much resources.

A

And as I mentioned, the scale-out process is very simple before we had to create a new virtual machine, install application and add the virtual machine to load-balancing guru. Some steps may be require may require manual process right, so multiple steps. But now it's just one step: 1c I recommend QT scale or you don't need any step if you use Auto scale option, so it's very good for operation.

A

Okay, so now logging and monitoring, I think logging and monitoring our mandatory systems for service operation. So let me share our configuration for those systems.

A

If you ma, if you are going to migrate some existing services, there may be many kinds of love files. Some files are custom log files from applications, so we we try to find our best handling way for the low price and the two questions regard first, the location of log agent in our case bluntly second question the location of log files and about the locational log agent.

A

There. There are some options you may put an agent in each container or you can put it in each part or in each node. In our case, we decided to put each agent for each part like this slide. You can see that each flan each part has its own plant e set as a separate container and all other application containers right. Log files in a directory which is provided in using anteater volume and empty the volume, is accessible from all the containers part. So plenty container can read the all row.

A

Files can send them to the elasticsearch, so I think it's a kind of a scalable way for this handling of log files.

A

Monitoring we have been using javac for our infrastructure monitoring and for coronaries monitoring. At first we tried to write some custom scripts. It collects paging matrix using kubernetes and our KPIs, but we soon found out that from a toast provides much better monitoring features. So we are now using zabbix and parameters together and also we are using macrofauna for the integrated dashboard so that we can see all the infrastructure metrics from Jagex and proven arithmetic from from math from metals. In a single page.

A

Next step as I had you so our first migration was NASA P and it was pasture, and now we are working with the game development team. They are developing new mobile game, which is fully containerized and a stateless, scalable and Linux based thing. They are going to launch this service in Crewe Benares environment soon over OpenTable AWS. So I hope that we can share that experience too soon.

A

So this is last slide, as I mentioned that earlier, as I mentioned all your sorry, there is complexity problem with hybrid cloud, so we have our own integrated approach, inning system to reduce the complexity.

A

Our current system provides our own standard, API and ui-managed the highest layer consistently and now we are adding features to manage kubernetes cluster and nodes, and also we are going to add features to deploy application, binaries or container images. So I hope that we can have a really integrated provisioning system from is layer to the serve slave.

A

Okay, it's or I have prepared for today. I hope that this was implemented for you.

B

A

Wheaton, we are not using Magnum or any OpenStack related solution. We are actually we we prepare using pure I, mean independent solution, because we are going to provide kubernetes crest over what the kind of.

A

Sorry but testing about.

B

A

B

B

A

So your your questions, why we put an agent for each part instead of no, because each service has eat your own custom love files. So we need to put some passing rules for service, so I think it's more better to for the service vellum team to handle the passing part for free Lundy for its service. That's why we really set up like this.

C

How are you ready, how are you man are they.

C

A

A

I'm not sure that pot.

A

He said the pot really started processing apart freshness.

A

A

A

So actually I am NOT on hands-on engineer so I'm, because that's why I'm here with my colleague here.

A

He said we are using open D. We.

C

C

A

Actually, we didn't find such issues, yet we have been operating this cluster now about four or five months and no issues like that.

A

That that's good question actually, but debugging means the subs debugging right, yeah, yeah I, don't have many information about that yeah, but now I think subs develop. Each development team is handling each service by themselves more than people, so will not know how they changed the debugging process. With this, but in infrastructure layer we are only focusing on the kubernetes cluster serve Logie, sir, so I think it's more separated.

A

Performance metrics about.

A

Yeah, actually, the average usage of the node is not so high yeah. Actually, we started with Sept 1617 I said, but now we have about 30 services over 13 provenance node, which is Porter machines, but actually that we, as I mentioned the main.

A

The game servers not migrate to the air. So usage is not so high yep.

A

Ok. Ok. Thank you. Thank you very much.