Ceph Cephalocon Barcelona 2019, 24 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Upgrade and Scale-Out an In-production Ceph Cluster on Mixed Arm Micro-server Platform - Aaron Joue

Description

Upgrade and Scale-Out an In-production Ceph Cluster on Mixed Arm Micro-server Platform (with OpenStack in Telco) - Aaron Joue, Ambedded Technology

A case sharing on Ambedded's experience to upgrade and scale-out an in-production Ceph Cluster, with its mixed Arm server platform. In this session, Ambedded will share how to overcome the challenges and the way to implement without service impact.

(1) Initial condition: Ceph cluster with OpenStack, running more than 140VMs for varies applications.
(2) Performance & Workload analysis
(3) Problem discovery
(4) Challenges: Migrate and Scale-Out an in-production Ceph cluster on mixed Arm platform
- Change the operating system from Debian to CentOS
- Mixed 32bit & 64bit Arm platform
- Upgrade Ceph version from Jewel/FileStore to Luminous/BlueStore
- Upgrade Ceph management GUI version from Jewel to Luminous
(5) Solution without service impact: alternatives comparison (pros and cons)
(6) The privilege of micro-server architecture while upgrading the Ceph cluster under production.
(7) Q&A

About Aaron Joue
CTO, Ambedded Technology
Founder & Chief Architect SDS Solution at Ambedded Technology Co., Ltd. | Aaron is passionate about promoting Arm microserver architecture and Ceph appliance because of the benefits of the small failure domain, efficiency, density, and power saving.

A

Okay good afternoon, my name is Aaron I'm coming from embedded technology today, I'm sharing our experience of telecom usage unsafe and the experience of upgrading the yosef and scale yourself in a mixed mode of arm server and operating systems. Okay,.

A

Okay, the the contents I will show you is about an initial safe cluster are building up the first self cluster in building a telecom company, which is the one of the biggest telecom company in Taiwan and how they meets their problem when they grow up to users. And how could we analysis the problem and finally decide how to solve the problem? Okay, so the detail I will show you later.

A

So. First of all, I would like to introduce our company that use two pages, introduce the company and the products. So you can understand what I'm talking about later. Okay embedded technology is a company that provides the safe appliance which integrate speak purpose designed for safe on server, and we also use not only the community safe but also the souza enterprise storage, above that we also have a measurement GUI, which called UVs manager. Let us to give the customer support towards and make the customer use the safe quite easily yeah and the company was.

A

We were upstart company start to promote this product studying from 2015, and we are. The product is a war winner that provided by the 2016 best of Interop, are in United States and best ROI golden award in Taiwan Computex.

A

And second page is about products the product, because it is quite special that to make it easy understand what I'm saying next so better to have an idea about what it is. All appliance is based on micro server architecture. Instead of using a big host of server host to host many OSDs in one server, we design to use one micro server based on our 64 bits and host only one hard drive and one SSD for the journal and any node can be configured as an ending anyhow.

A

It will stick and be configured as an OST monitor or MDS right. So so it is quite easy to deploy a safe trust. Even you have a one single node to be a high of ability, safe, cluster, okay. So the customer background is one of the biggest telecom company in Taiwan because of the competition pressure from from the market, so the telecom they have to change their mind that by migrating nail legacy cloud which use VMware or there are many very expensive storage to be open source. So that's a big change in our telecom company.

A

You know they are quite conservative because they are mission critical service, so this they decide to start with.

A

The internal is private, stuff private cloud because they have two kinds of cloud: one is for public, that's a public cloud for most of the telecom news customers, but instead of instead of mass production for the critical public crowd, they decide to have a small scale and grow up to a big one inside their organization. So they decide to develop their own, build OpenStack cloud based on the based on OpenStack and their own developed cloud management portal, that's their main focus in the beginning.

A

They also decide to build a safe storage but itself, but that is not easy job. They don't. They don't have also enough resource to make it work. Well that because of the storage, cannot be smoothly be used, so the project was delayed for almost three to four months s, and here they found us to help them to quickly resolve the problem. Okay, so this crowd start with a small cluster, but they want to make it grow and push the the internal users move to your service internal service.

A

To this opens next car yeah, so that's the company policy to set their cost for the future. So it's very important project in the inner telecom company and after six months of production, they pushed their internal department users to put their services. Internal services migrated their VM to this OpenStack crowd and ensured our six months.

A

They move around 140 VMs, but that's it's gradually move not in not move or 140 at a time, but when they starting more the more great grow to 140 VMs on this cluster cuz, the users starting to complain, the service become slow because they are, they have a lot of service internally. So even that is a primary crowd. They are have the users give them a lot of pressure, so we have to find them. What's the problem?

A

We we don't know they put so many VM on this OpenStack crowd with the safe, because the safe we built is only 21 OSD s. That is quite a small cluster yeah. So when they find us to find out the problem, we certainly know the they are stretching. They are stressing the the safe cluster very hard yeah 21 OSD work for 140 vm.

A

That cannot be imagined, yes, but we cannot just tell them that you are pushing the safe class too hard, but we can use the lock using numerous that data to show them how the safe cluster is working. So, first of all we checked the log of the safe throughput and the I apps for certain days like a week, we check the we check the log, so we find that the read and write throughput is quite low.

A

For example, you can see the rate is about is about less than 10 make up ten megabytes per seconds, and the right is maximum 120 megabytes per second. So throughput is another issue. Then we checked the IRS okay so, but we find that I've asked is quite high for 21 OSD. We can see the average is about 3000.

A

This is like three days: log and maximum can reach to like seven seven thousand hours. So this is this is we can see? This is why the cluster is quite busy and slow cutters lot of slow request. A report from the safe cluster.

A

So the major problem is under IRS is quite clear. We also checked two of the 21 OSD, because the the safe, the previous number is get in between there's cash in between. So we don't know, we can now say by this number to show that hard drive is overloaded, so we have to check other numbers. What we check is the the disk I ask. We get log from the every disk from this graph.

A

We can see that two of the which just showed here two of the 21 OSD disk in five days average every OSD keep always above 100 apps, all the time yeah and the maximum IRS on each disk is 180. So that's already very clear that the OSD are all overloaded.

A

And this is the the log for all the oil stay in one day and tells the similar information that the OSD disk this is without the cache data is over.

A

100 I asked every disk every all the time, and then certainly we have to check. Also. Is that the problem from the disk or CPU how CPU is utilized? It shows that the CPU is quite easy.

A

There's the utility of the usage of the CPU is less than of 50 is about 40 on every or every CPU, yes, and but the CPU load we can find always has several tasks. The severe load always have more than one task is huge waiting for the CPU to severe is waiting the I/o, so that's very clear that the bottleneck now is under on the I/o. So every even our note processor is a 32-bit arm. The CPU utility is still very easy, but the problem is under the hard drive.

A

So we can know that from the the log that at the level of the layer of the safe cluster, the I ops is about 200 apps per OSD. This is quite high and the total peak I ops of the cluster is 7000. So from this layer we can tell, and also from the hard drive the the IRS is very high. Yes and average the I have shared to single the virtual machine. You can see, that's only 20 a 250, that's that's! A PP are variously make the V n operators very slow.

A

Yes and beside that, we also collect what is the ratio of read and write and what is the size average e size of the block? So we can see that 60% of the operation are right in there in their use case, and the average praxis is about 20 to 40 K kilobytes. So that's also make the hard drive way of overloaded.

A

Okay, so I think I missed one page. Let me check.

A

Yes, okay: this is the initial cluster that we built for for this telecom company that we use three of our product- commas 200. That's our priest, previous generation on server, which has eight note in each chassis, and one of the note in each chassis are configured as the monitor and seven of the other, the other nodes we configure them as the OSDs and every every chases there are 4 by 10 Gigabit Ethernet connect to the top directs, which is a redundant high of ability configuration. So you can see each each chassis.

A

There are 4 by 10 gigabits, and so this is the original deployment and they are the. The set version is on June 10 point-to-point 10 and the the operating system is Debian.

A

They use the replica tree as the pool for their cinder gains and valid okay. So, let's back to the solution.

A

So the short-range solution is that they have spell spell servers, so so they quickly, we quickly deploy another v 15 OSD in into this cluster, to release the load of the AV OSD. So after we ate the the 15 new OS T's, they are also the the doer of version and it becomes 66 OSDs.

A

So after we add the OST, we can also monitor and check how the oils the load after we add more OST into the cluster. So now, after that, we can see this is before in spend. The OSD average is about 100 and after we add another 15, you can see the break. One is the is the average so always keep around under 50 for those V ends, because this is one of the solution in to expand. They also remove some testing VN about 25 out of the cluster, so these are after. This is still about.

A

115 V ends in a cluster, but the cluster loading is a lot of released, so we can see after we ate the OSD okay, but the for the long term for a long term solution. They would like to also add another more always ease more servers, because, because the there VN girls are quite fast that, because that is the company policy to push the internal applications or move to the OpenStack. So so the plan is that to add another three units of our answer verb.

A

But that is the new generation of the 64 bits on server and.

A

Then we have mixed generation of hardware, but the other problem is that our new new, safer Primus 400 is running the the newer version of say, which is luminous with blue stone. So there are some options to to choose. One is keep keep using our older generation server, but you still using the the jewel without just upgrading the software, but this is that the good choice, because finally, they were metal problem that an new safe version will come out.

A

They cannot just stop at the jewel, so we decided to two to eight the new hardware, which is 64 bits to a cluster, but there are some. There are some issue that we will meet by aiding mixing these.

A

The advantage of this is that this allowed the customer in the future can can or follow the pace of safe version. Migration and also the old version of server can also use the beta performance boost all and also our user interface can also migrate to the newer version which support the luminous, because the luminous and duo has little difference. So the software is separated.

A

Ok, so after this we review the why they are they made. This kind of problem is that they have no experience and plane in the in the beginning that plane how many? What is the size of the cluster, because the safe cluster is scalable not only on the capacity but also the performance, so when they start to plan their first safe cluster, they didn't think about the I ops they required for the VN, but only they think about the capacity. So that makes the problem, so we tell them how to plan their.

A

I ups, okay, the number that paste is an our hardware, that we have roll I, apps / OSD. This is what we know. If it is 100 percent read every OSD we can get seven seven hundred and every OST can get 200 to almost 300 I ABS on hundred percent right and there because they are load, is more on the more on the right and the right also have overhead and replication. So, based on this, we can also get the total I ops up to at least kind of workload.

A

Then we can get read, is to send every OSD can contribute this number of IRS, so the first one that is the problem, one that they only get 49 less than 550 ir, / vm and the phase two. When we it's paint another 15, they get double the IRS for the VN. That's certainly release the work eloped and for the fishery we pre plan to add another 21. So, but by this we can know that if every VM planned to have 100 IRS, they can finally reserve the capacity for VN up to 180 RV ends.

A

But for the challenge of the upgrade to expand the new hardware and the new up new safe version, there are several challenges we have to meet. First of all that we have to upgrade the everything without interrupt the service. They cannot stop the service for the upgrade so and then second is even the the upgrade.

A

The upgrade will not heavily impact their service, but we still want to shorten the migration period because there are 35 OSD in dual feistel will upgrade to were upgrade to the luminous blue stove that take time, and we have to set strategy to shorten this period and because the our new 64-bits arm server is the center's and still as a as a as its operating system.

A

The previous one is stadion, but the new generation is centers to unify the management user interface, so we have to not only migrate the the the safe, but also we have to change the old server operating system from Debian to to CentOS. That's another change challenge.

A

Okay, so this is the target of the migration. Originally we have, we totally have us of the OST arms arms 32 and Debian, and of them are jewel with Debian operating system. We want to migrate them or to the central operating system and luminous with the booster and mixed, also expend some of the OST with our 64 bits arm.

A

Certainly they are the Centers and luminous.

A

The standard migration of the safe, even from geo to luminous is quite simple. The center is that upgrade the monitor first and the restart the monitor and upgrade. Then you upgrade the OST note and then migrate firestore to boost all the first. Two is quite simple: the third one is just tech time yeah, but this use case cannot follow the standard procedure to to upgrade.

A

First, we have to upgrade webui to our newer version and there are two options to migrate: the cluster of the monitor when, when migrate, the cluster, we have to upgrade the monitor first, okay, so there are two options to migrate. The monitor first is that we migrate directly load three old monitor from at the same time, the luminous Tula luminous version and also replaced the oiss, but the lease is more risky.

A

We think so we decide to use another strategy, that is to add another two new notes, which is unsound 64, but it is Jo version before upgrade to the to the luminous. So after we add these two new monitor on door, we can kill those old notes and migrate that to new version yeah. So this give us more less give us less risk.

A

The migrate steps is quite many steps, but the very important is that, because we are killing the old monitor after we ate another two, so we have to change the crime site, safe calculation, fire to add new new, monitor, notes, host and their IP address. So that's when we kill the old monitors, the crime site will still find the class the monitor to make this easier to understand. I use this to to to explain so.

A

In the beginning, we have three monitor indoor and out all the web pays our UI and also the the important is that the NTP is running one of the monitor so to migrate. This we have to first to add one new monitor on CentOS, but it is dual version. We add another one. So now totally we have 500m in a cluster. After that, then, we can start to kill, delete those out, monitor nodes.

A

But before that we have to change the NTP server from the old monitor node to the new, only monitor node, and then we kill one by one. After we kill that, then we can. We still have to monitor in the cluster and we can starting to use our new version of web UI. But the next step is to migrate the this to monitor to the luminous version.

A

So we migrate one by one to monitor to luminous and after that.

A

A

Upgrade these two monitor is kind of easy that we install the upgrade rpm and upgrade the set version. Then we restart restart the monitor. Then they become the luminous version and at least moment the cluster become the winners version. Even the OSD are still in doer.

A

After we ate back all of the we used, we migrate the old hardware and totally change the image and let them to join the new new new monitor with CentOS and the luminous.

A

So after that the set become a luminous version, even they are still of the OCR, still keep in the jewel version, and now we have a five or monitor in the cluster SOCAN. After that we can check when we delete the monitor and finally, finally, the version of the monitor become luminous.

A

And we also can check the version of the cluster it's become after we upgrade those monitors if they become the luminous version as well, but the OSD are still keeping in in a jeweler version. Ok, so, finally, we have to migrate those old OSDs, but before we migrate loads old OSD, we have to add new Asti to the cluster that make the cluster have more OST to help the cluster to be revealed when we remove the OSD.

A

So after we 821 new OST totally, there are 57 OSD mixed with gia and the luminous and the role usage about 60 terabytes in total, and there are two options to migrate: the OSD from the safe documents, one is to mark out all the OSD and replay recreate it with the new version. It is more simple and less risky, but the data have to move twice in a network. When you remove it, you have to make wait and we wait it reveal.

A

Then you can use the after the cluster become healthy. You can edge that always be back with two winners. This is the first option and the second version apps is to use an extra host but swapped out the data from the old OSD to the new OSD. This is like this copy one by one yeah and we decide to choose the option. One is because we think this will be faster because they are when we delete one OSD.

A

We still have 56 or other OSD will keep rehear the PG and when you add back the us also many OSD can help to backfilling the data. So we think this is more faster than the second option, and this is the flow chart to show the loop. We use the ends Bob playbook, to help us to do the total migration. So the flow is that first we remove one OST from the cluster. Then we wait. The PG backfilling become recovery until the it the total caster become healthy.

A

So these take about an hour for that about one terabytes reciting, every OSD disk. After the the first OSD is removed, then we can replace the OS and the safe version and make that OST rejoin the cluster. So after this reboot, then we have migrated one OST after the new OSD with join to the cluster, you still have to wait. The backfilling data migrated to the new joined OSD, so this this one also takes about an hour. So this is one cycle to mark out.

A

One OSD and migrate is operating system and safe and make you join the cluster again. So totally we have make 30 36 loops of these. So after is the. If there is any arrow, we can just stop to do the next step.

A

So the the instable book playbook will check everything every step before it go to the next step.

A

So totally we use about one hundred and eight hours to migrate those statistics of OS DS and every time we only delete one OSD, so that will reduce the impact to the service to the minimum and because the the design of the AV OSD has its own processor and server that this also accelerate the migration of the OSD to be shortened so that we can migrate all the OSD Lessing them five days.

A

But we, if this is not in a micro server design, it may be different, for example, when migrate, every host, it will affect not only one OSD, it Moyle affect multiple OSDs. That makes this more impact on the service, but I have no idea how about a big note if they recover 6:36 already will take how long time to to recover.

A

Maybe this, if you have the experience you can share with wizards. So this is the whole migration process that we we made in this use case to share with you. Yes, so any question.

A

Okay, so thank you time. Thank you.