Cloud Native Computing Foundation KubeCon + CloudNativeCon China 2018 (Shanghai), 21 Apr 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: When Distributed Database Meets Cloud - Lessons Learned - Yanqing Weng, Pivotal

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

When Distributed Database Meets Cloud - Lessons Learned - Yanqing Weng, Pivotal

Database as a Service is one of the most interesting and challenging domains in the cloud industry. We’re migrating distributed databases like Greenplum (traditional MPP database) and Apache HAWQ (Hadoop native SQL engine) to make it a reality. This talk will be about the lessons we’ve learned - the best practices and pitfalls we’ve discovered during our journey which includes: - How to containerized distributed database and what help could Kubernetes provide? - How to redesign legacy architecture of distributed database to be cloud native? - How to make smart scheduler for optimal resource utilization? - How to distribute stateful workloads and improve performance by data locality? - How to leverage Operator pattern to automate distributed database operation? - How to integrate CNCF projects with distributed database to make it easy to use?

To learn more click here: https://sched.co/FuJW

A

Today, I'm going to talk about wind.

B

Distributed databases.

A

B

Have large work in these filters.

A

B

A

Do lot of things on.

B

The traditional database.

A

B

A

C

A

In the cloud we can.

B

Do a lot of teams.

C

A

Today, I'm going to share with you, how can we use contain.

C

A

Technologies do juries form.

B

The traditional database into.

C

A

A

Use the clout to database and also some lessons learned during the process.

A

Introduction, my name is.

C

A

Now I'm also appache our.

C

B

A

Remember that appache they trades up the coop, um see.

A

We talk about something.

B

About the Apache it a double acute project.

A

B

Also, what are the new requirements.

B

A

Lessons we've learned and a lot Secession, not.

C

So much where we keep the Q&A.

A

B

A

Can use a lot of key.

B

Words to describe.

A

B

Native advanced and movie, you must take quality, so so we use native.

A

B

We follow the best practices and.

C

A

Practices that we use.

C

In one system and also advanced amigas use.

A

C

A

C

Learn some advanced to functions and also.

A

B

A

Way that you do the quietly.

C

A

C

Do that efficient.

B

C

Didn't like the traditional database.

B

B

C

Have their own.

B

A

B

C

A

The users, so they can.

C

We have the quarry, so there will.

B

Be half a dispatcher.

C

A

A cutie there will be responsible.

C

B

A

Resources to the second.

B

Segments based on the resources allocation.

A

These QE or query.

C

Executors afterwards, they will send.

B

C

Live feedback orders to.

B

The master and assembler users or user data.

A

B

They're not stored in.

B

About stateless.

C

A

C

Actually doing.

A

The process, once we send out the information.

C

B

A

Those information.

B

B

C

A

Those query information web.

B

Simultaneously,.

A

B

B

C

B

C

Lease they are quite efficient.

C

A

Tricky part is that how to manage.

B

These high speed or efficient data.

A

C

Have to ride a QE procedure and.

A

Connect to other cutie.

B

C

A

B

Pcc Bronco.

A

C

B

The scalability issue we're.

A

C

A

Have these verification and mechanisms.

B

Of the system and.

A

Achieve a maximum scalability.

C

B

To carry so we can see so we have these.

C

A

The parser and optimizer, so you can see the point.

C

B

So just review in your database.

B

Maybe it can be viewed as like a collaboration system. This is cowboy isn't of a curry that can.

A

Be processed dynamically.

C

A

C

A

Restricted by the.

C

Physical features of the database, but it's.

A

Quite a dynamic query, processing.

C

So it will be then sent.

B

To their manager and it will look into.

A

B

C

System resource manager, if the parallelism.

A

Is N and it will recite that n east appropriate amount and then it were standard, huge segments and to.

B

A

B

A

Segment, you will have these virtual database. This.

C

Is virtual segment.

A

C

These segments.

A

And also we have a physical, so.

C

A

Can actually understand it this way.

C

A

Numbers like a.

C

Virtual segment so.

A

We can look at these.

C

Parallel query: after allocation.

A

Work on the scan knows.

B

A

And you need the motion of.

B

The data and then has these.

A

B

Procedure so the motion node.

A

C

Process different QE procedure to support enjoy.

A

C

A

C

B

Different emotions.

C

B

Redistribution so.

C

The first one is redistribution. You.

A

Really relegate.

C

The tears you send to.

A

The upper labor.

C

B

C

A

C

C

A

C

Notes to the master, node and also.

A

If you include a motion procedure.

C

A

Cannot be done independently by a CUNY procedure, so we have to.

C

Slice, the.

A

Curry- and this can be done independently.

C

B

A

A

B

A

A slice, so if a.

B

Virtual sacrament needs to deal with several different slices.

A

You will have to run different QE procedures.

B

And each one is a slice so for each so, for instance,.

A

The one on this ring includes three slices.

A

Will take place in a motion not so it.

B

Will be turned into two slices? Also, the more motion you have, the more slices you.

A

Have and the more the more procedures.

A

You can see that the segment is six, so that's the number of the virtual segments.

A

Virtual segments and the relationship between the virtual segment and.

B

A

Data allocation, so we can actually divide.

B

A

Into six parts and each.

B

A

Of a one of them is 1/6, so.

B

C

Has a motion query and.

A

C

A

Thunder motion motion so each process, the data and.

B

Redistributed to dirty slice and receive emotion, and they receive their data from this lighter.

C

A

0 is where we have the angular motion receiver and we and we send the.

B

C

To the users, so a short summary, so it's actually a resource allocation unit and it's also kind of like similar.

B

To a container.

C

Segment is also execution.

B

Kind of things.

A

That you can based on the complexity, complexity,.

C

Of the database and it's.

A

Based on the physical segment.

A

Short summary of the Apache hawk so.

C

It's their performances.

B

And parallel system, it's more flexible and elastic. It has an Alaska.

A

Query execution engine with.

B

These stat lace segments so with those features, so we can say that use.

A

These hawk to transfer the traditional.

D

A

A cloud database.

D

The cloud database compared with VM traditional life employee as a new requirements. First of all, traditionally we have this cluster for one cluster, for.

C

One to base and.

D

C

Support for multiple clusters, but.

D

It should be isolated, we do not, it does not need to share the resources.

D

Has make all cloud database more agility.

C

D

We can add and share more in Sharbel resources and to make the misapplication, and if we can increase usability, we can.

C

Own the cost for customers, and this.

D

We need to think about more.

C

On managing of the system.

C

D

That we, our cloud database, is suitable for different types of operating environment, including our private cloud of the cloud.

C

Because we cannot offer each client.

D

A different and DBA.

C

D

We need to try to achieve DBA free.

C

D

For the last two requirements, the deployment and DevOps, we need to consider them together.

C

D

A solution is container containing a little bit.

C

D

Need to seek for differences because on the container environment, then we do not need to consider a lot of the yet voiced in the operating system. That includes our private cloud and public clouds. All the same issues that we need to incur, comparing with VMs pm's, is basically for facing machines and containers are full. Apps containers is more friendly for the develop.

C

D

Container in need to have a good platform and unto that Cuba natives, it's a good solution.

C

D

Not only offer a lot of solutions.

C

D

Also offers a connection points.

C

D

Service discovery load, balancing in horizontal and vertical auto scaling rolling upgrade at.

C

D

Kubernetes is very good and monitored and metric collections. It offers a lot of solutions in it.

D

If you see this chart up here, this is our version based on Apache Hawk, so.

C

D

Is for cloud database there Mach 2 basic characteristic: one is the service structure, one storage and one is computing, and also this is a deep integration of our database and Easy's.

C

C

D

C

D

C

D

Of a more like s3 storage,.

C

D

The same time, we don't need to.

C

Do devops ourselves if.

D

This is remotes. We need to only need to get a new pouch as.

C

D

C

D

Is a resource management service because it's for facing your water all cluster? We need to have the optimization of our resource managing, so we can get all the segment resource and to offer them.

C

D

We have a new management.

C

Layer or storage it.

D

C

D

All the segment resourcing here when master is Kevin queuing operation, they can distribute it the right amount of segment resourceful master after master finish case hiring the resource will go back to.

C

C

D

What is the segment's here.

C

D

In clock, there's no container container they're limited the source of the resource.

C

D

Size of the resource is alignment with the source.

C

C

D

Way requires different resource, therefore different segment, that's different automatics, so we all form segment, we'll pull all the segment poles together and to form a segment cool.

C

D

From prove to pool.

C

D

According to the character acquiring, they will find the most relative resource that can be allocated.

C

D

Next, maybe we'll talk later about the.

C

D

After we have the computing service, then the storage service will become easier.

C

D

Now our customers only need to set up and master notes here, and you can solve all the problems.

D

Nowadays, we don't have clusters, but we can still achieve the effect of building.

C

On the cluster, and because.

D

Of these queries, we can allocate the right resource to.

C

The right statement, grille.

C

D

Finally, use all control panel after we container lies our data. We need a lot of controller and parameters for demos.

C

D

C

D

Like the integration and automation, DBA of all controls.

C

D

This structure.

C

D

Is basically based on cloud control? Basically, everything is full Cooper native to do the DevOps.

C

D

This we have talked about it.

C

D

Graphic here: how do we look into deficit by natives? Look, the blue one is hogs component.

C

D

Use all the second of the quilt to the customers resource within that pool. There are different segments and segments are in the pots, and we used episodes full demos.

C

D

Reason why we use demo site.

C

D

That it's easier for analysis later on and.

C

D

They offer segment parts more stabilized environment, so these are really helpful for our DevOps because fools coz and we have the a located the right controller to focus on the different circumstances.

C

D

The hot must and standby most because these two components they are locally stay, so they do have the move, so they do have new status and we've had on the PVS. For these notes.

C

D

Is mister and one is my master.

C

D

I over to that there's a service to offer our customers they unchangeable address.

C

D

Have makin a hot cluster C Rd for.

C

D

Will do the dev up so only thing they do is to maintain the clock muster to.

C

D

C

Hoc operator.

D

Need to monitor the cluster if.

C

D

Master is arrow and it cannot be solved by restart and then hoc master. We automatic fires, the standby.

C

D

This operation is suitable by.

C

D

And our users can visit this by sue CMD. The last part of hawk is the resource manager.

C

D

C

D

Can have a request for resource management that is called the query CRD. So the pool and.

C

D

Resources and the managing of the resource is for carry controller, is in charge of that control, o a fire, a new control cid to search for the queries resource that they need. If there is no right resource, then it can wait or trigger the hot pools scaling up mechanism after finding the right resource. The address will be right under the query: CR DMV fired.

C

D

The resource is allocated, it will update its status information.

C

D

Under the process, the query controller will also monitor its current.

C

C

D

We'll be out of memory or out of the situation, it will be monitored and that event will be recorded into queries, CRD and it will be recorded as historic, full and elastic and optimization.

C

D

All these, you can see that yellow part are the basic service provided like Cuba native, including loved Prometheus, for monitoring, and the result of my train can be seen by graph, not dashboard. So all these basic service can be really helpful for DevOps.

C

D

Are metrics their logs and their events? All these can be collected and recorded.

C

D

Puff out of scaler will, according to these information, to decide when to scale up or when to scale down the.

C

D

C

D

Rely on the event to choose.

C

D

Right status for the pot, the size of the pot, so.

C

D

Will learn from experience if there is a query out of memory or.

C

Failure next time it.

D

Will recommend to fix the paramedics of a pods or to use a segment group to satisfy the needs, and this is for query and recommender. This we've spent a lot of time to do all the optimizations, including how do we decide whether the query size and the analysis of metrics and algorithms? How do we use our current system to use a close circle of it?.

C

D

I'm not going to that.

C

D

Have integrate many commands to CRT over here, so there are most.

C

Important sorry, yes,.

D

You can see.

C

D

Hot clusters: CRD: it's a customer, oriented a cluster pot and.

C

D

User talks, including the user matching situation and the master resident and stem bar resource limit, is for master nodes and standby Moss nodes combination.

D

Aquarii c rd is ass, is a requirement for the resource, including the basic information of queries, including, where does this query belongs, to which user units and a request info is the description of what this query? What kind of segment does it need after the relocation of the segment resource.

C

D

Will add up the address information of the segment notes and the query will use these information and to check out whether the query is successful and.

C

D

The very right hand side it's the food of CRD, the first one is group. The group information.

C

D

Have different type of resource allocation, including CPU memory.

C

D

Some other resources they in it.

C

D

Under that you can see the resource competitiveness. That is the main resource that we have the total resource we have, it will be updated. Real-Time.

D

And these controllers are used under the develops, hawks and optimization. We have talked about this priori.

C

D

Graph, you can see is the cloud-based hawk, comparing with the traditional basically.

C

Is all the same? It.

D

Is how do we get the situation? That's a different point. Under the traditional method, we need to acquire image from the ER.

C

D

The resources from our containers.

C

D

What I've talked about is, how do we do the integration and to manage the resource? Relocation hell? Do we get talked into a cloud database and under that we have a lot of issues when we stored it. We think that if we put this into image into containers like service like pavan's likes tableside, well, that should be enough, but as we go deeper, we find out that this is not sufficient.

D

The database requires more than these. It needs a lot of change to become a cloud database.

C

D

Our my summing ups first of all, is about the architecture. Traditionally, we have this monolithic structure.

C

D

Have these app for all the resources because performance, the.

C

D

This is not agile enough for service providing structure or resource managing.

C

C

D

Same resources, a compiler service is more easier.

B

For the sharing of the resources.

D

And to increase the usability of.

C

C

D

Nowadays, we can just go through and through api's at that point,.

C

D

It's connected for service operation and our DB is these resources, so different components can have easier to be compatible by.

B

Creatives, for instance,.

C

D

Of step aside, it can become.

C

A customer's resource so.

D

We needed to find some controller to control these resource. If we isolate them, we can have a lot of advantage. Db only needs to consider the resources enough resource. It does not need to consider how to manage this resource. So it's killing up scaling down just hand them to kuba. Natives is enough.

C

D

Is about containerization if talk about DP containerization.

C

Not just being docker image, we.

D

Know that we need to define our cpu resources for containers, so how much is the right volume? This is a difficult question to answer, because if we do not put limited in our resource or just to use it as default, if we can easily use our container as a VM.

C

D

The optimization of our resources flow and it will be hard for us to be centralized according to understanding. We think that container is actually needs a right, reach, right amount of resource and that right amount needs to be defined. For instance, five Sora can be easily defined. What is the right amount? Other application, other applications.

C

Might be difficult because.

D

Hawk is defined person.

C

D

On the physical machines.

C

D

Logic, segments, it's like a container, it has to be you manager, and the differences is that it is a logical concept.

C

D

The resourcing needs the NA surpass the limitation of the machine. Then it can be run saying it can be relocated to write real or resources that it need.

C

D

Allocation of the resources defined based on container.

C

D

The container resource cannot be it's not shareable, so we need to define the right amount.

C

D

This we can prevent failures.

C

D

C

D

We have a high efficiency on quiring.

C

D

Can make our running of flowering execution failure.

C

After cogeneration.

D

We can see the resource management as we said that traditional database is based on fixed resources.

C

D

Based on the fig resource, how do we allocate the resource on different queries.

C

D

Based on pedicure resource group cloud database, we can have this dynamic.

C

C

D

Do not need to worry about the limitation of the resource. We only need to think about. How do we maximize the resource sharing, and this is a lot of difference between the clown and traditional database.

C

D

Are just monitoring tuning.

C

D

Have a lot of we spend a lot of time on talking about the resource resource, managing.

C

D

Research under container environment, how do we optimize our resource.

C

D

The characteristic of dB, it is very hard to calculate how much resources.

C

That actually needs so.

D

Different query requires.

C

Different resource.

C

D

We have these different queries. It requires different amount of data, so request your friend amount of resource. So we offer.

C

This model, and also we.

D

Can do analysis to see the similarity of these queries to manage the best resource usage.

C

D

We collect this, including pot runtime metrics, including application logs into native.

C

D

See the resource, allocating situation and to monitor and to optimize them. If you do time limits, I mean what.

C

You talked about find 2 points, so.

D

I like to say that we need to make the.

C

B

C

Of the cuba native ecosystems, these are my experience.

C

D

We want to utilize the CMD, then we need to have.

B

C

Utilize it and the.

D

Older structure.

C

D

Finally, high priority based on the process, because we have.

C

A lot of plots.

D

If we do not use our PI pot priority, the part can be fighting for their resources and can be a credible. Thank you, they'll be do.

A

You use the documents for storage or anything else, so.

A

Actually, if you use distributed database, all those segments can be Leo as some we use.

B

Door together, so we have.

A

C

A

Each one of them when they scan they were not covered, so there would not be interface, and then we need more about data to get a more complete view.

C

A

That depends on.

C

A

Your document is based in site to contain an oh, it's on the cloud. Somebody's.

B

A

Face inside the container, then, when it's lost it's gone forever.

C

A

Just a volume based on my own experience.

B

A