gRPC Community, 23 Nov 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: gRPC November Meetup/ gRPC integration and its applications in Hive Metastore by Zhou Fang

Description

gRPC is a modern open source high performance RPC framework providing advanced features such as authentication, service mesh support, and streaming. Dataproc Metastore (DPMS) on GCP is integrating Hive Metastore with gRPC as an access path in addition to Thrift. This talk introduces the design of gRPC integration, and how it enables new DPMS features such as Cloud Run endpoint, find-grained IAM, and metastore federation.

A

Hello, everyone thank you for joining us in the japanese netapp. My name is joe. I'm a software engineer in the data clock, metastore metastatic. So for us we are all users of grpc. So today we will talk about our experience.

A

How do they replace the old apache, strict rpc endpoint with grpc, and how did we build new application uh based on that at first? I guess uh because uh maybe not everyone I think familiar with a hive or headline stores. I would like to briefly introduce this product.

A

High metastore is a de facto meta data service in the open source data analytics ecosystem. So in this ecosystem we have example, hive presto, spark software and also hadoop like this.

A

So what's stored inside high metastore is the metadata.

A

For example, database tables schema partition, statistic information, so with the information of database and the tables, for example, when you, you have a lot of files in some directories and when you use spark to run a job to process the data you need to tell spark which database which table are you processing it and the information of location and also, what's the how your data looks like the schema is stored in this meta store, so spark need to use a rtc call to this meta store to get this information back and there are other metadata, for example, petitions statistics those are used to make your query faster and process less data process only in the necessary data.

A

So this api is currently apache script and this api is what is used by all the interface defined here. I presto spark the architecture of a high metastore looks like this. So, as I mentioned, uh it has a selected server. The interface is swift and there is a mypasto server, which is a stateless rpc server and all the metadata is stored in a relational database, which you can see on the right side. The rdbms and the database and the server together uh becomes the high metastore, and the hive spark presto c will read.

A

Metadata will write method from the script.

A

So uh we have a lot of pinpoints uh using swift, actually, okay, uh because this is a product uh like built decades ago. At that time, uh maybe swift is the most popular rpc and point and is a beginner.

A

So almost um all the apache ecosystem are not using jpc. But today, when uh we look at uh how swift is used here, they identify the like limitations, for example, if the swift does not support streaming apis like decades ago metadata here, as you can see, is just like informational database tables. They are not real. Data data is very big, but metadata is small, but today things has been changed. The metadata itself. It becomes very big because data is bigger and bigger, so even a rate to the store. We all written a lot of results.

A

For example, it will return thousands of petitions, uh it's a a huge payload. So if we have streaming api, uh it would be very helpful to improve the performance, but unfortunately, it's not available in strict and also uh swift. We cannot use it together with serverless platform uh in the previous diagram.

A

uh As they say, there is a stateless back in the server behind a relational database.

A

uh On that server be deployed on gke today, but in future we would like to improve our infrastructure to use serverless platforms, uh so we plan to use cloudplan, but unfortunately, cloudflare only supports a grpc and http, which is a blocker from a deploying econ cloud brand.

A

So this is one limitation of stress, and there is another limitation is from the identity and access aspect. So today, strict does not have an integration, the solution for authorization and authentication.

A

uh It asks users to use self-managed approaches, for example, kerbals and ldap. For example. Then customers need to set up those uh authorization by themselves. They need to freedom, have a kerbalist kerberos k, distribution center or ldap server, which is very complex and not secure.

A

What what is missing here is that the rpc tool does not have a fully managed uh authorization or authentication solution, and in future we also plan to have better uh identity and access, for example, based on which rpc method you call, we verify your identity according to different permissions. For example, if you access different database, we will check whether you have permissions on this database. So all those complex uh authentication approach, we don't have a good way to build it in through the day.

A

For those reasons, we decided to invest to migrate to grpc, and uh here are a few benefits to be identified when we do this. Migration first is high performance, it's used http 2 and the profile 3. it has built in authentication support, and it's called token-based uh authentication approach uh and also it has sl and t os pdls, and so uh many of those approaches are compatible with cloud native uh authentication frameworks. For example, on google cloud we have iam, which is an integrated, managed, uh authentication framework.

A

uh If we can let users use iem, they will be very secure and they don't need to deploy any authentication layer by themselves and also the identifier is very easy to extend the functionalities through the intercept api of grpc. I will show how it is used for later in our applications.

A

Also, japanese is better supported by service mesh, for example, istio, because, uh as a cloud service, we have metastore high meta stores, endpoint exposed to customers, but that endpoint is not directly exposed to customers. We have a layer behind uh ahead of it so which we use is you, but the strafe does not have uh uh uh the better support with structure. So jrpc is still the best here and also, as I mentioned, jrpc is better integrated with a cloud run, which is a serverless platform to deploy servers.

A

Also, it does support streaming and request streaming requests and responses. So, with all of the spanning pages, we integrate higher methods with jpc and we build a few new applications.

A

For example, a cloud drive endpoint, so we expose a japanese endpoint, which is public on cloud run for our users, and we have implemented fine-grained, iem control and also metadata federation, which is a single endpoint that you can connect. The different meta store in the back.

A

So here is how we do the jpc integration, so the first. This is actually very hard for us because uh in straight high metastore, so the swift rpc interface. It has a over 100 methods.

A

If we want to replace it with jpc, we need to rewrite the intel server, which is a huge amount of work, and another issue is that the whole ecosystem is using. This service interface, if we change it to jpg our client, cannot use it because they are clients, they are swift clients uh in the swift client. They are integrated, impressed or in spark, and also they don't want to introduce a huge change to the upstream open source world uh to replace it with the entire grpc. So they want to do it step by step.

A

So, in the first step they didn't change any code in strict uh metastore. Instead, they decided to build a proxies. We deployed a proxy on the client side and we deployed another proxy on the server side.

A

For example, as you can see, in the first picture, the upper one. On the left hand, I have a jpg client. On the right hand, I have a swift server. So how can this stream the server talk with this jrpc client? So the solution is that we deploy a proxy sitting together with the server when the jrpc client sends a jpc request. This proxy translates the jpc client request to a strip.

A

The request, under the thrift request, is processed by the server the server return of server response, the proxy again translated back to jpc response and give it back to the client then choose this client, uh the proxy, together with the server the tool behaves like a single jrpc server. So this is the proxy on the server side.

A

So, on the client side the same actually, today we only have select client, but if we run a run, a proxy uh sitting together with the client and they do strip the jrpc translation, then a swift client can work together with a jpc server. So we need two proxies and then you can imagine the local communication between client, proxy and the local communication between server proxy the security rift, but the communication between the two proxies on the public internet now becomes jrpc.

A

uh So here is uh the idea of the proxy. So now you can see we don't need to change any code of the server right now we can make the communication jrpg here is how we implement the proxy. There are three steps so on the left side you can see. This is the three of the spec of the strap on the right side. This is an equivalent jrpc struct and the the first step is that we manually add an equivalent jpg interface.

A

We define every rpc method, so there are over hundreds of those methods and as well as the request and response in protobuf, they they need to do it one time manually, but high metastore has different versions, so they did it for each headband as well, and the second step is happening at runtime, so we use a library called abstract.

A

It is a mapper of java object, so we use it to build the core of the proxy. Every time we receive a swift java object, for example, a request. We use this map, strap to map it to a jpc request and send it out and in most of time it comes to the mapping automatically. But there are corner case, for example, in strict you can define a list of objects as a key, but it's not possible in jpc.

A

So, in this case, we will define a new jpc interface and we will manually rewrite the abstract to do the conversion.

A

The third step is that, uh because in the proxy and the switzerland jpg proxy, it's lessening on thrive's request for lessening on jrpc requests, so we need to strip the server or jrpc server and this server. We, we don't want to write this server manually because it has hundreds of methods, hundreds of handler functions, to write.

A

So we use a message: it's a templating code generation tool that we use a template to generate all the code once so, with all those three steps uh we compute the proxy.

A

And now we have the proxy uh how to deploy it. We have actually a few different phases. The fifth one is the simplest, we're just running those proxy outside heimlich store. This is what we do today. So in future, we plan to push it to high metastore open source. The solution is uh like this uh plot.

A

When the customer talked to hyman store today, it will forward all the traffic to a high-level store, thrifted server, and it will forward the traffic to the backend regulation law database, and so the next step we plan to add a configuration, and if the configuration says it will use jrpc, then it will launch a new high method.jpg server and the front-end server here will route off.

A

Actually, the proxy and the proxy will translate it to a strict and give the series request to choose original swift server. So this approach, we actually we pushed the proxy into high metastore, but it's still antroxy.

A

So the final step, we plan to throw away this swift server and use only this jpc server or make this jpc server the primary server and the user proxy when people need to use system server for backward compatibility- uh and we have a high proposal here so, which is uh the works we are currently working on- to make it open source.

A

Okay, so next I plan to have a demo about how this proxy works.

A

Let me open the demo file.

A

Okay, so I would uh like to introduce the tools that I use this demo a little bit. I use google cloud platform and I use a service called data clock, so data probe is a manager service, running spark hive, hadoop press or all those attached or open source uh data processing tools by google cloud.

A

Now I will play with them in this demo we will use jrpg proxy to connect a dataproof cluster to hivemaster. I already created the resources that we will use in the demo. First, we have a data pro cluster called jpc demo.

A

It has a master node and no work node, because we don't need the work node in for this demo and I already use ssh to log into this master, node and open the two windows here also, I have created a database metastore service which runs a high metastore.

A

It exposed a uri, a stripped endpoint here and I already connect the data flow cluster to this head metastore.

A

So if I take a look at the type configuration on this data broker cluster, we can see that the hive metastore uris, is the same as the ip here, which means that it will use this high metastore in order to verify it. Let's open a new head session.

A

At the same time, I opened the backend log of this head data store.

A

So first, let's take a look at the databases that we have. It only has the default one. So let's create a new database.

A

Called jpg demo one- and we can use this to quickly test that we we are using this high method store by checking whether this database will be seen in the log here. So let me refresh the log and you can see that the create database request has been printed here, which means that the connection is successful and we are using this uh high metastore and we can verify the name jrpc demo, one here.

A

So this is basically how it works for strict connection now, let's demo, how to use a jrpc proxy to do the same thing.

A

So let me close this session, so the first step is to create the server side jrpc proxy. I use cloudflare. To do this.

A

The cloud run is a serverless platform that I can run a docker image, and so this is my docker image, which is a proxy and after deploying it, it will be listening on a uri here for grpc request and once it receives any jpg request, it will convert them to thrift required and forwarded to a high metastore.

A

The head meta, swift, endpoint, is specified here you can see. This is the same. uh Stripped endpoint of the high method we just used so with this cloud run service and the high metastore service, uh we actually get a new grpc high method service because it will be lessening on grpc high methods request.

A

So this is half of the story, so the next step is to deploy, uh deploy a proxy on the client side. So the client is a data pro cluster here, this node. What we will do is that we will run a local proxy using this script. Let's take a look at this script.

A

This is uh because we will use docker to run this proxy, so this is docker image. So similarly, it's the image of the proxy binary.

A

This proxy will be listening on this thrift port once it receive any uh swift request. It will forward it to this jpc uri, which is the scene as we just deploy here.

A

Now, let's run this uh proxy.

A

So the proxy is running now now we have a new swift endpoint, which is this proxy, and the next step is change. Our height configuration to let height, use this local proxy local street proxy instead of the original remote swift endpoint.

A

So it talked to his talks to localhost. Now let me restart hike.

A

During the initialization of this hive session, it starts to talk to highmanastore. As you see, the proxy is getting uh requests.

A

Okay, so the initialization is down. Let's show the database that we have.

A

We say you have to say number one here, which means that it's the same high method store. We just used for the last step.

A

So let me do the same thing: create a new database, jrpc demo called javascriptdemo2, and you can see that the proxy received is this request and we can see on the log that is printed.

A

So this log is for the command that we just run is for create database jpg, demo 2, which means that when we create database, it sends out a script request under the proxy received it, and then it sends this request to the grpc request to this club run proxy and this proxy forwarded to the server which is. Finally, it is a high metastore and we can verify on the high metastore. Do we receive this request?

A

So you can see here the another database, jrpc demo2, create database request is received here. So this is how jpc proxy works. So, as you see with this local proxy now, this high hive start to use a grpc client to send out the grpc high metastore request and together with this proxy on the server side, this high metastore becomes a new hamburg store which exposes a grpc endpoint, which is this one. So the network connection in between between the client and the server is not jrpc. Instead of swift.

A

Okay, let me jump back to the original presentation, so in the previous demo I showed how the proxy idea I described, works in our product and now uh the next slide is uh now we have jrpgc. Now, how do we build new application uh when we have the superpower of grpc, so the important features that we build is I give an example here called the fine, green iem access control and the public customer endpoint, so the before we have jpc, we only have swift.

A

So what we do is that there's no enforced access control in high metastore. It means that if you have the the ip, if you access the certain point, uh assuming that you are not deploying, for example, kerberos or ldap server, so then everyone can access the enterpoint.

A

So in order to protect the endpoint from others, because you can imagine the hive methods to actually store all the metadata of our organization, so the data security is very important. What we do is that we hang this ip inside a private network which means that uh outside this vpc network, uh no one can access it. So this is how we protect it, but it's there a lot of complexities uh using this infrastructure.

A

The issue is that when using private ip, you need to do a ipre, because all of our resource is not created in the customer visible customer project, we created all the resources in a hidden uh tenant project. So in the customer's view they only say that I created the resource. It gave us a served endpoint, it don't need the customers, don't need to worry about all the resources.

A

So, but in this way we need a vpc theory and we also need to drag your education in the customs project and another solution without be disappearing is called a private service connect but easily. We need to do a lot of like a network work, but now, after having jpc our status now is that they expose the public endpoint. So this endpoint is public. Everyone can access it, there's no restriction of private ip, but the access is protected by a iem.

A

It means that when someone access some resource, we will check this permission. For example, it can apply to database or tables and if the commissions are like, create delete, update, get list, set, item policy and get a policy.

A

So this is the I find green iam access control and this permission can be granted to users to groups or to service accounts so how to be implemented. It is exactly the jpg proxy.

A

Now we deploy the jrpc swift proxy on cloud ram, so the cloud run can give the user a public endpoint and on the customer side, most of our customers. They are using databrick on google cloud. So on the virtual machine of dataproc, we will pre-install and run the switcher proxy, so customers don't need to do anything and for external customers who don't use data pro cluster. We are working on like releasing this to the public, so everyone can run the swift proxy by themselves.

A

For this I find green iem access control. We actually reuse the jrpc interceptor and it will. uh It will check the user's token and forward the token to the im service to check this permission and it will deny the service if the corner don't have the permission.

A

uh Okay, this is all I have today. So to summarize, uh in this talk, they uh like enhance high microsoft with japanese integration, as we enable new features on it, for example, bankrupt. I am because endpoint and the methods of federation- which I don't give a detail here and uh in future uh there are actually still a long way to go because seriously is still used in the metastar core.

A

They only build a proxy now, so the the communication between the proxy and the server is still unscalable for a huge request and the proxy itself introduced additional latency. So the long-term solution is still to completely replace a thrift budget pc.

A

That's all what I have.