Cloud Native Computing Foundation gRPC Conf 2020, 5 Aug 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Talking to Go gRPC Services Via HTTP/1 - Malte Isberner, StackRox Inc. & StackRox GmbH

Description

Talking to Go gRPC Services Via HTTP/1 - Malte Isberner, StackRox Inc. & StackRox GmbH

gRPC works perfectly for service-to-service communication in private networks, where a client talks to a server via a direct TCP connection. Things are more complicated if services, users, or CLI tools talk to gRPC servers exposed on the internet, as many of the commonly used application-level load balancers/reverse proxies only support gRPC with additional configuration (e.g., nginx) or not at all (e.g., AWS ELB). In this session, an approach for communicating with gRPC services through reverse proxies that only support HTTP/1.1 is presented. While the approach comes with some restrictions, it requires next to no changes to both the client and server code, no additional components to be deployed, and ensures maximum compatibility between different client and server setups. Along with the approach, a Go library implementing it will be introduced, demonstrating via code examples and a live demo how this technique can be used as a drop-in replacement in existing Go-based gRPC code.

A

Hi, my name is malta, I'm an engineer at stack rocks and today I'm going to talk about talking to go drpc services via http1.

A

That's a problem that I recently had to solve at stackrocks, and I want to jump right in at stackrocks we're building an application which is deployed in the kubernetes cluster running um our customers and our application uh is based on a microservice architecture. So we have a fleet of microservice deployed in the kubernetes cluster and those microservers talk to each other and we're using grpc for that communication.

A

This intra cluster service to service communication using grpc works great, however, we're also using grpc to communicate from uh entities outside of the cluster to services running inside the cluster. For example, we have a command line utility called drugscuttle and for this command line, usability we're also using grpc to talk to our backend service and in such a setup, the traffic from outside of the cluster has to enter the cluster, and it typically does that by using a load, balancer service or an ingress controller.

A

um The problem with such a setup is that basically, it boils down to jrpc being based on http 2 and a lot of load balancers, don't like or don't support, http 2. more precisely. I'm talking about level 7 or application level load balancers. The most prominent example of such a low balancer is alb by aws, but there are also like some other load balancers which have problem with http 2 and grpc, such as nginx in older versions.

A

What can we do in such a scenario? Well, there are a couple of options. We could opt for tcp, only load balancing, for example. If we use a tls encrypted connection, we can configure it to use tls pass through, in which case the um the load balancer would not terminate the tls connection, but instead just uh forward the bytes one by one from the client to the server.

A

We could also use a load balancer that supports http 2 between the client and the server. The l7 load balancer available at google cloud platform, makes it possible, for example, but also sufficiently recent versions of nginx or h.a proxy, there's also a module for envoy the android proxy, which is called the grpc http 1.1 bridge. This is actually pretty similar to the technique I'm going to present in this talk, the main reason why we didn't up to use it was.

A

We felt it was to um too heavy way to kind of like spin up an envoy proxy sidecar for all of our services. um We don't assume that we run in an issue. Environment necessarily- and I also put an asterisk nest viable, because those are the technical, technically viable options that doesn't necessarily translate into practical viability.

A

Our customers might be more or less opinionated on what technologies they want to use for exposing services, and if we tell them, they can't use alb, which they use for everything else and that they need to use tls pass through and can't use managed certificates in their infrastructure provider.

A

They might be less enthusiastic about that, so um the functional requirement for the solution uh that we are that we are looking for is to replace the grpc, which is used in a rocks called cli with something like grpc that works via http http1, preferably it would be something that's just as nice to use and just as reliable as grpc.

A

There are also some compatibility constraints. Obviously, the new functionality of being able to work through a grpc incompatible load balancer only needs to be available when using new versions of both the cli and the server.

A

But we can't always expect our customers to upgrade the software in lockstep, so if they use a combination of a new cli with an old service or vice versa, this should at least preserve the existing functionality of working through a grpc compatible load. Balancer. One thing that that, for example, precludes us from just switching to a different rpc protocol altogether, because that would only be supported in the new software and not in the old software.

A

So the first idea we came up with was to use rest apis, and that sounds a little bit like contradiction to what I just lined out as a compatibility requirement. However, we have rest equivalents for pretty much all our grpc apis throughout the product, thanks to the grpc gateway which automatically takes care of the translation, what it would look like on a technical level.

A

We have our cli and that cli makes a request, a grpc request, which technically is an http, 2 post request to the server, and we specify the respective jpc content type and encoding scheme, usually protobuf, and instead of doing such a grpc request.

A

We instead do a plain http, 1.1 rest request, for example, to a v1 metadata, rest endpoint and using json as the encoding scheme for all our payloads um to use that, on the code level, our idea was to use protoc gen swagger to generate open api specifications from our jpc server subscriptions and then generate go bindings for these open apis back using swagger code gen, we weren't really thrilled with the result. It would have required lots of changes in our code.

A

It will also kind of has led to a duplication of our data structures instead of using protobufs everywhere, we'd also be using the open api generated data structure somewhere. We also found some bugs in the generated code. In all fairness that might well have been due to user error. I didn't dig too uh too deeply because it became apparent uh pretty early on that we should find a better option that the less invasive in the changes that require so for finding such a solution.

A

um An important first step would be to look at some background as to why jrpc uses http 2 and what http 2 actually entails the difference between http 2 and http 1.

A

Some of these differences are performance related, for example, http 2 is based on a binary format, and it also supports multiplexing of multiple streams onto a single tcp connection. Those are advantages for performance. On the functionality side, there is support for full duplex client streaming. That's important for some rpg calls and there's also support for trailers in http 2., um and there are also some other aspects in http 2 such as server push. It's really not relevant for this presentation or even for jrpc at all.

A

um So the first aspect we want to take a look at is client streaming. Client spring basically means we can interleave the request flow between server and client um http, one the client talks and then finishes talking sends all the data and then the server responds in http 2. The client starts talking by sending a request and request headers and the server may start responding at any time, regardless of whether the client still has more data to send or not. Grpc uses this for client streaming, rpc calls and bidirectional streaming. Rpc calls.

A

Client streaming is actually a pretty significant step in terms of functionality over http one. You cannot emulate client streaming in http one. You would need to resort to a technique such as websockets. In order to do something similar.

A

uh The good thing is rockstar cli only uses unary rpc calls, so no client streaming required. uh The second aspect is trailers. Trailer method is metadata sent by the server after the response body. You can think of traders as kind of like deferred headers.

A

The typical use case for trailers is to calculate checksum on the fly of a response body without having to to buffer the entire response body on the server side before sending it back jpc uses trailers in order to communicate status, information and error messages, for example, think of a server streaming rpc call. The server sends a message message after message: until eventually a deadline expires and then this deadline expires status should be transmitted to the client.

A

So, even though the server has already sent data, the the server can still indicate that the request failed, and thus the status needs to be transmitted cannot be transmitted in the headers but needs to come after the response body.

A

The advantage here is that traders are not significantly different from http anything http one offers compared to client streaming.

A

They are on the hcpe protocol level, but if we can assume that we can make modifications to the application level protocol, we can just embed trailer information as part of the response. Buddy, just conca like append it to the original response body, and that brings us to jrpc web jpc web- is an alternative transport specification for jpc, um specifically uh targeting web clients such as browsers in a grpc web.

A

You can, you have support for unary and server streaming rpcs, so client or bi-directional streaming are not supported due to lack of http 2 features and in grpc web trailers are encoded as part of the response body. So gpc response body is a sequence of messages and a jpc web. The trailers are contained in a specifically designated message, which is sent after the last message of the server response.

A

um That sounds like this is exactly what we're looking for and it would solve all our problems. uh Unfortunately, it's not quite that easy. um I wasn't able to find the golang based client library for grpc web, which we would have needed to use on the on the cli side, um their support for jpc web in the envoy proxy, um but we cannot expect the onboard proxy to be present in our customer setups.

A

um There are also standalone proxies for jrpc web and translating between grpc web and grpc, but deploying them in front of application breaks, compatibility requirements because again it will require our client, like all of our clients, to use grpc web requests.

A

So the solution we came up with is what we call automatic jpc web downgrading, and this is uh instead of just using jpc web for everything.

A

It's an adaptive technique um which works by the client, sending a regular jrpc request to the server, but also indicating that it is able to accept and process grpc web responses, and we use the standard accept header to indicate our readiness for that and the server looks at an incoming request and if certain conditions are met, it decides that the response will be sent uh as jrpc web and not just regular jrpc and those conditions are on one hand the ability of the client to accept grpc web responses. That's communicated via the accept header.

A

We also look at the http protocol. If the hp protocol, major version is one then probably there's an incompatible proxy in between and the more reliable indicator is the te header te stands for transfer encoding and if the of all the proxies are compatible, um the te header will have a value of trailers. um If there's an incompatible proxy in between those, this te header will be stripped so making a grpc request from our rockscaller cli to our backend servers.

A

We start the the regular jpc request as an http 2 post request with the te trailers header assuming a compatible proxy. This request gets forwarded, as is to the backend servers. We have http 2. We have te trailers. Everyone is happy, the server is happy and can respond with the normal http to 200. Okay, jpc response: what happens now if we have an incompatible proxy in between? So if we have a proxy that drops the traders header well in that case, um and also we assume it downgrades the connection to http one.

A

We don't have a traders header, we don't have a http 2 request. The server doesn't know it can't process this request in a normal grpc fashion. The server is unhappy and it just will send back an error response now, using our automatic downgrading. We modify the client to inject the accept header on the client side.

A

We say, accept application, jrpc and also application jpc web accept as a standard request, header and the product proxy will forward this header to the backend note that we still the backend, still sees an http1 request incoming without te trailers header. In that case, however, we see the client is able to accept a jfc web response.

A

Server is happy again and it can send back a jrpc web response. The client, having indicated so, can also process this grpc web response and the client is happy as well.

A

How did we go about implementing this functionality? On the client side, we spun up a local reverse proxy. That's actually very simple: to implement in go, there's a reverse proxy utility type.

A

The reverse proxy modifies the outgoing request by injecting the accept header and it modifies the incoming responses by transcoding, the grpc web payload to regular grpc over http 2 traffic, and it does so by by detecting jpc web payloads through the content type header sent by the server. This transcoding effectively means we look at the last message in the stream. We verify that there's a traders message.

A

We pass the traders from this message and we forward them to the client as actual native http 2 trailers, what's also relevant, to consider in the setup is that the the connection between the application and the reverse proxy uses a local in-memory pipe. So it's not hijackable, and uh does we don't tls encrypt it?

A

um The reverse proxy needs to uh needs to uh intercept the request anyway and then the reverse proxy makes an outgoing tcp connection with tls encryption and that even allows us to if we have a direct connection to the remote pr to use client certificate authentication, because tls really only starts at the reverse proxy and is not terminated or men. In the middle. In between, on the server side, we use a function, a feature from the grpc server implementation, which is that the jpc server supports or offers the serv http method.

A

That's a standardized, go http server handler it. It handles a request and it writes back a response and we wrap the server handler by um adding logic to to see if we need to do gfpc web downgrading um and if we can do jfc web downgrading. So we look for we test for the presence of the accept gpc web header and for the absence of the te trailers header.

A

We modify the request to fake an http, 2 protocol information and also to re-add the te trailers uh header, and that is to kind of like prevent the grpc server implementation from complaining about the malformed or not processable request.

A

We also intercept the responsewriter for this server handler such that it will transcode http2 trailers into trailers sent as a message as part of the response body and then, with this modified request, we delegate we make a delegate call to the original grpc server serve http implementation, so showing this in one picture, I have to apologize for the low resolution of the picture. I didn't have the raw materials anymore. um The client makes a http 2 post request to the reverse proxy and again this is slightly outdated.

A

We no longer use a localhost, tcp, listen socket, but just an in-memory, pipe and local reverse proxy injects the accept jrpc web header. It goes to the proxy the modified handler on the server side. uh Re uh basically replaces the protocol information http one to http two and strips the accept header. The server is happy about the request.

A

um There's also the te trailers header which is injected, and it sends back a normal grpc response, which then gets transcoded by the handler to a jpc web response and the jpc web response reaches the local client proxy and again gets transcoded to a grpc over http 2 payload, which the original vanilla, grpc client can consume.

A

So, if you think that's cool or think it could be potentially useful, we decided to publish this code as an open source library hosted on github it's written in go so you will only be able to use if you use go for both your client and your server. While it's not true, you can use it on either the client or the server, but you won't really get any benefit out of it if you're not using it on both sides, but at least you also won't break any any compatibility.

A

If you use it only on the client or only on the server, you can install it with a simple go, get command and the source code is hosted as on github and published under the apache 2 license.

A

Now I want to give you a very brief overview on how it used the library, so the first example is showing you how you would use grpc. Normally without our library, you have a tls configuration, you have some dial options for establishing connection.

A

You would call grpc dial context function, you give it an endpoint transport credentials, tls and connection options, and then, from this grpc client connections, you can instantiate service clients do rpc calls on the server and eventually also close the connection.

A

Now, when using our library, the code doesn't change all that match you, you import the client package from a library and then, instead of calling jrpc dial, you call the function, client connect via proxy.

A

You still pass it the connection context, an endpoint, the tls configuration becomes a first class configuration option. It's not just another dial option, but if you have other dial options, you can also pass them to this function, just as we pass them to grpc dial and the return value of this function is a normal, grpc client connection. You can use that in same way as you use the one obtained with grpc dial.

A

You can use it to instantiate, a service client make rpc calls, and you can also eventually close it in the same way and if you close this connection, the reverse proxy, which runs in the background, will also get automatically shut down. There's no need to do this explicitly on the server side. uh The example I have is pretty simple um and then very stripped down, and it actually won't work like that out of the box, but yeah just to give you an idea, um you create a new grpc server.

A

You register your service handlers to this, like the echo server in this case, and then you call grpc serve with a respective listener. In this case, we listen on port 843 with the tls settings. uh When using our library that injects your hpc web downgrading handler, um you would start with creating the gpc server.

A

In the same way, you register your service handlers, but instead of having the jpc server directly serve network connections, um you create a go http http server with respect with a handler which is populated by calling the create downgrading handler from our library, and this create downgrading handler function, returns an http handler which does the grpc web downgrading in an adapter fashion.

A

You pass this function, the grpc server object and possibly also another http handler, and it can be used to multiplex the same port for grpc and plain http traffic. You don't need to use that you can also just pass nail here and then you have that http server serve network connections, they can also be tls, enabled and that's really it and then you're basically ready to go. You won't be used, be able to use client streaming calls, but you can use unary and server streaming calls and no further changes required to the application code.

A

Before I conclude, I just want to talk about two lines of future work that we're planning. The existing code is not really performance optimized, because the usage is not performance, sensitive at least not at a kind of like individual rpg call overhead.

A

uh We definitely want to do some performance optimizations, and one idea that I have is to basically do an auto sensing request to determine if we actually need that proxy on the client side, uh or we can just bypass it, because we know that the connection is going either directly to the remote peer or only through compatible proxies in between the other line of work is to support client streaming and bi-directional streaming rpcs.

A

As I said earlier, that is generally client. Streaming is not possible over http1, but we could use websockets. Websockets are generally wider, supported among proxies and reverse proxies. For example, aws's alb does in fact support websockets and by using web sockets for the connections and implementing a grpc over websocket logic. We could actually also realize client streaming calls that's work, that's actually actively ongoing and it's being implemented. It's already implemented in our product offering, and we are currently working on upstreaming this to the open source repository to make it available for everyone.

A

This concludes my talk. I want to thank all of you for listening I've added here. My email address in case you have any questions or feedback or comments or maybe ideas, feature requests.

A

You're also welcome to use the github issue tracker for leaving this. The url is I've printed here at the bottom of the slide, just file a feature request or back or just drop a note to saying that you found it helpful and that's it. I want to thank everyone for listening again good night.