Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2018 (Copenhagen), 14 Nov 2015

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Deploying SQL Stream Processing in Kubernetes with Ease - Andrew Stevenson & Antonios Chalkiopoulos

Description

Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).

Deploying SQL Stream Processing in Kubernetes with Ease - Andrew Stevenson & Antonios Chalkiopoulos, Landoop (Intermediate Skill Level)

Real-time processing allows you to act faster and SQL allows you to construct flows quicker and reuse existing skills. Apache Kafka is a key component but how do you peek into the data, the topologies and create repeatable deployments that can be promoted to production? Andrew and Antonios will show you how to easily create end to end data pipelines on Apache Kafka and deploy them to Kubernetes with no code required, only configuration.

"About Andrew
Andrew is CTO of Landoop, creators of Lenses for Apache Kafka, a unified Streaming platform for Apache Kafka. He created the largest collection of Kafka Connectors in the open source Stream Reactor project. He's spent many years in the trenches implementing big and fast data solutions and now focuses on simplifying development and deployment of streaming solutions into Kubernetes.

About Antonios
Antonios, CEO of Landoop, is a published author in Big Data and functional data pipelines, and an open source contributor. In his past he has delivered numerous successful Big Data projects in the Finance, Media, FinTech industries in London, and is currently pioneering with Lenses for Apache Kafka https://lenses.stream - and committed to providing the industry the tooling they deserve for building native Kafka / and Kubernetes data-streaming pipelines and operating them with confidence."
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

Join us for KubeCon + CloudNativeCon in San Diego November 18 - 21. Learn more at https://bit.ly/2XTN3ho. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Ok, everyone I think it's time, we'll we'll start now. Thank you for coming to our talk about sequel, processing, on kuba knees and doing it easily.

A

So my name's Andrew Stevenson I'm, the CTO of land OOP. My background is basically data fast 8, a big data doesn't care what it is, but I spent a lot of time in financial markets, high frequency trading. So it's all about speed and doing things. Real-Time, and we have Antonius.

B

Hello there, my name is Antonio's I'm, co-founder and CEO of land OOP, and let's jump straight into the topic so from a basic data containers like a JSON or even XML, to even more modern data containers like Apache Avro, where we get a binary format, we get a data contract between our micro services or our data pipelines. We get type type safety and pipeline safely and we can even have that evolution sequel makes sense, so it makes sense for three different reasons.

B

First of all, we can query data whether this is streaming, real-time, real-time data or historical data. We can also easily beat data integrations. Let's say: I want to be moving messages, storing them into elasticsearch cherry-picking, selecting fields, defining the primary keys. It just makes sense. It also works for a continuous streaming pipelines.

B

We can perform data aggregation, so we can be enhancing our data and we can be building continuous and bounded queries with expressed via sequel and when everything is stateless, it makes sense for us to have our entire streaming ETL in key, but netis have multiple pods for fault, tolerance and resilience and continuously operate data pipelines over kubernetes.

B

Again, when everything is a configuration, we can store it in github. We can manage it. We can promote from environments from development into UAT and thing two action in the modern enterprise world. We have multiple data sources and mullet read data sinks, and the idea of today's talk is how can we operate data pipelines via sequel and have a modern architecture? And what about my state?

B

The state needs to be managed, and for that we can use a distributed parallel file system like pata, Kafka, pulsar or not so a bit of an introduction of who we are. We started back in 2016 introducing sequel layers into Abaza Kafka in the open source with the stream reactor project that has almost 50 Kafka connectors and we moved on with building lenses. That is a industrial-grade set of tooling for building and operating data pipelines. I'm gonna pass on the microphone to Andrew for for the rest of presentation.

A

So I'm sonia stoked a lot about why sequel is great, but we still actually need something to do the sequel processing a lot of we're. Basically talking about data pipelines, we're talking about moving data around doing manipulation, do enrichment on data. A lot of people are running around and saying: ETL is dead. It's not we're just breaking it apart from a big big monolith into small and micro services. But what do we need for that?

A

We need certain aspects we need to get data in, we need to transform the data and we need to potentially get the data out to our long-term stores, so getting the data in. So this is the stream reactor project that we started. So this is using Kafka Connect, which is one of the API, is from Apache Kafka. They focus on getting data in and out. We have 30, plus connectors they're, all open source.

A

You know we do welcome contributions and we also have the Dockers for them as helm, charts ready to go, but maybe what differentiates our connectors from some of the others out there is. We have a sequel layer in it. This was first sequel layer in Apache Kafka back in 2016. It's not a true sequel engine, this but sequel like so what is it? It's a quick-connect query language. So this is what we use in our connectors. So imagine you've got lots of different sources that have different characteristics.

A

You may want to use the different features of those connectors. So in this case we have case we have hazel caste, which is an in-memory grid. Here's our caste, our different data structures. Maybe you want to write it into it into a ring. Maybe you want to write it into reliable topic, so we allow you to write a simple sequel like configuration to be able to target whatever you want to be able to cherry-pick fields out and write it into the correct datastore.

A

So each different sink will have different grammar attached to it because they all have different things. Hazel caste is different to influx, for example, what connectors do we have? Well yeah? We have quite a lot where they being used, so we have a Cassandra source. That's been used by Walmart they're, actually contributing a lot on that. The sink is certified by date, the stacks we have an elasticsearch sinks or British Gas in the UK connected homes. They do about 5 billion messages a day using our connector there.

A

We also have connectors for pulsar pourcel's, very interesting, maybe better than Kafka performance wise. It seems to be we'll have to see, but the connectors in Cuba need is what does it actually mean? What does it actually look like well, CAF could connect has two different modes: it has a standalone mode and a had as a distributed mode in the distributed mode, basically starting JVM workers, so that translates into a port. They all subscribe to.

A

A set of topics in CAF gave the backing topics they subscribe with the same consumer groups or Kafka sees them as a consumer group which is effectively your cluster. So in queue beneath this, we can simply deploy them as a deployment and have each replica as a port is the actual worker. Once the workers are up, they basically listen to Kafka for all the actual work that they need to do, and then they'll spin up more consumers, for example, to write data out into Cassandra, elasticsearch or whatever you want to do so.

A

What's good well, Cuban, eat, listen Shu! Is that our desired number of workers are there and we can scale up and we can scale down and the number of workers quite easily, and it was also very easy to deploy the good thing about Kafka connect. All the state is stored in Kafka was passing the book off to Kafka it taken care of the states of workers dying, come back, it's going to reconstruct all its configuration from Kafka.

A

What's not so good, Kafka rebalancing I, don't know if you're aware of how Kafka works, but you have consumer groups. If you join the group category balances the workload, but it means you stop processing. While that rebalance is happening. If you leave the group, the same thing applies now. If you're outside Cuba needs on VMs in your VM dies, your work dies. It's not going to come back that fast in Cuban. It is, it will so you'll get a rebalance, stop processing when your pod dies and then the pod gets restarted.

A

You get another rebalance, so you're getting a lot of rebalances, that's something you want to avoid. So how can you avoid that? Your liveliness probes, CAF could connect? Has a REST API to tell you what's happening with the state of your connectors inside there? So you can use the liveliness probe to say: do I want to make sure that I have my number of workers up and running or am I happy that something going to fail and Kafka is going to rebalance on to the remaining tasks that are happening inside there.

A

What you can also do in Cuban II is because we can actually spin up connectors quite easily if you can actually limit the number of connected instances in the cluster, so that limits the disruption on your calf could connect connectors running. Imagine you've got a cluster running and you've got ten instances right into elasticsearch writing to Apache CUDA. If you lose Airport you're going to get a rebalance and it's going to affect all those connectors.

A

So if you separate the mountain, have one connector per deployment, it's a lot, it's a lot easier and you limit the disruption.

A

So, let's connect this is the sexy bit. So we also have a sequel engine to do the transformation inside and on top of Kafka. So what you want to be able to do is you want to be actually be able to join topics together. You want to be able to aggregate. You want to be able to count. So how do we do that? We use sequel. We go through the Apache color site, optimizer that maps down on to Kafka streams.

A

Grafica streams is an API, it's part of the Apache Kafka distribution, so we actually take our sequel. Translate that into Scala code. It's an actual running, then Kafka streams, application that lenses or then monitor and deploy for you. We do a bit more than that because we'll also actually yet let you subscribe with sequel for either historical data or continuous queries on the casket topics, so you can do that by a Gore Lang. If you want to, we have a go line. Client. We have a Python client scholar.

A

We have a JDBC views, JDBC from end. Javascript react, read acts so you know a lot of trading systems might want to subscribe on FX currency rates. On top of the cafe, with sequel they can do that with lenses as well. Now the nice thing about sequel is it's configuration if I'm a date, scientist I might not know Scala I might not know. Java I might not be familiar with actually building. First, my artifact hosting the artifact creating an image creating a helm chart managing all that well see quartz.

A

Actually it's just configuration so imagine that you're, a dear scientist sitting in Jupiter you're working away. You can actually hit lenses via the Python client. We have tell lenses to deploy a sequel processor for you and then subscribe with a continuous query to get the sequel back out into your jebaited notebook. So you can deploy and manage everything if you want to I'm, not saying you're going to do it I'm not a date scientist from Python in a notebook.

A

Execution modes: how do we do it? We actually can run these sequel processes inside of lenses itself.

A

Development only really don't want that to be running in production or we can actually deploy and push it out into CAF can't connect. That is typically if you're not include bonita. So we have a lot of enterprise customers still, you know ever have their own data, centers, etc, but more than likely, because you're at cube con you're going to be deploying and pushing it out into air into cuba new.

A

So we use the fabricate api behind the scenes to deploy and monitor and watch what's happening and also then collect the metrics out of the Potters themselves. Again, what is what does it look like? What Kefka streams is just a library, it's just a jar application, so we can see it. The kubernetes give me three instances of the same application with the same sequel, which is just configuration we can pass in via environment variables and Kafka will do the balancing of the load for us now.

A

I have deployment and stateful setup there, because if you're joining and aggregating data you have state what Kefka streams does is it actually uses rocks DB internally rocks DB use, write data to disk and we also back that data into Kafka. So if you have a failure of your of one processing node of your Kafka stream application and it restarts on another machine, it can rebuild that state from Kafka. That's great, that's awesome! But what happens if your state is massive? What happens if you've got 40 48 gigabytes of state there? You could?

A

You could be building up a position report of a messages coming through Kafka and it could be a billion messages in there. It will recover, but it can take some time. It can bootstrap itself if it already sees that it has the rocks DB on the local disk. So if you use a stateful set, then you get this persistence following you around, so you get faster recovery times and faster boot. Ups again so I mean this is the important. But this is what it looks like. This is what we deploy.

A

This is a snippet from our helm. Chart. It's not there's nothing very fancy in here. You know we have an environment variable and we push in the sequel. We have the pre-built docker, which has added gmx exporter in it for Prometheus as well. That's all we need to do obviously there's a bit magic going on with the the kind of site optimizer in there, but basically, what is what it gives us? It means that we can use kubernetes to actually scale out and ensure our desired State. How many consumers do I want running?

A

Also, then we have metrics on there as well. We have consumer lag, we know what's going on with the processes, so we can then start building intelligence. You know you're moving into the realm of a lot more Casca operators here at the application landscape do I need to scale up. Do I need to scale down based on what it's doing, Africa's, obviously handling the balancing of the data streams.

A

For us, it's awesome and configuration is code, so we managed now one docker one helm chart and then we're just managing configuration that we can do and promote into production. Just do merge, requests into different environments and we deploy our desired state into production because, if you're not getting into production you're, not making any money and all your money, your business value is generated by doing the business logic, which is things like this so state. Yes, you, you will have state.

A

So if you do have state, if you are aggregating, you are joining use the state for States, so I said you can deploy via lenses. If you want to you can do for deploy by helm, that's fine, we'll see it will monitor it. However, we don't know who deployed it. So if you go via lenses where you have an extra bit of security and there we have rule-based controls, we have access policies on catechu itself.

A

We have topic black listing white listing as well, so we can not only stop you seeing the data, but we can stop you deploying a connector or a processor. That's gonna! Actually, maybe take that data and write it somewhere else, plus you get all the other extra goodies that come with lenses as well, the WebSockets, the rest api is the clients. We have monitoring integration with prometheus alerts with other managers as well.

A

So at the end, what we get to is that we can build your entire data pipelines simply and easily using configuration using sequel. We have lenses it's sitting. On top, that's doing the monitoring during the deployment we have a UI that we'll move to in a bit.

A

There allows the views of experience on it and we also have a lens of CLI as well written Gor line, so we're probably going to be doing a lot more integration with helm as well there. So we can actually say deploy lenses then deploy your application flow. Maybe you want some monitoring, maybe you want the Prometheus operator with it. Then we can give you the whole the whole shebang.

A

We don't deploy cathegory at the moment, bring your own Kafka, there's lots of good distributions out there and maybe we'll be supporting more than Kafka in the future. Patchi pulsar nice piece of kit, so I'm gonna, move on to a quick demo I'm going to ask Antonius, maybe to help me exit PowerPoint.

A

So this is lenses it's, but on the screen. How do I get it on the screen.

A

Now I can't see.

A

So this is lenses. This is our main application. So what you're looking at here is the dashboard. You can actually see that we have a number of schemas here. We have a number of connectors running. We have. A number of processes is actually what we're interested in.

A

These are all the consumers that are happening on it, audit trails, who did what there's me practicing a few minutes ago, and these are alerts that are firing, so these are actually alerts will be coming out from the loan manager if you want to so this is quite difficult doing it this way, but.

A

So this is the topic screen so where you can see all the topics that are in the in the system that the kafka that were monitoring, ingest rates, etc, configurations about all the different properties that was the replication fact, the key types with its Avro, whether it's Jason etc. So we can drill into a topic this one. So what you're seeing here is sequel fired directly on top of Kafka. So this is a live, continuous queries coming out with sampling, because we don't want to bring the browser down we're going to kill the browser.

A

If there's too much data, we can then drill into the stream to see what what what the data is, so that you know we get a lot of use cases, for example the traders risk managers. We want to look at what's happening when the trading had different trades they're flowing through the system. We can also do it historically as well, so this is firing. You know show me the data from this point in time for this time range. If we want to, we have different different views. We can flatten the data out.

A

We can show you the raw. Basically, this is the raw Jason. Well, it's that have Robert converted to Jason it's coming through various different information about the topics themselves in Casca, how many partitions, how it's being replicated across courses alerting everything on this as well and who's consuming from this topic and what the process of lag isn't there. So that's a metric that you'd want to scale. Maybe I want to scale automatically up, because this process too slow, okay, kuba nice, give me an extra runner, so I'm gonna.

A

Obviously there's a lot in now: I'm not going to it's, not full bloom demo of lenses, but what I'm gonna do is I'm gonna create another processor, so this is actually pointing at one of our little mini Cubans and Sue's book doesn't matter.

A

You so I'm gonna give it a name.

A

Cube Khan just spill it right here.

A

I'm going to deploy a mini coop, we could deploy into any cluster. We want this lenses instance is actually sitting outside kubernetes, but you can deploy inside if it's deployed inside it's only going to deploy to that kubernetes cluster book. We have some clients that you know they have a lot of kubernetes clusters.

A

We can pick a namespace, so I'm just going to stick it in the default. Namespace and I'm gonna, give it a pipeline. All this is going to do is just going to attach another label onto the port right and then it's going to collect, get collected and gmx metrics. So you can start asking questions of get how to promote this pipeline into production. What's the metrics of this pipeline? Imagine your date, scientist and you're interested in just your flow.

A

You want to be able to maybe tag your flow and be able to see how it's performing so.

A

Copy/Paste into here this is out sequel, so all I'm doing is selecting and filtering from the topic we were looking at before. So this is just going to say: okay right, it's a simplistic, it's a simplistic example, but all it's doing is filtering out data where the speeds greater than 10 at the bottom. There we're setting the deserialize to type that the key in the value value, because in this case it's Averell and we're actually going to sit or to create a topic.

A

So these are all properties that you can set on there on the consumers and the producers of of category itself. Now you can do a lot more than this. You can join. No. Maybe you forgot time or sure an example of that in a bit.

A

So if I click create demo God's Wi-Fi permitting, we should have a new processor there. We can see that the pods are being spun up into cube. Anita's, yes, got lovely pod names, we have a topology that will come through. So this is the actual topology that's coming out of the CAF extremes after it's gone through a patrick color site, so we can then drill in to see what its it's doing, for example. Here this is yeah. We can make that pretty able. Basically, that's the filter, that's happening.

A

We have the raw data coming in we're doing this, let and then we're pushing data out. So we should. Yes, it comes so we have data, that's coming out. So all these values, on the other end speed greater than 10.

A

We can drill into the monitoring as well, so here we're seeing the lag on what's happening on on across these five five runners and now I can't see what that number is, but this spun up and they've done half a million messages each since we start them and we can drill in a bit further. We get bit more metrics on what is consumed, what is produced and if we want to let's scale it down, actually, let's go for it.

A

Let's give us some more now we're getting 10 more runners, so we've just deployed a sequel processor in Turku beneath it's nasty and actually do that from Python. Do it from Gore line, whatever doesn't matter to the rest, the API for it and what's also a more interesting, is if we've got the topology right, because we're deploying Michael services but I want to see my end-to-end flow. You deploy a Michael servers, you deploying it actually as a end-to-end service, so we're deploying everything into kubernetes. This also works outside kubernetes as well.

A

We know what you've deployed. We know the topology. We know how to stitch it together, so we can actually build the bigger picture. So if you think of things like GDP, our Kafka is not going to solve. Gdp are no way, but you can start answering some of the questions. Where is my data going to what? Because we have schemas as well, what data do I have? Where is it going? Who consumed it?

A

It's not going to get rid of your data for you, the right to be forgotten, not quite going to work, but there you go so here's our processor that we deployed again it's the same information that you were seeing before metrics coming in it catches up. So obviously there the rate comes down, and that's it really. One last bit before we go to questions is that we have our connectors, obviously as well so I want to I want to do insert into Cassandra. So I can tie my case equal statement insert into.

A

A

Into this is not actually going to run because we don't have a Cassandra entrance select star from topic, something like that.

A

First deploy and now you, if you've written another piece of sequel configuration configuration, is called sequel, is configuration to deploy and write data out into Cassandra. It could be elasticsearch, doesn't matter spin up lots of them again, do it from whatever language you want, because we have the we have the client, bindings and I think I. Think. That's it really so questions.

A

We are looking at the different connectors that are out there. I think Google do quite a good job on their pub/sub one that they have so we're not necessarily gonna write that ourselves. There are some other ones that are a bit flaky in the open source world that, where we're taking over we're going to be doing a hive connector, for example, for or seeing Park a support as well a lot on the user side we'll be doing as well. Google pub/sub.

A

It seems to be doing a good job from what we see so no immediate plans to do it at the moment.

A

Yes, so we were asked actually and we've got an open PR. So if you want to do a click house, connector then feel free, there's a Norton Pyaar to do that, but it has a JDBC driver for it. So there is a JDBC source, sorry sink out there, that we were actually commissioned to write a long time ago. It doesn't belong to us anymore, so you could just use that now, where they click house has a more performant API that you want to rewrite somewhere else. Then maybe.

B

I, don't know, I, don't know the details.

A

Of click house.

A

Well, a lot of people are, you can also export everything out from the UI as well. So what we're working on as well is more integration for a bit creating a poor request, for example in in lenders itself, but you can actually just export everything else out and put it into whatever versioning strategy. You want to have some people, you know if they're clients who have a branch environment and then they'll fill the date signs as me, or the traders may use lenses as an actual just a playground.

A

Then they'll export out the configuration, and then that goes into a version, control and whatever process controlled. The different people have to promote into production, but we are working on it because we want to be able to say: okay, there's this topology. This belongs to me. I want to take that topology and I want to promote that through production. There's some tricky parts to tackle in their secrets, so a calf could connect. You have passwords all over the place for databases, they're actually just stored in Kafka.

A

So if you can actually query calf Bay, you can see passwords, we mask it. You know we do a bit of protection on there, but you how do we handle that? Maybe fault integration? This is something that we're or it's on our roadmap.

A

Conceptually they do the same thing we're slightly different. We used Apache color site parser, so that's a bit more of an industry standard, for example sparklink. They use that there is some technical details of. Why we're better. We think that, obviously we're objective. We allow you to join on multiple fields, for example joining value from the key on to on to a value from the value as well.

A

We obviously deploy natively now into kubernetes, eventually, there'll be parity between the two in the long run, we're not tied to Kafka I've mentioned it before they're, all the streaming technologies out there. You know why not have the same sequel engine across everything.

A

For example, you know we can there's no reason why we can't deploy spark job spark I've in raishin. Now with cuba, news we're not tied Catherine, but conceptually they do same thing. We just think we've implemented better, but we're objective.

A

Any other questions, no okay. Thank you very much.