Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Serverless Integration on Kubernetes with Apache Camel K - Nicola Ferraro, Red Hat

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Serverless Integration on Kubernetes with Apache Camel K - Nicola Ferraro, Red Hat

“Serverless” is one of the most trending paradigms for designing applications, but what most early adopters find out is that doing something more than a “Hello World!” is hard. The more your application is deconstructed into smaller pieces, the more you need better communication patterns. Real-life applications also interact with tons of external systems and managing this complexity is hard. Camel K is a lightweight integration platform for Kubernetes created specifically to address these issues. It’s based on Apache Camel, the most powerful open source integration framework, and it leverages Knative to deliver integration patterns in a serverless way. We will show how Camel K works under the hood and, with coding examples, we’ll also demonstrate how Camel K makes it easy is to connect (almost) anything using integration patterns and the 300+ components that Apache Camel provides.

https://sched.co/ZeoP

A

Hello, everybody welcome to this session about serverless integration on kubernetes with apache community. um My name is nicola ferrara. I work for red hat and today we are going to talk about first of all, apache camel. What is this um camel k, how it fits in the serverless space and especially in k native, and then I will show you a demo and talk a bit about future development.

A

I'm nicola ferraro. As I said, I work as principal software engineer for redact. My twitter render is me underscore ferrari if you want to be updated with the latest news about apache camel, because I work on apache camera. I work on camel k also on redat integration and from time to time, I'll also contribute uh to k native.

A

So let's start talking about apache kamala. I know that some of you may not know about it, um so this will be a brief introduction, but uh I'm not going into much detail so we'll just give you an idea. So I think that many of you have experienced this joy of developing things. Software development is fun after all, so when we start a new application, for example, many of time many times, we want to do it with the best of breed of technologies that is available.

A

We want to do it serverless and we want to use whatever is needed at best suited for for the application that we want to write. But sooner or later you realize that. um Well, your application doesn't live alone in the world, because uh there are a lot of other application that you need to connect to, because you want to use their data because they want to use your data. So it's an ecosystem. We are putting your application, not alone, but it needs to communicate with others, and communication is hard.

A

So when you want to connect to integrate even two systems, you need to face a lot of communication problems, so one application may be synchronous, the other one may be asynchronous. They may use different protocols different languages, so they for sure have different business domains.

A

So you have to map objects from one side to the other and different data access pattern, and once you fix all these problems, you have another set of problems like what happens if the other application becomes unavailable, how to deal with consistency issues when there are failures in the other side, only on your side and also sometimes, you cannot change the other side, because we have different ideas and you have to do all the integration from your site. So these are some of the problems that happens when you try to integrate to systems.

A

So we need some tools, okay, and we need a tool. The perfect tool that can help us integrate so system should have, uh for example, connectors to be able to connect to any kind of technology protocol that is available from the other side, or also software as a service.

A

It should be based on well-known integration patterns, for example, because we don't want to reinvent the wheel anytime.

A

We need to connect another system and also it should work on and off the cloud on, kubernetes, okay native, but also uh without that- and it should allow you to integrate systems with a simple language, for example a domain specific language that can allow us to say, let's move data from here to here like this um all this is apache camel, but chicamo uh is a project, an apache project that starts some years ago and allow you to integrate a system with.

A

um So let's make an example to to make this clear. So, let's suppose that you want to create your application using a native, but you have to connect and share that with other applications, one developer with pulsar and the other one that uses kafka under the hood. So how do you share data with them? For example, you can create a camel integration that has this simple language from pulsar to canadian, with some transformation in the middle. So this is a camel language.

A

It's a domain specific language that allows you to move data, and you can look at uh at the language, I'm not going to explain on the detail of the language, but it's it's self, explaining I'm taking data from a pulsar topic in a specific namespace and converting it into a cloud event that can be received by my uh application inside kinetic. We'll talk more about the okay native later, and I can do it also in the other side, I want to move data from kinetic to cafe.

A

What they can do is to listen for all events generated by my application, okay native and then um send them to us kafka topic after doing some transformation, for example, in this example, I'm getting a file from s3 using one of the many components of aws 2s3 that is available in apache common and with these two integrations, basically, you can do whatever you want in the party area in the middle so use whatever you need, and specifically, we provide a lot of useful things for building applications that are serverless and based on creative.

A

As we will see. Okay, apache camel is a project that is more than 10 years, but this is something really cool because um we have a huge community. We have a lot of new users also recently, because we have started creating new some awesome sub projects. One of them is camel k and camel supports a lot of components.

A

346 as the last time I checked it all that components allow you to connect to a specific protocol to a specific external system, like the ones that you see here in the picture like twitter sales for but also old protocols and software as a service, but also middleware and other uh platform that are available out there. So everything you can imagine can be connected with apache camera nowadays, and so now we can talk about this sub project called camel k so camel k.

A

Is we say it's a lightweight integration platform born on kubernetes with serverless superpowers, so it's a project that we've started two years ago and it it's about running the camel dsl, the camel coded integrate system directly on a kubernetes cluster um latest release. 1.0 was released uh two months ago. Basically, so the idea of camel k is simple: this you create this simple dsl, this simple language. It tells uh how to connect two systems or more systems. You know not limited to only two and once you've created this file.

A

You have the cli called camel with the k and you type camel run the name of the file that contains your integration, and you are connected to an openshift cluster to a kubernetes cluster that works in any environment, and this integration basically run as uh apple with other resources created for let this integration run, everything in camel k is called driven. So you have just to write this code and everything else is managed by an operator.

A

uh It's called the temple k operator that you can install on the cluster it's available on helm, it's available on operator, retrohab and the basic workflow. Is this: the developer creates a file with integration routes.

A

The routes are the things that I've shown you before that connect systems. This browse is wrapped, are wrapped into an integration custom resource. It's a cooperative custom resource called integration. Then the operator takes this custom resource and converts it into running functions, services and whatever it's needed to run that integration from the code. Everything as I said, is called driven it. It means okay, let's skip the slide.

A

It means that you can write the integration here and uh it says that in this case, I'm exposing an api, I'm using the net http command component and then sending the data into k native. It means that um for the operator point of view, the predator parses this code and says: okay, we need to use the native http component, which is a camel library they can native component, which is another camera library. Then it pick all these libraries that are needed. It creates a container image for uh running this.

A

Integration camel under the hood is based on java, but we will see that we, we will have more optimization uh in the later in the latest slides.

A

um This creates a container image, for you creates all the services to the deployment, the secrets and the conflict maps to let this run and it can run as a serverless integration on on kubernetes.

A

So this is how it works. um Let's keep this and talk about canadian a bit so um camel k loves k native. So this is a love story that started two years ago, because the two projects started almost at the same time. This is not the official logo of camel k, but somebody put the common logo and the canadian logo together. So it reads as camel and k. So um we like this, but it's not the official logo camel k can work also without k native but um yeah, it's better when they run together.

A

Just this um just to be on the same page. I'll give you a brief. Really brief. Introduction about canadians, okay native is about serverless and it's a series of building block uh for running serverless applications on kubernetes, the building blocks are divided in two main parts, and one is called native serving that is about auto scaling and the other one is about kinetic event. It is about even driven applications. Okay, so when we talk about connective serving the most important building block is a custom resource called uh canadian service.

A

You can create a connected service to run your container, and if there is a request for your container for your service, then a photo will be created to serve the request, but as soon as no request arrives for a certain amount of time, this code is destroyed, so the service scales down to zero and it works also when you receive multiple requests, so multiple pod can be created by the native services so to serve these requests uh in parallel uh when using kamal ka with canadian sorry.

A

So, basically, nothing changed from a developer point of view, so you created the camel dsl in this case, I'm creating a a rest and point called path uh with the post meter that whatever you send to this endpoint is sent first to system one and then the response from system one is sent to system two system. One system two are just to place: order for uh one of the three hundred and fifty and sixty components that are available in apache gambler.

A

That can be any software as a service or any protocol that you can imagine this rest dsl. This camel dsl is converted automatically into a native service, so you have auto scaling automatically, but from a developer point of view, everything is code driven income. Okay, talking about kinetic eventing, um it's about even driven application. The most important building block here is called broker, so the broker is an abstraction. Is an interface for a publish subscribe resource.

A

You can send events to the broker and you can let services with the ones that you see on the right to be triggered when a certain, even to the event with certain features is published into the broker. So this is how it works when it comes to camel cake.

A

Mlk is really awesome to be used in this concept sub context, because you can can create even sources from many components that you have so you can take, for example, all the tweets that talk about a certain topic and send them into the broker so that application can subscribe later, but also works in the other side, for example, suppose that you want to um send emails to to your customers, and so you can create a chemical integration that converts an event into an email that is sent actually to the customers, and so from your application.

A

You can just publish an event and that will be converted automatically to an email that goes to your customers, but not anything here can be. It should be done with camel k. You can use chemique whenever it's needed to do integration with other systems or do transformations and use your the technology that is best fitted for all the rest, and so you can plug whatever you want in connect with anything.

A

Just one thing before we start the demo, I told you that the main way to use camel k is to create a file with the camel dsl, and then this file with this camel cli is converted to an integration, uh custom resource, and then the operator creates some running posts, creative services and whatever is needed to run the integration. But there is a second way to run camel, especially when you want to create sources, you can directly create a camel source. That is a custom resource that belongs to canadian.

A

So this is released together with k native, and uh so uh it wraps a chemical integration. So they have the same power, the same expressive power, but the camel source is compatible, for example with the kn uh toolings. That is the the the de facto uh cli uh it's the official cli for 4k native.

A

So it's recognized by that, but also from up by all the tooling that is based on top of the k native interfaces uh just that, but from under the wood they are uh almost the same thing, so we can now start the demo. Okay, this demo is about uh bitcoins. You know probably that today one bitcoin is worth about 9 000. That seems something crazy, uh but the question is: should I buy more bitcoins or should I sell uh if I have someone some bitcoin in my portfolio?

A

So let's try to answer this question by building a system that can predict uh this if the price is going up or if it's going down, so we will use an existing machine learning algorithm that some other team has developed and uh create something using camel k.

A

So listen for the real data that comes from the bitcoin market, uh push this data inside the uh canadian broker and send then create some breach to send this current value of the bitcoin versus us dollar to this machine learning algorithm list them from some prediction from from it and then send the prediction back to the broker, where there will be some syncs in this case I'm creating a sync that will convert the prediction into a message that will go into my smartphone.

A

So I see that if I need to buy or sell bitcoins directly and I'm using the telegram component from apache camera, let me share my screen.

A

And we can start the demo okay, I hope you can see the screen now. I have here an open shift cluster, where I've already created a namespace called the camel k native and I've installed the camel key operator that can be also installed globally. I've also enabled the kinetic broker here. So you see here two parts in the latest version of canadian. The the broker is multi-tenant, meaning if you can create it in your name space, but it won't create these two posts because they will be global um here.

A

I've deployed also the quercus based machine learning algorithm. We will talk more about workers later and it scales down to zero, because this is a canadian serving service. Okay, let's do um the magic with column k. So let's start with this, so this is an example made with didact, so you can follow it. There are instructions around it on on your own, you can find the links in the slides.

A

um Okay, let's start from the end, we want to listen some prediction from some prediction in the event measure from the broker and then transform this prediction into a message to our telegram chart. So, in order to do this, you can create, for example, using the camel cli an empty file like file.java, and it will create a file, but here because we don't have a lot of time, I will show you directly. The integration already done, so this is the the sync that will send transformer and a prediction into a message to our smartphone.

A

um You can read it as this, so we start from like a native event called of type the prediction.work, so we can define our own types. We transform it to a string, so predictor suggests to buy or sell at this price, and then we send it to uh telegram uh telegram chat. uh So this is this is using the telegram component. You know that in apache camel uh you have a lot of components. This is camelapatch.org and you have really a lot of components.

A

You can filter here and, for example, the telegram component that is well documented here and how to use it, etc. In this case, also, I've installed also plugin for the ide. So I can also about the completion here of the properties that are available for the component. Even if this is a string, so we convert an event into a message to our telegram board. So we'll see the message here in this telegram chat and that also my smartphone and we reply to with the empty string to the to the broker because uh the broker.

A

I expect that we can return some response, but in this case I just want to consume this event and send it to a telegram to run this. Integration I'ma run telegram single java and it also has loaded this property file. It is the clarity here in the model line, and uh this property file contains also authorization token to talk with telegram.

A

If we go back to um openshift that is running remotely on on the cloud we have created, this camel contribution, it support it. It is running currently, but it will scale down to zero if there are no events coming to that, and currently there are no event because we have just deployed this piece, so we want first of all to create some data, but before going to do it, let's see what has been created. The k in my laptop is cube. Ctl, so k get id ip means integration. There is an integration running.

A

You can see it also from this uh panel here that belongs to the vs code extension. We have this rescue plugin, so you can get. For example, ksvc. You see that there is a quarkus uh application that is running as a canadian service, but also telegram seeing is running as creative service, and you can get also the triggers that are available in canadian and we have a trigger here created for telegram why?

A

I will show you um so this trigger says basically with whatever it says, to the false broken canadian with the attribute type equal to prediction.workforce should be sent to the service called the telegram sync, but I have not created all this yaml uh by hand. This comes from the operator because what we have done is propel canadian to build your camera key operator to create a trigger from prediction.workers from the default broker, so that all this has been created by the camel k operator. You can add those additional filters.

A

You can also receive the different kind of event on the same integration.

A

Okay: let's go back to the diagram and create the source, so, as I said before, uh sources can be created as a chemical integration run with the canon cli, but they can also be created as custom results belonging to canadians or camel sources. This camel source is a yamo that contains the camel dsl but written in the animal. So we have many dsl so even tempting to write them in java. We have, we can write them in yellow, as you see here, but also in grooving and kotlin, and also in javascript, if you want.

A

So this is a variant of a chemical integration. We start from the timer component. Timer creates a periodic trigger. Basically every 10 seconds. We use the exchange component from camel to get from binance the current value of bitcoin versus cs dollar. We transform that into a json, and then we log and set the content type set the cloud event type to market bitcoin. Usdt so remember this and send it to the default broker.

A

So to create this we will copy cube, ctl, apply, insert market source and it has been created um yes now. So if I get, if I type source pn's also is using the kinetic cli and see this source and that has been created and it has also created a command k integration. As you see here, we can follow the logs and you will see that um yes, it's generating some json with this currency pair bitcoin sdt. The opening value and last is the value that we are interested into.

A

As you see here so last is the last observed value of bitcoins versus us dollars. So let's close this, this is running in the cluster. As you see here, this icon is because this is a standard kinetic source. Okay, let's create the last piece of the picture, so this predictor that bridge the quarkus machine learning algorithm. Just let's see one moment the machine learning algorithm. It contains uh samples and points that can receive adjacent, but it needs to receive only a table value.

A

So the current value, not the full json that we produce using the source. So we need some basic transformation using camel. So, let's do it um just I'll show you briefly uh the code without uh focusing too much, we listen from the json produced by the source uh using a connective component, so it will create another trigger, and then we send this using the red http component. Don't care about the error. This is just the tooling.

A

We send it to quarkus ml, using the sampling point using a post. We set the content type and do set the transformation from the json to only the last value here and reply, nothing to the broker and we take from the wire the response from the class from the quercus service, which is a simple service. It's not a real machine learning algorithm and we evaluate it. If the response is not empty, we emit the prediction. The prediction is something like buy or sell, and the value at which you need to buy or sell.

A

This is sent as cloud event is transformed into cloud event of type prediction.workers, so it will be received by the telegram sync integration. Okay, let's run it come run prediction breach, so it will run in the cluster. This has been already created.

A

um We can follow the logs from here, so it's already starting- and you should see this also here in the openshift console- you see that quercus and telegram has been scaled down to zero, but now quarkus is starting up, because this integration is calling it, and if this generates some prediction, also the telegram um integration will start up. Let's, okay, we see here that the prediction has been emitted, so the telegram endpoint started- and we should see here once this is ready.

A

uh The prediction to sell or to buy bitcoin yeah no buy and then sell. So I told you that machine learning algorithm is not so efficient, but yeah. You see that there are a lot of variation in the price of bitcoin, so I think it's it's okay to say, buy and then sell okay, but this completes the demo and I'm going back to the presentation now.

A

Okay, I think we are back to the presentation and let's talk a bit about the future. um So what I've shown you is that uh not showing this to you? But actually, but this is what happens when you run. An integration runs as a java pose, and you can also enable quark, because parkours is a framework for building high efficient java applications.

A

um But what we do now is to create the java application with open jdk with the jvm and then the quercus framework. What we are going to do in the future- and we are working currently on this- is to compile natively the code into a an an native binary that is comparable to what we achieve if we run a go application, for example, but it has the full power of all the components that are available in camel. We have started a project recently.

A

It is called camelparkus that is about running all the common components as native um applications on top of quercus, why? It is because if an integration now um can use, for example, even 200 megabytes- and so you may be incentivized to, for example, uh run together the integration, for example, the telegram sync and the prediction bridge could have run all together, so we can camel run all the files that you want to run and give it a name, and it will run as a single integration. But you don't want to do it.

A

You want to independently scale all the pieces of your uh code, so um with quarkus you can uh lower the amount of memory that is used from 200 megabytes to just 25 megabytes and also the the startup time you have seen. The telegram has started up in five seconds, so I've received five seconds later. The message on my telegram chat and on my smartphone, but with workers it can be less than one second, with all this.

A

So mandrel is a new project to run qr code run and grab vm on top of open jdk, and we are going to leverage that another thing that we are going to use is camellits. So this is. We are working also on this white hamlets, because many of you and also a lot of people, don't know anything about apache camera and don't know how to write a line of camera tsl, but they want to use these connectors that are available in camel. So camouflage are about that.

A

The experts will create um those connectors for twitter for sales for for service now and then make them available so that you can pick one of them and then configure some parameters, for example the query. I want to uh search all the uh tweets on twitter that talk about quarkus, configure the authorization token and then choose a destination. This will be available also on the openshift console.

A

For example, this nation can be a connective broker on a native channel like native service, directly or whatever you want, or even a kafka topic, um and this will run without writing a line of common dsl.

A

uh Another important things on which we are working on is the axiax is a framework for testing, for example, integration, but also general purpose service on the cloud um the x is based on bdd style testing. So uh you define a feature file like this. uh Let's test this feature that the api gateway works, given this url when we send a get to the selection point and we want to receive a 200k.

A

So this is sims network language, but this is accessible via yaks that is based on cucumber, so this will run and we can check can be used to check if your service or your integration in the case of camel k are working correctly.

A

Okay, other important future developments are web books, because uh sometimes so you need to have a pod, for example this. So the camera source needs to have a pod. That is always running because we were doing polling from the finance service because there is no workbook available, but for many points that are web books and we are working on improving the number of components that can use web books to be called when there is a new data and not to be always on and and be cold and cool.

A

Cada is another point where we're working on to have auto scaling not only for http based and points, but also for others, and if you want to learn more about apache camel, then there are other thoughts. Most of them are from that nation. uh Klaus, easton, dragos and tina have talked a bit about that. Also. We have a talk with by looking soleil.

A

If you want to learn more about the details, because we have not a lot of time, so if you like it, come okay, go use it and, let's contribute with anything you have in mind. You also sign in bugs. If there are- and yes there are, don't expect it to be perfect, but we uh we want you to join the community and uh help us to improve it in the future. So follow me on twitter me underscore ferraro and yes, that's it. Thank you very much.

A

Okay, so thank you for attending this session and I hope you have enjoyed it also. The demo- and I hope I have not talked too much uh about the details of pachikamalif and if uh yeah you know there are a lot of things to say, but yeah many of you don't know uh apache camel. So it's difficult to explain it in details. So I will answer some um questions that have been asked in the chat. First of all, um someone is asking what is actually apache. Camera apache camel is a framework.

A

uh It's a java framework. It has been around for a lot of time. It is one of the main apache projects and it is based on many components that are small libraries that are used inside, camel cake. Mlk is written in, go because it's basically a controller, an operator for kubernetes, but the runtime is still java running on jpm, and so it supports many languages like groovy code, linux, yaml and xml.

A

All of them can run on the jvm and we plan, as I said in the in the presentation- that to also run integrations on part, because with that advantage, quarkus is a framework for a highly efficiently highly efficient um java application that can run in the cloud. An advantage of quercus is to make sure that some applications, some integration in this case, can be compiled natively. So we want to compile our integration natively um to be small and to uh be much more efficient when and reactive, especially when running as uh connective integrations.

A

Another question um that was asked is, if you can run this demo on um kinda cluster, for example, on a local cluster. It's easy to run, camouflage on minicube.

A

The um the reason is a simple because camel k, as I explained in some slides- needs to create some images, because what you write is the dsl, but what you can run on a kubernetes cluster is a container image, so this container image needs to be put somewhere and while you have an open shift default registry that camel k uses automatically, while you can enable, for example, on mini cube register that is available out of the box, uh um it's the rages.

A

The internal registry is not available on kind, but, of course, kamal k can be also installed on kind. uh We also run part of our ci on a kind cluster, so yeah it's possible to install it. We will make sure we have also a guide on how to install camel k on time, because this question has been asked frequently in recent times.

A

um So this is it I'm not shown it. There is another question about: um can I um develop kubernetes scrolling job if we don't take a native? um What what? What this means is that, with camel k when you write integration, the integrations are materialized in what is uh most suitable to run your code.

A

So if you create a periodic integration, um the chemical operator understands that, um instead of creating a deployment that is always running uh the kernel operator, if you use the chrome uh component or the timer component of apache camel it creates a kubernetes chrome job after the box, and you will need kinetic to uh have this chrome job running, but it works on any kubernetes cluster and even a kubernetes class.

A

So this is it. I think I've answered most of the questions and uh yeah yeah. Others have been replied in line in the uh in the speaking in the question view. So thank you and thank you for joining and uh yeah see you.