Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2019 (San Diego), 22 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Deep Dive Into the Latest Kubernetes Scheduler Features - Abdullah Gharaibeh, Google Inc.

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Deep Dive Into the Latest Kubernetes Scheduler Features - Abdullah Gharaibeh, Google Inc.

Kubernetes Scheduler is the component of Kubernetes that assigns pods to nodes based on the configured scheduling requirements. Users can choose to run their clusters with high resource efficiency, high reliability, or other custom policies. The scheduler also implements a number of critical Kubernetes features, such as "Node Affinity", "Inter-pod affinity and anti-affinity" and the new "Even pod spreading" features. This talk will provide information on recent SIG Scheduling projects and features, including the the scheduling framework and even pod spreading. We will dedicate about half of the time of the presentation to audience questions and users' feedback.

https://sched.co/UahP

A

Hello, everyone, my name is Abdullah gorobei I'm from Google and I'm, the co-chair of six scheduling today, I'm gonna talk about what we've been up to in say, scheduling for the Pat in the past two Childress's.

A

So this is the agenda for, for today's talk, so I'm going to quickly go through a quick introduction of what the schedule is and then an assorted list of recent developments and planned features.

A

My plan is to go through this quickly, like maybe 20, 25, minutes, 25 minutes and then leave the rest for for questions. So what is the schedule so like everything else in in kubernetes? It's just another controller. This controller usually runs on the master, pre watches for pods and nodes that are created on a CD and what it does. Basically, it assigns pods to nodes, has nothing to do with the pod life cycle at all. It doesn't actually create and run the pod in the node. It only thing it does.

A

Is it creates a sub resource in the pod object that says pod X should run on nor do I once this happens. The assumption is that the cubelet that runs on the node that was assigned to the pod should pick up that pod create the pod and started up at that point, this kid or his hand off what has nothing to do with the pod.

A

So, as I mentioned, the scheduler watches for pods and and nodes those pods that are added there are that the new pods are get updated. They are added into a scheduling queue and this scheduling queue is sorted by pod priority. So it's basically a priority queue and the scalar basically picks one part at a time. Runs it through so far has been two phases.

A

The first is what we call like filtering fees, so we run the part through a bunch of filters that decides where what are the feasible nodes that the pod can run on example. Filters are, as you would expect, is whether the node has enough resources to run the power based on the pod resource requests. Other filters could be, for example, Prada affinity like required port affinity. If the we need only to look at the nose. That has the pause that this pod has affinity with and and so on and so forth.

A

So after this phase, we filter out the nodes that are unfeasible and we end up with a set of feasible knows those feasible nodes goes through what we call scoring phase.

A

Basically, we have a bunch of scoring functions that rank the feasible nodes, and we pick the highest scoring. Node scoring functions can be, for example, preferred pod affinity rather than like required. So if you decide that I want, like I, prefer that pod XP run with pod Y. So if the noise that has pod Y, the norther has would have higher ranking than other nodes. Other priority functions, for example, is one that ranks nodes with. That has already the image that the Parden needs.

A

It has like, for example, a higher ranking and so on and so forth. So after we again after we score the nodes, we pick the highest scoring node and then we just do the binding that I mentioned earlier, and then the cycle goes on and on and on. If, for some reason, the pod ended up after the schedule, exciter will zero feasible nodes, we place the bot back and what we call a back of queue. So, basically, like the back of queue for each part has a back of timer.

A

We add it back to the active queue so that we can try to schedule the part again. So what I just described before this is what we call a scheduling, attempt and a pod. Could we could attempt to scheduler called multiple times and schedule, and this is goes until infinity, so until the project schedule, they will always be moving between the back of queue and and and the active queue we do have. Something called unscheduled will queue, but I don't want to get into too much detail on that right. Now.

A

Okay, so after this quick introduction, I want to talk about what we have been up to recently. So we have I have like five major developments that I want to talk about. The first one, which I think is the most important, is scheduling framework I believe in the passing pass cube. Cause have talked about our plans to restructure refactor the core of the scheduler into what we call a scheduling framework and, basically, what we wanted to do is we wanted to turn it into a pluggable component.

A

Like the previous talk, that was here, core DNS or if you know envoy as well, we wanted to turn the default schedule into an engine that executes plugins and you and- and the idea is that we wanted to define a number of extension points that we've identified from our past experiences with their scheduler and those extension points.

A

You can register callback functions and a collection of callback functions that define a specific behavior, we're going to call it a plugin and I'm gonna give a number of examples about this, but what we've identified is the following: the following session point so from the top left, the sort. So if you, what I'm describing here this this line, is the scheduling, a scheduling attempt of a plug-in? It goes through these phases, while we are trying to schedule a specific part.

A

The sort extension point is a little bit special in that you can only define a single plug-in that defines how we sort the scheduling queue. The default scheduling behavior is that, as I mentioned earlier, we sorted by pod priority, but maybe you don't care about priority. You want to schedule based on requests or resource requests or whatever custom, sorting behavior that you want. So you can't customize that now the second phase is again like. If you look at it is, there is pre, filter, filter and post filter. This is what I talked about earlier.

A

It's the phase where we define the feasible nodes, but we found that we needed another two phases, another two extension points before and after the filter that allows us to run like to optimize that behavior sometimes features like what affinity. For example, it runs much faster if we do some sort of pre calculation and that's what we would usually do in a pre-filter phase. So a pre-filter phase. We look at at the pod, for example, state or even the cluster state.

A

We summarize it and then for each node in the filtering phase, we execute the filters using the exact same metadata. Let's call it or pre calculation that we did in the previous. In the previous extension point, the post filter. We felt that we didn't really need the pre scoring, so we call just post filter. So you either want to release some resources that you are located in the and the fill and the filter or preferred the phase. You can also do some sort of pre scoring.

A

You can also do some sort of pre calculations or pre computations before you go into for we trying to score scoring the nodes and then there's a phase called normalized scoring, so even usually the way that we we do things that we execute those extension points per node. But then there there are some scoring functions that you want to normalize like after you have scored each node.

A

You want to do some sort of a normalization, and this is what this this extension point allows you to do and then there's a reserve where you actually allocate the resources. You decide that okay, this node, is going to take up these resources from this, in that node from this node, for example- and this is where we're going to update the cache that we have in the scheduler to determine that this. This part is going to run on this on this node, and this is the first phase. We call it a scheduling cycle.

A

The next phase we called the binding cycle, runs in a separate girl routine. So now the scheduler in the main go routine, will pick up another schedule, another part and will go through the green phase we have here and in parallel, the binding phase will run for previous parts that are already been associated with specific node in the binding phase, as I mentioned before, we try to bind the part with with the node.

A

The reason we do it in a separate go routine is because it requires calling into the API server, because we want to write and update the part object with whichever node the pod will run on and because, as you might know, we we do throttle components that talk to the API server, because we don't want to dust the API sever, and for that reason the binding might not like succeed quickly, and so we needed a separate go routine to keep trying until the binding actually succeeds.

A

Now we, we added an interesting accession point before that, which is what we call the pyramid, plugin and in pyramid. What we will allow plugins to do is to hold on to binding the pod until a specific condition is satisfied, and one use case that we had in mind is what we call gank scheduling. If you want to not actually start the pod until all the pods that belong to whatever, like, for example, we can define them using a label selector.

A

Schedule label so they have been reserved and like some resource in specific node. Only at that point you want to basically say like release: okay bind all those parts together and then we will schedule together. So that's one interesting use case. We had as trying to make a schedule sort of friendlier for batch workload. I don't think we are there yet, but I would picture, but we are trying to make it as such.

A

So there are a number of benefits for a schedule framework. Again, as I mentioned, it makes it kind of easier to extend and isolate features. So before it was like, the specific feature was spread across different files and every time you want to add a new feature. You know, like these ugly patches was harder for developers to actually fork the code and add new features, so we wanted to be more pluggable and basically isolate the features into specific.

A

All the feature will be in a specific file that implements all the callbacks that gets executed in these extension points and, last but not least, not least previously, we only had predicates and priorities, which is the one I discussed earlier scoring and and filtering now, I wanted to add more more behavior and more complex behavior, as I mentioned, the permit extension point and Aang gang scheduling.

A

So what is the saddle scheduling framework like I'm, we're really happy to announce that the framework itself is fully implemented? It's an alpha right now, but then I think the next big, like the other biggest thing that we did in 1.17 and I want to thank the six scheduling community for a lot for allowing us to do that.

A

Is we wrapped all existing predicates and priorities into plugins, so we had now we had a chance to actually really make sure the scheduling framework its implementation actually works covers mostly at least the cases that we already have for the features that the scheduler implements.

A

It requires us I think over 100 PRS in the past few months, so it was a massive massive effort during that, though, we don't want to break existing functionalities, we don't want to break the existing API that allowed users to define predicates and priorities through what we call policy policy files, and so we added this translation layer that translates legacy policy file definition of predicates priorities into what we have now as as plugins, so those plugins that trap the predicates and priorities they don't actually have substantial amount of code on them.

A

The only thing they have is they actually falling back into the older predicates priority functions, and so so far we have two execution paths is one is active. One is not in the course scheduler, hopefully now in milestone, two or 1.18 we after we make sure that people like in 1.17, we have no major regressions or or bugs we're going to completely duplicate all the predicates and priorities and move their coding to native plugins, basically run as native plugins, and that would allow us to duplicate the old policy API.

A

And hopefully we will have the component config in GA by that time and the replace is going to be the component conflict, plug-in API for it and milestone 3 and 1.19 we're going to actually duplicate the policy API completely and remove it and implement more features that we have in the scalar as plugins like preemption, and the sorting logic and the binding logic actually moved them as as plugins and hopefully framework could be in GA.

A

So in the second improve. The second development is performance improvements in 1.17. During this major refactoring, we identify the number of opportunities to improve the scheduler behavior. So we managed to improve the scheduler scheduling latency by 2x, but that's only for large clusters like 5k clusters and the other thing. What we managed to do is also improve, preferred pod affinity. That was an algorithmic improvement.

A

We managed to change the complexity of the algorithm from and and P, where n is the number of nodes and P is the number of pods to to P, which is basically depends on how many poles you already have in the cluster because affinity depends now. You need to look at whatever parts they have running and then based on that, you calculate where I'm going to place the pod.

A

Another thing that we felt was a little bit lacking is improved observability. We had already a bunch of metrics that the scheduler produced, but we felt we can do better. So we we try to to improve things across three different categories. The first one is latency so tip. Until recently we had scheduling latency of the attempt, so you don't know how long a pod was. Actually it took. It took us to schedule a pod, but the only metric we had was the scheduling attempt.

A

As you know, a pod can be attempted multiple time so with it. So we add a new metric to track how long it actually took from the time we picked up the pod from the informal cache until the pod was bound to a node and the other thing we we added is latency breakdown of of each scheduling, step or or plugin. So we have better tracing and ability to break down by. If you want to see okay, why is my partner getting scale? Oh, my my schedule is slow.

A

You can now break down by plugins, basically kinds you can see if pod affinity is the one that is what affinity plugin is the one that is, you know, slowing things down, etc. The other thing is a catalyst traffic.

A

We added metrics to track number of incoming pods per second, so you can see how fast is scheduled and like draining that that that queue and again, like scheduling attempts per second, all those we felt are lacking in trying to understand whether the schedule itself is misbehaving or is slow or like, for whatever reason we just want more more insights into the scheduler saturation again is related to how much resources the scheduler is consuming in terms of number of binding goal, routines that I talked about before and the cache size, so the the scaler has a hazard has its cache like, after even on top of the informal cache, because we want to annotate the nodes and the path with extra information.

A

So we have our on our own cache that we actually snapshot for each scheduling cycle so that when we schedule apart, we want to have distant state while executing all these extension points. So we were concerned about this overhead, so tracking the cache size which going to be proportional to how big is your cluster? That probably should help you also with like, if you look at the history, how you provisioned the resources for the for the scheduler itself.

A

Another thing that we've graduated that we implemented and we promised in previous few cons, is pod quality spreading I think it was one on one of the keynotes it was. It was mentioned before that we had a pod ante affinity, but partly wasn't really good enough, because if you, it will only allow you to place one pod in different topology domains with pod spreading.

A

Now you can define a skew, so you can say: okay, I want the pods that belong to a specific group to be placed in two different topologies, with a skew of maybe at most one. So you can always basically try to load balanced properly across the party domains.

A

I'm running a little quick because I want to leave time for for questions. We in 1.17 we graduated to GA a couple of interesting features. The first one is scheduling demon, seed pods before that demon seed pods were actually scheduled by the demon seed controller, so the demon sect controller was actually doing the binding, not the scheduler.

A

So what we did in 1.17 is the demon seed controller, just sets not affinity on the pod and then leave it for the scheduler to actually schedule the ball, and that is better because it gives a more consistent view of the schedule of what pods are running on which nodes and how much resources are being allocated.

A

The other one is think notes by condition. There are some resources that don't get allocated like disk pressure or pids. This can't be ID so and and sometimes the node gets into disk pressure or PID pressure, and even when you schedule the part in the node, even if you have enough resources not get get scheduled because of that, so the node controller.

A

What it does is it: it monitors the north disk pressure, for example, a depression and taints those nodes with saying, for example, this node is unscheduled because of there's some some resource pressure on it. Before that we were looking at what we call node condition now we're looking at its, which is the more canonical way of how to play sports on segregating pods and nodes, is using teens and generations.

A

There are a couple of plant features: they're, not completely scheduler related, but it's it's basically cross-leg. The first one is pod overhead, so with pot overhead. The problem is that when pods are starting on a node, they have none zero overhead, think about the post control post container that that is associated with every with every part.

A

Also, the overhead of the components running on the node that runs the pod, for example, cubelet has some state associated with the pause that running so typically, what the way that we deal with this overhead is that we we have a predefined amount of resources on the node for system components, but the problems that we ignore per pod overhead, like the partner as I mentioned before, and now this problem becomes even more complex because we have sandbox pods, like think about divisor cut a container, they have their own agents running beside the pod and those consume resources, but those are not accounted for and see.

A

He comes overhead, so with pot overhead you can see at the runtime class level. This is how much resource is that for this, one-time class should be allocated per pod and it gets propagated, and what we've done in 1.17 is that we implemented a scheduler part of that. So now the schedule is aware of pod overhead and when it's schedule spots, it adds that overhead to the resources requested by by the pod, when it makes a decision where to place the node.

A

So the schedule part is implemented. The cubelet part is not, and that's why it's it's plant feature it's under development, the other one that's I, I really like, which is in place update of pod resources. That's that's really interesting. Until recently, the resource requests part of the pod spec was immutable. We can't change it. So, basically, if we change the API, several would inject the change, and so the only way to actually increase the resource of a pod is to actually delete the pod and create a new one.

A

It's basically restarting the pod now with features like vertical pod scalar. We want to actually try to change the pod mark producers association with the pod. Why not starting the pod? And so now we we have the cap already merged. We agreed on the design and hopefully, in 1.18, will start developing.

A

There are two parts of it: the scheduler part and the cubed part, as well as similar to pod pod overhead. Also there are many nonces he related to okay. How do I do the memory etc? Scaling? Because you know, for example, java based applications. They they take. All the memory feeds it will be hard to scale it down, but I mean this is this is a long discussion and that's the end of my talk.

A

That's a very yeah.

A

Yeah right so like right now you could configure, can have two schedulers running and did I mention you can set the scheduler name and you can configure them separately. What I would like to get to is I, don't I, don't like the idea of having two schedulers like and what I? What I would hope for is that the at the limit is to have what we call like scheduling profiles.

A

So you write similar to runtime classes or storage classes, and so every group of parts you can associate them with a specific scheduling, behavior and right now. It's something that we're discussing in scheduling is how we I did how we define that API that you express a scheduling behavior so that you can run multiple like workloads with different behavior on the same in the same cluster, because I mean heterogeneity is, is useful right, like it allows you to bend back better.

A

It allows you to to have better utilization, but in the short term your only solution is to run multiple schedulers or multiple clusters.

A

Weighted round robin.

A

Are you talking about like how we gonna pick the nodes from the queue? Sorry pick the pods from the queue yeah? So that's exactly like I, if you, so we have this sort plug-in here. This is gonna change, the way that you order the pods in the queue.

A

So the problem is that this is this is fixed. So if we can't have multiple sorting criterias, so you can have only one.

A

Yeah exactly so you would you need to write your own, your own, plug-in and and compile it in the difference.

A

I think you would like it's right now, it's so you would implement the you know. The list function. That's what you do. So it's much simpler, yeah.

A

So at the high level the permit allows you to wait on a part from a specific time like we have a timer and if that timer expires, the pod gets basically gets rejected. Now the idea is that, like, for example, you have poured one two and three so pod one will, when your plugin will return, for example, wait for five minutes, but two will come and then you say: wait for five minutes, then pod three will come and then you can what you can do like what the framework will allow you to do.

A

Is you can check which parts are waiting right now and so you're gonna have to define your own criteria of when to say: okay, those parts that are waiting, accept them and accept myself and move on so the state. Is there the pods that are currently waiting? You have access to them, and so it depends on the plugin implementation of when to decide how to how to accept all the pods are currently waiting state.

A

A

B

A

Basically, getting more insight into the scheduling behavior, so basically, what we wanted to do is to get more metrics into what the scheduler is doing. For example, how many pods is scheduling per second? What is the length of the scheduling queue? How long it takes to schedule a pod, so those metrics in collection? We call them like it's it's under observability. We wanted more insight into what the scheduler is doing.

A

Right I think we have Vivek here right. You have talked about it.

C

So the question was: how does a cholesterol task a little interact with a jeweler?

C

Basically, we have the scheduler code inside a cluster or a scalar, so the cluster order scalar needs to understand how many nodes to create for a certain amount of pending parts to be scheduled, and for that we need to understand how many parts we require. So we basically have to run some of the filters to figure out if we, if these many nodes are enough to run these many parts. So there is basically a one-to-one mapping between the amount of between the you know, cluster or the scalar version and the cluster and the scheduler version.

C

So if you're running, let's say kubernetes 116, you would require a cluster or a scalar, the exact same version, because the cluster order scalar, would use the exact share. You live code for that particular version.

C

Yes, you can choose thanks.

D

How does that work with the schedule or framework? Do you have just configure the autoscaler with the same plugins and stuff exactly.

A

D

A

Hope is that if we get to the point where, when you find this API that you can define those different scheduling, behaviors is the cluster or scale as well. Looks at that API, for example. Imagine that it is again CR th is like now it's the hot thing right. Do you have that state? This states basically defines what are the plug. Is that I want to run which waits and then the the cluster autoscaler basically has.

A

The part knows which exact you know a qualification has to configure when it's trying to evaluate that part for auto scaling.

E

Hey good talk lot of improvements, so one question about the gang scheduling: oh.

A

E

Question about the gang scheduling which you just mentioned: how can you make the make it consistent without the quota management on top of it? So I'll take an example. You have like three parts. You are waiting for two parts and the third part is not arrived yet and you are waiting for it, but if you are not doing a quota management at the admission control level, what is going to happen is you don't have enough resources and the modes other parts gonna come and take the resources and the first two parts is gonna time out.

E

Then it will go into work again into the scheduling cycles right and then this churn will keep on happening. Yeah.

A

E

To have quota management on top of it before we do that, I have.

A

To admit like we did not really make any like progress on that yet so those are very legitimate concerns and I think it's one of the plan features for us is to start thinking more and more about batch workloads and how we how we integrate them. With the with the with the current scheduler. We.

E

Have peloton which we can talk about it? Yes, does that yeah.

F

So, regarding the framework structure, do you plan to add any observability plugging something that produces metrics at some points of the pipeline? That tells you about the quality of the scheduling you're performing like. If you want imagine, you implement one of the modules apps like other or like the sorting, because you want to enforce a certain. You want certain jobs to be a schedule first, but you want to measure if those jobs actually caught sort at first booty mixers.

F

Not will it make sense to have a plugin at the end, at the pine stage that you will produce metrics that you can program? Well, your.

A

Own yeah I mean like some of these. Actually extension points are like we call them like information, and our vision is that some of them are going to be go to Ethernet logs or in met metrics, based on previous, like the behavior or in the previous accession points, did I get your question yeah.

F

But if I want in real-time, like, for example, what we do, for example, is we we produce metrics, saying: okay, we place one job for this user and that goes to Prometheus, and we in real time we can see what is the rate of jobs per user that we and that we are facing right and if I will have this framework, I would like to have to make a wrapper for the bin thing to add the metric that popular you know that propagation is somewhere else you create, if, if I could monitor all the jobs I get then without modifying anything just by making a plugin that observes whatever goes into pain and then I extract my custom metrics that have to do with my way of doing scheduling and I.

F

You know I just publish them, but I don't want to taint any snails in a structure. That's what I mean yeah.

A

I, don't have any thoughts on that I, don't know if it'll work, it'll work. The problem is that metrics in kubernetes in general, is like it's I. Think it's strict, some sort like you have to pretty find them and.

G

Do you have any updates on the D scheduler, whether that's going to be rolled into the schedule or at all, or is that just a side project? Still it's.

A

Still a side project, but it's Ravi's here, can you talk about it.

H

So the D scheduler is just recently updated to 116, so it has been kind of dead for a little while Oh Robbie is here so.

H

Yeah, it's I, don't know if there's any plans to move that soon, but it is a plan to move it to GA eventually and we're working on getting it up to pace, and you know a spot where everyone can use it.

H

I

To add to what my tol, so we haven't thought about it, but if you think that there are some use cases where T scheduler would actually benefit, if it's part of course scheduler we can make it as part of a plug-in at some. So it depends on the use cases like if you have something please let us know we'll be more than happy to look at it.

B

What city scheduler.

I

Did you attend the intro yesterday? I'm sorry miss kidding, so this scheduler at high level is an a Victor whenever the scheduling decisions are not met any more. For example, once the part comes into running state, the default scheduler would not even care about the parts anymore.

I

So at that stage, if the scheduler sees that there are some parts that are in running state and they are not conforming to the scheduling constraints anymore, it would evict those type of parts again, while everything we have some constraints in place like we will not have with every part. But this is what is the job of a scheduler? There are some other use cases for it too, like for example, if you want the cluster to be defragmented say there is lot of fragmentation we can.

I

We can use these scheduler to as a high-level defragmented, defragment 802.

B

So I have a question for the sorting function, because that's assumes you already have all the ports but imagine making a batch case. You have like a stream of ports. Can we make that interface like more, like kind of like so.

A

What we have is actually a priority queue and what you're gonna give is the compare function? So that's what it is.

B

It pushes like many times some functions could be more confident and just compare function. Yeah.

A

B

A like a Yankee writes okay, so you have to paste all the current entitlement of that queue to decide who's. Who should move to next right.

A

If ya, if you, if you have like a specific like really, you know like use case, for it, be more than happy like if you can find an issue that the framework right now is again, it's still in alpha. We are interested to improve the extension points that we have to cover more use cases.

B

B

Yeah, which is awesome but now my question is: we also have a lot of use case, which we kind of need in place contain update, for example, like you know, for some machine, meaning use case inference, we have some influence container and then we'll also want treat each model as a container, so they are in the same pod, but we want to upgrade those model containers without print out. The inference container, I think.

A

It's fair container, like the resources, are actually different per container, so when I user subscribes like the name, is misleading, it's actually a container like you when you when you, when you modify the resources, your.

B

Mocha you mean like a little modified resource of Canada but like if we wanna kind of replace the image of the Canaries they're gonna be supportive. Oh okay, yeah we gotta bring down the port of.

A

B

Can anywhere be restarted but, like the other, can let's say you have one part with two containers? You know if I'll only replace the image of one container, then I don't want to bring down here right.

A

So you just want to restart one container. Yes,.

B

A

That's a good signal question you.

B

A

Nobody that that's a like yeah I, didn't arresting.

I

A

D

So it sounds like for the scheduling framework your plugins have to be compiled for environments like a gke. It's not really available right. Is there a demand for making like webhook plugins? It seemed like I had heard about performance concerns, but it seems similar to like a mutating webhook admission. Controller would have the same sort of performance impacts, not.

A

Likely that's the problem, as I mentioned, like it's it's performance once you start making these calls to various. You know, yeah.

D

My point is, it seems, like the amount of performance impact would be similar.

I

Are you talking at the scheduling stage or the pod admission stage? If the admission, sorry, if the web hook needs to be called during the scheduler or after or at some stage of scheduler, it is already available? We have a HTTP extender like a predicate stage or priority stage or bind stage. You can make an HTTP call if that's what you are asking for and that still exists and that will exist continue to exist as well.

I

But it's it exists in.

A