2 Nov 2022
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Closing Remarks - Matt Klein, Software Engineer, Lyft & EnvoyCon Program Chair
Closing Remarks - Matt Klein, Software Engineer, Lyft & EnvoyCon Program Chair
- 2 participants
- 13 minutes
2 Nov 2022
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
How Lyft Used Envoy to Rethink Microservice Development - Matt Grossman, Lyft
To scale Lyft’s engineers' productivity, the Lyft developer experience team pivoted away from costly and hard to maintain custom individual environments, and rebuilt the development environment around a shared staging environment.
The goal was to enable Lyft developers to run multiple versions of the same service (both in Kubernetes and on their laptop) without stepping on each other's toes. To do this the team designed a scriptable ingress proxy as well as used custom Envoy filters within Lyft’s service mesh to route traffic to these isolated instances, inject and propagate custom metadata, and offload the traffic to custom developer tools. The end result gives engineers special networking debugging superpowers during development.
In this talk, you’ll follow Lyft’s progression designing and developing these different components. You’ll deep dive into the custom Envoy filters and how they combine tracing, Original DST clusters, a custom xDS control plane, and local tooling to build this new developer experience. We will also cover realized benefits to Lyft’s engineering productivity as well as problems encountered along the way.
How Lyft Used Envoy to Rethink Microservice Development - Matt Grossman, Lyft
To scale Lyft’s engineers' productivity, the Lyft developer experience team pivoted away from costly and hard to maintain custom individual environments, and rebuilt the development environment around a shared staging environment.
The goal was to enable Lyft developers to run multiple versions of the same service (both in Kubernetes and on their laptop) without stepping on each other's toes. To do this the team designed a scriptable ingress proxy as well as used custom Envoy filters within Lyft’s service mesh to route traffic to these isolated instances, inject and propagate custom metadata, and offload the traffic to custom developer tools. The end result gives engineers special networking debugging superpowers during development.
In this talk, you’ll follow Lyft’s progression designing and developing these different components. You’ll deep dive into the custom Envoy filters and how they combine tracing, Original DST clusters, a custom xDS control plane, and local tooling to build this new developer experience. We will also cover realized benefits to Lyft’s engineering productivity as well as problems encountered along the way.
- 1 participant
- 32 minutes
2 Nov 2022
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Lightning Talk: Access Control and URI Path Normalization - Yan Avlasov, Google
Majority of Envoy’s access control policies are defined using the path component of the request URL. How can we be sure that URL Path Based Access Policies (PBAC) can not be bypassed? Two nominally different URL paths may in fact identify the same resource, and need to be normalized to their canonical form, before comparison. The normalization is standardized in RFC 3986, however it may not be enough to ensure safety of the access control. In this talk we explore the effects of URL path normalization on request access control, Envoy’s configuration options for path normalization and general principles for ensuring the safety of the PBAC policies.
Lightning Talk: Access Control and URI Path Normalization - Yan Avlasov, Google
Majority of Envoy’s access control policies are defined using the path component of the request URL. How can we be sure that URL Path Based Access Policies (PBAC) can not be bypassed? Two nominally different URL paths may in fact identify the same resource, and need to be normalized to their canonical form, before comparison. The normalization is standardized in RFC 3986, however it may not be enough to ensure safety of the access control. In this talk we explore the effects of URL path normalization on request access control, Envoy’s configuration options for path normalization and general principles for ensuring the safety of the PBAC policies.
- 1 participant
- 12 minutes
2 Nov 2022
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Lightning Talk: Distributed Tracing Without the Pain! - Sachin Ashok & Vipul Harsh, University of Illinois at Urbana-Champaign (UIUC)
Monitoring and debugging modern cloud-based applications is challenging due to their highly distributed nature. End-to-end distributed tracing (tracing individual calls through a request's lifecycle) has emerged to be essential in a developer's toolbox as they're critical to describing a request's flow through a microservice. Unfortunately, to enable distributed tracing, each service along the request tree needs to propagate a global request ID to help link the related requests (i.e., link incoming requests with spawned backend requests). Enabling such header propagation can be non-trivial for large microservices with 100+ independent services or for legacy apps where instrumentation is hard. This talk explores whether distributed tracing can be made more accessible by eliminating the need for application instrumentation. We describe a method that combines observations external to the app (using an envoy-based service mesh) with timing analysis of the requests to construct end-to-end traces. In an evaluation with a simple microservice, this preliminary method boosts trace reconstruction accuracy to 96% (compared to 77% for a baseline), and can help answer useful developer queries.
Lightning Talk: Distributed Tracing Without the Pain! - Sachin Ashok & Vipul Harsh, University of Illinois at Urbana-Champaign (UIUC)
Monitoring and debugging modern cloud-based applications is challenging due to their highly distributed nature. End-to-end distributed tracing (tracing individual calls through a request's lifecycle) has emerged to be essential in a developer's toolbox as they're critical to describing a request's flow through a microservice. Unfortunately, to enable distributed tracing, each service along the request tree needs to propagate a global request ID to help link the related requests (i.e., link incoming requests with spawned backend requests). Enabling such header propagation can be non-trivial for large microservices with 100+ independent services or for legacy apps where instrumentation is hard. This talk explores whether distributed tracing can be made more accessible by eliminating the need for application instrumentation. We describe a method that combines observations external to the app (using an envoy-based service mesh) with timing analysis of the requests to construct end-to-end traces. In an evaluation with a simple microservice, this preliminary method boosts trace reconstruction accuracy to 96% (compared to 77% for a baseline), and can help answer useful developer queries.
- 6 participants
- 21 minutes
2 Nov 2022
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Lightning Talk: More Intelligent Global Rate Limiting - Tianyu Xia, Google
Effective traffic management is the key to allow customers to ensure that their micro-services and overall architecture are highly available and highly reliable . How to design the rate limiting system for distributed systems handling millions of requests per second at Google scale? How to make the rate limiting system smarter to ensure fair sharing between various clients and handle the service surge in a particular region? This talk will go into details about the design of this more scalable, intelligent and performant rate limiting service and how your own service can benefit from it.
Lightning Talk: More Intelligent Global Rate Limiting - Tianyu Xia, Google
Effective traffic management is the key to allow customers to ensure that their micro-services and overall architecture are highly available and highly reliable . How to design the rate limiting system for distributed systems handling millions of requests per second at Google scale? How to make the rate limiting system smarter to ensure fair sharing between various clients and handle the service surge in a particular region? This talk will go into details about the design of this more scalable, intelligent and performant rate limiting service and how your own service can benefit from it.
- 2 participants
- 16 minutes
2 Nov 2022
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Welcome and Project Update - Matt Klein, Software Engineer, Lyft & EnvoyCon Program Chair
Welcome and Project Update - Matt Klein, Software Engineer, Lyft & EnvoyCon Program Chair
- 1 participant
- 13 minutes
28 Oct 2022
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Envoy Gateway Project Update - Daneyon Hansen, Tetrate & Alice Wasko, Ambassador Labs
If you're interested in using Envoy as an API gateway or Kubernetes ingress, this is your session. Envoy Gateway is a new project within the Envoy ecosystem that was announced at KubeCon EU 2022. The goal of the project is to attract more users to Envoy by lowering barriers to adoption through expressive, extensible, role-oriented APIs that support a multitude of traffic routing use cases. Agenda 1. Envoy Gateway Introduction & Demo 2. v0.2.0 Release Highlights 3. Roadmap 4. Q&A
Envoy Gateway Project Update - Daneyon Hansen, Tetrate & Alice Wasko, Ambassador Labs
If you're interested in using Envoy as an API gateway or Kubernetes ingress, this is your session. Envoy Gateway is a new project within the Envoy ecosystem that was announced at KubeCon EU 2022. The goal of the project is to attract more users to Envoy by lowering barriers to adoption through expressive, extensible, role-oriented APIs that support a multitude of traffic routing use cases. Agenda 1. Envoy Gateway Introduction & Demo 2. v0.2.0 Release Highlights 3. Roadmap 4. Q&A
- 3 participants
- 33 minutes
28 Oct 2022
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Fuzz Testing of Envoy - Adi Peleg & Teju Nareddy, Google
How can we be assured of the correctness and safety of the many Envoy parsers and state machines in the presence of untrusted or adversarial input? While developers cover main scenarios using tests, complex edge cases may be missed. Adversaries may be able to exploit these cases to trigger denial of service attacks, access Envoy process memory remotely, or trigger remote execution of malicious code. Fuzzing is an automated software testing technique that provides randomized input to the system under test (SUT). Some tests may use a variety of sanitizers to check for violations of memory safety, check for invariants expressed as assert statements or abnormal program terminations or timeouts. Other tests may compare behavior of different SUTs to the same input. This talk will include an overview of different fuzzers in Envoy, the OSS-Fuzz infrastructure for running fuzz tests, some bugs fuzz tests discovered, and examples of creating specific fuzz tests for ESF components in Envoy.
Fuzz Testing of Envoy - Adi Peleg & Teju Nareddy, Google
How can we be assured of the correctness and safety of the many Envoy parsers and state machines in the presence of untrusted or adversarial input? While developers cover main scenarios using tests, complex edge cases may be missed. Adversaries may be able to exploit these cases to trigger denial of service attacks, access Envoy process memory remotely, or trigger remote execution of malicious code. Fuzzing is an automated software testing technique that provides randomized input to the system under test (SUT). Some tests may use a variety of sanitizers to check for violations of memory safety, check for invariants expressed as assert statements or abnormal program terminations or timeouts. Other tests may compare behavior of different SUTs to the same input. This talk will include an overview of different fuzzers in Envoy, the OSS-Fuzz infrastructure for running fuzz tests, some bugs fuzz tests discovered, and examples of creating specific fuzz tests for ESF components in Envoy.
- 2 participants
- 28 minutes
28 Oct 2022
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Leveraging Envoy to Implement Micro-Segmentation-Based Security Policies - Hermann Lueckhoff, JP Morgan Chase
JPMorgan Chase application security architecture follows least privilege network and micro-segmentation principles. For instance, incoming requests from external users need to be validated in a designated security zone before it can be forwarded to upstream endpoints in a different security zone. Communication between these segments is highly regulated and involves various identity providers and different levels of authentication and authorization checks including token validations and exchanges.
In this talk you will learn how we deal with these complexities leveraging standard Envoy routing capabilities as well as Envoy filters such as JWT Authentication and External Authorization. AWS X-Ray Tracer is leveraged for added observability. For our token exchange requirements we utilize the External Processor filter with a Golang gRPC implementation leveraging Unix Domain Sockets (UDS) for improved performance and robustness. After validating the authentication status for a given incoming request we mint new tokens and inject them into the upstream request. The External Processor filter also us a clean way to logically separate standard routing requirements from very specific token exchange needs.
Envoy has become a strategic tool for operating in an elevated security requirements and the resulting additional traffic management complexities. We have been able to replace expensive, inefficient, and hard to maintain custom proxy implementations with Envoy and the External Processor filter. As our teams investigate Istio adoption, Envoy provides us added long term viability since we should be able to port our custom extensions into a service mesh environment. We have realized substantial cost savings on top of improved performance, agility, resource efficiency, and maintainability. Based on initial interest from other teams we see our Envoy-centric traffic management approach as an evolving pattern in our broader organization.
Leveraging Envoy to Implement Micro-Segmentation-Based Security Policies - Hermann Lueckhoff, JP Morgan Chase
JPMorgan Chase application security architecture follows least privilege network and micro-segmentation principles. For instance, incoming requests from external users need to be validated in a designated security zone before it can be forwarded to upstream endpoints in a different security zone. Communication between these segments is highly regulated and involves various identity providers and different levels of authentication and authorization checks including token validations and exchanges.
In this talk you will learn how we deal with these complexities leveraging standard Envoy routing capabilities as well as Envoy filters such as JWT Authentication and External Authorization. AWS X-Ray Tracer is leveraged for added observability. For our token exchange requirements we utilize the External Processor filter with a Golang gRPC implementation leveraging Unix Domain Sockets (UDS) for improved performance and robustness. After validating the authentication status for a given incoming request we mint new tokens and inject them into the upstream request. The External Processor filter also us a clean way to logically separate standard routing requirements from very specific token exchange needs.
Envoy has become a strategic tool for operating in an elevated security requirements and the resulting additional traffic management complexities. We have been able to replace expensive, inefficient, and hard to maintain custom proxy implementations with Envoy and the External Processor filter. As our teams investigate Istio adoption, Envoy provides us added long term viability since we should be able to port our custom extensions into a service mesh environment. We have realized substantial cost savings on top of improved performance, agility, resource efficiency, and maintainability. Based on initial interest from other teams we see our Envoy-centric traffic management approach as an evolving pattern in our broader organization.
- 1 participant
- 26 minutes
28 Oct 2022
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Lightning Talk: Envoy Upstream Enhancements - Alyssa Wilk, Google
Over the last year, Envoy's upstream feature set has grown dramatically due to the productionization of Envoy Mobile. New features such as automatic upstream protocol selection using ALPN, HTTP/3 upstream with seamless TCP fail-over, and Happy Eyeballs support have enriched not just Envoy Mobile, but Envoy as a dynamic forward proxy. This talk will run through these new features, and how they can improve your Envoy deployments.
Lightning Talk: Envoy Upstream Enhancements - Alyssa Wilk, Google
Over the last year, Envoy's upstream feature set has grown dramatically due to the productionization of Envoy Mobile. New features such as automatic upstream protocol selection using ALPN, HTTP/3 upstream with seamless TCP fail-over, and Happy Eyeballs support have enriched not just Envoy Mobile, but Envoy as a dynamic forward proxy. This talk will run through these new features, and how they can improve your Envoy deployments.
- 2 participants
- 10 minutes
28 Oct 2022
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Lightning Talk: Honey, I Broke the Things: Debugging Gray Failures in Production! - Radha Kumari, Slack
Migrations are one of the most challenging tasks we do as infrastructure engineers.
These are sometimes long, tedious and come with many technical challenges of their own.
At Slack, we switched from HAProxy to Envoy Proxy for all ingress traffic. Overall, this migration was a success, and did not cause any downtime, but even so, we ran into several interesting edge cases that caused minor problems, such as failing a small percentage of requests, or increasing latency for requests, or sometimes an unhappy bot.
Troubleshooting these sorts of 'gray' failures can be difficult, so this talk will discuss some of those facepalm moments: how they were detected, steps taken to investigate them, and how they were solved.
Takeaways from this talk include a specific set of approaches for debugging such problems with Envoy Proxy and other web proxies that we learnt via these events along with some engineering practices that eases the stress during a large migration.
Lightning Talk: Honey, I Broke the Things: Debugging Gray Failures in Production! - Radha Kumari, Slack
Migrations are one of the most challenging tasks we do as infrastructure engineers.
These are sometimes long, tedious and come with many technical challenges of their own.
At Slack, we switched from HAProxy to Envoy Proxy for all ingress traffic. Overall, this migration was a success, and did not cause any downtime, but even so, we ran into several interesting edge cases that caused minor problems, such as failing a small percentage of requests, or increasing latency for requests, or sometimes an unhappy bot.
Troubleshooting these sorts of 'gray' failures can be difficult, so this talk will discuss some of those facepalm moments: how they were detected, steps taken to investigate them, and how they were solved.
Takeaways from this talk include a specific set of approaches for debugging such problems with Envoy Proxy and other web proxies that we learnt via these events along with some engineering practices that eases the stress during a large migration.
- 1 participant
- 9 minutes
28 Oct 2022
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Lightning Talk: Manipulating HTTP Headers Using a Full Set of Substitution Formatters - Christoph Pakulski, Tetrate
Envoy allows for modifying HTTP headers when sending requests upstream and responses downstream. The syntax used for creating header content is the same as for creating access log’s entry. But surprisingly, only a small subset of formatters were available for header modification compared to the full set of access log’s formatters. This has changed and now all formatters can be used in both, in header manipulation and in access logs. This talk describes logic used when creating access logs and when manipulating headers. It also explains why using the same formatter in the access log can render different results than using it in the header. Note: This feature is still under development, but I hope that it will be finished and merged to main before the conference. See https://github.com/envoyproxy/envoy/pull/21932
Lightning Talk: Manipulating HTTP Headers Using a Full Set of Substitution Formatters - Christoph Pakulski, Tetrate
Envoy allows for modifying HTTP headers when sending requests upstream and responses downstream. The syntax used for creating header content is the same as for creating access log’s entry. But surprisingly, only a small subset of formatters were available for header modification compared to the full set of access log’s formatters. This has changed and now all formatters can be used in both, in header manipulation and in access logs. This talk describes logic used when creating access logs and when manipulating headers. It also explains why using the same formatter in the access log can render different results than using it in the header. Note: This feature is still under development, but I hope that it will be finished and merged to main before the conference. See https://github.com/envoyproxy/envoy/pull/21932
- 1 participant
- 9 minutes
28 Oct 2022
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Lightning Talk: Protecting Envoy: Overload Manager - Kevin Baichoo, Google
How can Envoy protect itself from OOMs? Envoy has a number of different protection mechanisms out-of-the-box -- how do they work? When should you use them and how should they be configured? Let's find out! Kevin will conclude with some experimental results using these protection mechanisms.
Lightning Talk: Protecting Envoy: Overload Manager - Kevin Baichoo, Google
How can Envoy protect itself from OOMs? Envoy has a number of different protection mechanisms out-of-the-box -- how do they work? When should you use them and how should they be configured? Let's find out! Kevin will conclude with some experimental results using these protection mechanisms.
- 1 participant
- 9 minutes
28 Oct 2022
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Pushing Envoy Beyond the Edge - JP Simard, Lyft
Over the last decade, infrastructure has been moving away from monolithic centralized servers and increasingly towards end users, with a focus on Edge Computing to run code as close to the people who are accessing it, wherever they are in the world. Envoy Mobile has pushed Envoy beyond the edge, all the way to your fingertips, unlocking a world of possibilities (and challenges) by being able to run Envoy on every node in the network chain from app to service and back again. Come learn how we’ve adapted Envoy to run as native embedded libraries for iOS and Android that feel right at home no matter the platform ecosystem; how we narrowed and in some cases exceeded the performance gap with established mobile networking libraries; what mobile-specific use cases or problem areas we discovered along the way, and how we solved for them; how Envoy’s rich observability tools helped us roll out safely to billions of requests a day; and how this is just the beginning of pushing mobile networking to the next level.
Pushing Envoy Beyond the Edge - JP Simard, Lyft
Over the last decade, infrastructure has been moving away from monolithic centralized servers and increasingly towards end users, with a focus on Edge Computing to run code as close to the people who are accessing it, wherever they are in the world. Envoy Mobile has pushed Envoy beyond the edge, all the way to your fingertips, unlocking a world of possibilities (and challenges) by being able to run Envoy on every node in the network chain from app to service and back again. Come learn how we’ve adapted Envoy to run as native embedded libraries for iOS and Android that feel right at home no matter the platform ecosystem; how we narrowed and in some cases exceeded the performance gap with established mobile networking libraries; what mobile-specific use cases or problem areas we discovered along the way, and how we solved for them; how Envoy’s rich observability tools helped us roll out safely to billions of requests a day; and how this is just the beginning of pushing mobile networking to the next level.
- 2 participants
- 37 minutes
