{"id":104491,"date":"2019-12-19T20:15:01","date_gmt":"2019-12-20T04:15:01","guid":{"rendered":"https:\/\/www.paloaltonetworks.com\/blog\/?p=104491"},"modified":"2019-12-20T06:22:49","modified_gmt":"2019-12-20T14:22:49","slug":"cloud-envoy-vulnerabilities","status":"publish","type":"post","link":"https:\/\/www2.paloaltonetworks.com\/blog\/2019\/12\/cloud-envoy-vulnerabilities\/","title":{"rendered":"Recent Vulnerabilities in Envoy Explained, Including Impact to Istio"},"content":{"rendered":"<p><span style=\"font-weight: 400;\">On Dec. 10, three vulnerabilities in the <\/span><a href=\"https:\/\/www.envoyproxy.io\/\" rel=\"nofollow,noopener\" ><span style=\"font-weight: 400;\">Envoy proxy<\/span><\/a><span style=\"font-weight: 400;\"> were made public, one of which was classified as \u201chigh severity\u201d and two as \u201cmedium severity,\u201d affecting all versions up to and including Envoy 1.12.1.\u00a0\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Istio, which relies on Envoy, is also directly affected by these issues. The vulnerabilities may affect many Kubernetes deployments using Envoy, including many that are managed by cloud providers. On Dec. 13, Google issued an email alert to all its Google Kubernetes Engine (GKE) users, urging them to upgrade all Envoy instances in their clusters. In this article, I will shed more light on these three issues, their impact and how they were fixed. I\u2019ll also present a proof-of-concept video of a full bypass of Envoy rules.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Before diving into the details, let\u2019s go over the fundamentals of the affected components:<\/span><\/p>\n<p><b>Service Mesh\u00a0<\/b><\/p>\n<p><span style=\"font-weight: 400;\">A service mesh is a transparent software infrastructure layer designed to improve networking between microservices. The service mesh ensures communication is fast, reliable and secure. It provides useful capabilities such as load balancing, traceability, encryption and more.<\/span><\/p>\n<p><b>Envoy<\/b><\/p>\n<p><span style=\"font-weight: 400;\">Envoy is a high-performance proxy designed for cloud-native applications. It was originally developed by the engineers at Lyft, and it grew to become open sourced and graduated to be a Cloud Native Computing Foundation (<\/span><a href=\"https:\/\/www.cncf.io\/projects\/\" rel=\"nofollow,noopener\" ><span style=\"font-weight: 400;\">CNCF<\/span><\/a><span style=\"font-weight: 400;\">) project. Additionally, it is a key component in the most popular service mesh \u2013 Istio.<\/span><\/p>\n<p><b>Istio<\/b><\/p>\n<p><span style=\"font-weight: 400;\">Istio is a service mesh, originally developed by Google, IBM and Lyft. It uses Envoy as a sidecar proxy, which means every microservice or pod has an Envoy running beside it and all the communication in the cluster goes through these sidecar components.<\/span><\/p>\n<p><b>CVE-2019-18838 \u2013 Denial of Service and Potentially Other Issues<\/b><\/p>\n<p><span style=\"font-weight: 400;\">Envoy can be customizable with different encoding filters. <\/span><span style=\"font-weight: 400;\">If it is configured with an encoder, containing a specific router manager API call, and it receives an HTTP request without the \"<\/span><span style=\"font-weight: 400;\">Host<\/span><span style=\"font-weight: 400;\">\" header, it will send back an \"Invalid request\" response. This response is dispatched through the configured encoder filter chain before being sent to the client. If an encoder filter access requests \"<\/span><span style=\"font-weight: 400;\">Host<\/span><span style=\"font-weight: 400;\">\" header, it will cause a NULL pointer to be dereferenced and result in abnormal behavior of Envoy.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">It may cause Envoy to fail and crash, but under certain circumstances, an attacker may be able to craft an exploit even more dangerous and execute code on Envoy instances.<\/span><\/p>\n<p><b>CVE-2019-18801 \u2013 Heap Overflow<\/b><\/p>\n<p><span style=\"font-weight: 400;\">Envoy can be configured to accept HTTP 1, and while doing so, it assumes that all the HTTP header value sizes are less than 4KB. However, in HTTP 2 there are no restrictions on the size of the values, and as a result, untrusted input can overflow the buffer allocated for the encoding of the headers and crash Envoy.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">So far, an RCE exploit was not published, but a proof-of-concept exists in which two malicious requests are able to result in bypassing Envoy\u2019s access control.<\/span><\/p>\n<p><b>CVE-2019-18802 \u2013 Policy Bypass and Potentially Other Issues<\/b><\/p>\n<p><span style=\"font-weight: 400;\">The security problem here is that Envoy\u2019s parser incorrectly fails to trim whitespace after the HTTP header value. This can result in many security issues like policy bypass, privilege escalation or information disclosure. For example, <\/span><span style=\"font-weight: 400;\">one of Envoy\u2019s features is the ability to reject HTTP requests by filtering the hostname header. The problem is that <\/span><span style=\"font-weight: 400;\">Envoy will treat \u201cforbiddenSite[.]com \u201d\u00a0 as a different string from \u201cforbiddenSite[.]com\u201d (notice the space character after the dot com) and forward the request instead of denying it. After that, the HTTP server will trim the whitespace to send a response.<\/span><\/p>\n<p><b>Proof of Concept<\/b><\/p>\n<div style=\"position: relative; display: block; max-width: 100%;\">\n<div style=\"padding-top: 56.25%;\">\n<p><iframe loading=\"lazy\" width=\"300\" height=\"150\" style=\"position: absolute; top: 0px; right: 0px; bottom: 0px; left: 0px; width: 100%; height: 100%;\" src=\"https:\/\/players.brightcove.net\/1050259881001\/default_default\/index.html?videoId=6116874329001\" allowfullscreen=\"allowfullscreen\" webkitallowfullscreen=\"webkitallowfullscreen\" mozallowfullscreen=\"mozallowfullscreen\"><\/iframe><\/p>\n<\/div>\n<\/div>\n<p><b>Mitigation<\/b><\/p>\n<p><span style=\"font-weight: 400;\">On the same day the vulnerabilities were disclosed, Envoy released version 1.12.2, which contained the fixes for these issues. Istio released version 1.4.2, which contained the new Envoy as the sidecar proxy.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Although all these vulnerabilities require special configuration to successfully exploit, I would still highly recommend all Envoy and Istio users upgrade their instances to the latest versions as fast as possible. This is not just because these vulnerabilities can lead to a denial of service, but because their impact is not yet fully understood, and some exploits can potentially lead to much more damage. In a worst-case scenario, this could even potentially lead to a remote code execution.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">The original advisories can be found on <\/span><a href=\"https:\/\/github.com\/envoyproxy\/envoy\/security\/advisories\" rel=\"nofollow,noopener\" ><span style=\"font-weight: 400;\">Envoy\u2019s Github security page<\/span><\/a><span style=\"font-weight: 400;\">.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>In light of 3 recent vulnerabilities, we highly recommend all Envoy and Istio users upgrade their instances to the latest versions ASAP. <\/p>\n","protected":false},"author":663,"featured_media":103306,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[6768],"tags":[6964,6965,515],"coauthors":[6869],"class_list":["post-104491","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-secure-the-cloud","tag-envoy","tag-istio","tag-vulnerabilities"],"jetpack_featured_media_url":"https:\/\/www2.paloaltonetworks.com\/blog\/wp-content\/uploads\/2019\/11\/Container-Security-Featured-Image.png","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www2.paloaltonetworks.com\/blog\/wp-json\/wp\/v2\/posts\/104491","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www2.paloaltonetworks.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www2.paloaltonetworks.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www2.paloaltonetworks.com\/blog\/wp-json\/wp\/v2\/users\/663"}],"replies":[{"embeddable":true,"href":"https:\/\/www2.paloaltonetworks.com\/blog\/wp-json\/wp\/v2\/comments?post=104491"}],"version-history":[{"count":10,"href":"https:\/\/www2.paloaltonetworks.com\/blog\/wp-json\/wp\/v2\/posts\/104491\/revisions"}],"predecessor-version":[{"id":104642,"href":"https:\/\/www2.paloaltonetworks.com\/blog\/wp-json\/wp\/v2\/posts\/104491\/revisions\/104642"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www2.paloaltonetworks.com\/blog\/wp-json\/wp\/v2\/media\/103306"}],"wp:attachment":[{"href":"https:\/\/www2.paloaltonetworks.com\/blog\/wp-json\/wp\/v2\/media?parent=104491"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www2.paloaltonetworks.com\/blog\/wp-json\/wp\/v2\/categories?post=104491"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www2.paloaltonetworks.com\/blog\/wp-json\/wp\/v2\/tags?post=104491"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www2.paloaltonetworks.com\/blog\/wp-json\/wp\/v2\/coauthors?post=104491"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}