Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

contrib/envoyproxy: envoy external processing support #2895

Open
wants to merge 11 commits into
base: main
Choose a base branch
from

Conversation

e-n-0
Copy link
Member

@e-n-0 e-n-0 commented Sep 27, 2024

Motivation

This is the part 1 PR to support Envoy's External Processing.
You can find all related document for this implementation in Confluence ASM - GCP Services Extensions.
You can find the part 2 of this PR here.

What does this PR do?

This PR adds a new gRPC Interceptor (StreamServerInterceptor) to support the interception of ext_proc v3 calls to gRPC server. When the interceptor is applied, all messages of the external processing protocol are instrumented without returning an handle to the original server code. The implementation of a server using this instrumentation can be found in the part 2.

The implementation includes:

  • Analysing synchronously HTTP Requests and Responses data (headers, ip, path, host, status code)
  • Blocking requests (supporting blocking with content-type and redirect)
  • Some refacto of existing code to handle more easily crafted requests without an actual valid http.Request object.

Tests

This PR includes unit testing in the envoy_tests.go, simulating scenarios of malicious or benign requests, validating span tags, security events and blocking results.

System-tests have been implemented on this PR. A new external-processing scenario has been added in the golang stage.

Reviewer's Checklist

  • Changed code has unit tests for its functionality at or near 100% coverage.
  • System-Tests covering this feature have been added and enabled with the va.b.c-dev version tag.
  • There is a benchmark for any new code, or changes to existing code.
  • If this interacts with the agent in a new way, a system test has been added.
  • Add an appropriate team label so this PR gets put in the right place for the release notes.
  • Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild.

Unsure? Have a question? Request a review!

@DataDog DataDog deleted a comment from datadog-datadog-prod-us1 bot Sep 27, 2024
@DataDog DataDog deleted a comment from datadog-datadog-prod-us1 bot Sep 27, 2024
@DataDog DataDog deleted a comment from datadog-datadog-prod-us1 bot Sep 27, 2024
@DataDog DataDog deleted a comment from datadog-datadog-prod-us1 bot Sep 27, 2024
@DataDog DataDog deleted a comment from datadog-datadog-prod-us1 bot Sep 27, 2024
@DataDog DataDog deleted a comment from datadog-datadog-prod-us1 bot Sep 27, 2024
@e-n-0 e-n-0 force-pushed the flavien/service-extensions branch 7 times, most recently from fcbd354 to a587a09 Compare September 27, 2024 13:33
@pr-commenter
Copy link

pr-commenter bot commented Sep 27, 2024

Benchmarks

Benchmark execution time: 2024-11-28 15:42:06

Comparing candidate commit b0844b8 in PR branch flavien/service-extensions with baseline commit 63e7470 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 59 metrics, 0 unstable metrics.

@e-n-0 e-n-0 force-pushed the flavien/service-extensions branch 16 times, most recently from 4134743 to a96cc2a Compare September 30, 2024 15:26
@github-actions github-actions bot added the apm:ecosystem contrib/* related feature requests or bugs label Nov 6, 2024
@e-n-0 e-n-0 mentioned this pull request Nov 6, 2024
6 tasks
@e-n-0 e-n-0 added the appsec label Nov 6, 2024
@e-n-0 e-n-0 marked this pull request as ready for review November 6, 2024 16:50
@e-n-0 e-n-0 requested review from a team as code owners November 6, 2024 16:50
Copy link
Contributor

@eliottness eliottness left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for contributing 🙏. Couple of requests:

  • Normally you should not have to import any listener packages, nor to export functions from it.
  • Could you link to a manual CI run from the System Tests workflow pointing on your system-tests branch where the workflow run is green ? You probably have to add your scenario in the workflow for it to run
  • I suspect we need to add tests for the changes you did in the emitter/waf/actions package. Let's see if this would make sense to separate it in another PR
  • Let's review together the code of envoy.go on fridayso we can structure it better

internal/appsec/emitter/httpsec/http.go Outdated Show resolved Hide resolved
internal/appsec/emitter/httpsec/http.go Outdated Show resolved Hide resolved
internal/appsec/emitter/httpsec/http.go Outdated Show resolved Hide resolved
internal/appsec/emitter/waf/actions/http_redirect.go Outdated Show resolved Hide resolved
internal/appsec/listener/httpsec/request.go Outdated Show resolved Hide resolved
contrib/envoyproxy/envoy/envoy.go Outdated Show resolved Hide resolved
contrib/envoyproxy/envoy/envoy.go Outdated Show resolved Hide resolved
contrib/envoyproxy/envoy/envoy.go Outdated Show resolved Hide resolved
contrib/envoyproxy/envoy/envoy.go Outdated Show resolved Hide resolved
contrib/envoyproxy/envoy/envoy.go Outdated Show resolved Hide resolved
@eliottness eliottness force-pushed the flavien/service-extensions branch 2 times, most recently from 9630788 to 0929de5 Compare November 12, 2024 12:54
@eliottness
Copy link
Contributor

Consolidated system-test run: https://github.com/DataDog/dd-trace-go/actions/runs/11797728094

Comment on lines +19 to +25
// ResetStatusCode resets the status code of the response writer.
func ResetStatusCode(w http.ResponseWriter) {
if rw, ok := w.(*responseWriter); ok {
rw.status = 0
}
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This required because of this if statement.

@eliottness eliottness changed the title appsec: envoy external processing support contrib/envoyproxy: envoy external processing support Nov 12, 2024
contrib/envoyproxy/envoy/envoy.go Outdated Show resolved Hide resolved
contrib/envoyproxy/envoy/envoy.go Outdated Show resolved Hide resolved
contrib/envoyproxy/envoy/envoy.go Outdated Show resolved Hide resolved
contrib/envoyproxy/envoy/fakehttp.go Outdated Show resolved Hide resolved
contrib/envoyproxy/envoy/fakehttp.go Outdated Show resolved Hide resolved
contrib/envoyproxy/envoy/fakehttp.go Outdated Show resolved Hide resolved
contrib/envoyproxy/envoy/fakehttp.go Outdated Show resolved Hide resolved
contrib/envoyproxy/envoy/envoy.go Outdated Show resolved Hide resolved
contrib/envoyproxy/envoy/envoy_test.go Outdated Show resolved Hide resolved
// This product includes software developed at Datadog (https://www.datadoghq.com/).
// Copyright 2016 Datadog, Inc.

package envoy
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you please add an example_test.go file showing how this integration is used with some sort of real world example? 🙏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
apm:ecosystem contrib/* related feature requests or bugs appsec
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants