[CrowdStrike] Add Support of Crowdstrike Event Stream #11773

mohitjha-elastic · 2024-11-19T12:27:25Z

Type of change

Enhancement

Proposed Commit Message

Add support of new input type to collect logs for the falcon dataset- CrowdStrike Event Stream via streaming input.
Update the minimum kibana version to 8.16.0
Add the entry of the crowdstrike event stream in readme

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Screenshots

elasticmachine · 2024-11-19T12:47:06Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

efd6

There are not tests here. Can you remove the comment in the PR that says how to test locally since that obviously has no effect on this addition.

Have you tested? What do you need in order to add CI testing?

efd6 · 2024-11-19T21:54:47Z

packages/crowdstrike/data_stream/falcon/agent/stream/streaming.yml.hbs

+processors:
+{{#if processors}}
+{{processors}}
+{{/if}}


Final new line.

efd6 · 2024-11-19T21:58:34Z

packages/crowdstrike/data_stream/falcon/agent/stream/streaming.yml.hbs

+redact:
+  fields: ~
+program: |
+  bytes(state.response).decode_json().as(body,{


Suggested change

bytes(state.response).decode_json().as(body,{

state.response.decode_json().as(body,{

We should not need the conversion since we are now on 8.16 which has mito v1.15.0 and the newer type checker.

Add new line at last in input file. Remove the byte conversion.

mohitjha-elastic · 2024-11-20T11:06:08Z

There are not tests here. Can you remove the comment in the PR that says how to test locally since that obviously has no effect on this addition.

Have you tested? What do you need in order to add CI testing?

Thank you for pointing that out! I've removed the comment from the PR description.

It has been tested on the CrowdStrike Live Instance using the Event Stream API.
I am under the impression that system testing for streaming is not yet supported.

efd6 · 2024-11-20T20:52:24Z

I am under the impression that system testing for streaming is not yet supported.

As far as the input is concerned, chunked and non-chunked connections are the same. It should be possible to simulate the behaviour of the API with stream as it currently exists.

efd6 · 2024-11-20T20:52:29Z

/test

efd6

Looks good, but I'd like to see whether we can have system tests.

elastic-vault-github-plugin-prod · 2024-11-20T21:19:32Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

Add system test.

efd6 · 2024-11-25T22:32:45Z

packages/crowdstrike/data_stream/falcon/_dev/deploy/docker/streaming-mock-service/main.go

Have you tried just using stream?

If we do need to use this, please put callers before callees. Other comments below.

I was encountering issues with using the stream , as it wasn't matching the requests and ended up in a loop. To resolve this, I implemented a different approach.

Is this because stream is not adequately stateful?

efd6 · 2024-11-25T22:33:25Z

packages/crowdstrike/data_stream/falcon/_dev/deploy/docker/streaming-mock-service/main.go

+// Mock OAuth2 client credentials
+var mockClientID = "xxxx"
+var mockClientSecret = "xxxx"
+var datafeedCalled bool = false
+
+// Mock OAuth2 token
+var mockAccessToken = "abcd"
+var mockSessionToken = "xyz"


Let's use flag for these so that they can be controlled in the docker-compos.yml file.

efd6 · 2024-11-25T22:33:53Z

packages/crowdstrike/data_stream/falcon/_dev/deploy/docker/streaming-mock-service/go.mod

@@ -0,0 +1,3 @@
+module streaming-mock-service
+
+go 1.21.3


Let's use a more recent version.

efd6 · 2024-11-25T22:38:58Z

packages/crowdstrike/data_stream/falcon/_dev/deploy/docker/streaming-mock-service/main.go

+	data := []string{
+		`{"metadata":{"customerIDString":"abcabcabc22221","offset":8695284,"eventType":"RemoteResponseSessionStartEvent","eventCreationTime":1698932494000,"version":"1.0"},"event":{"SessionId":"1111-fffff-4bb4-99c1-74c13cfc3e5a","HostnameField":"UKCHUDL00206","UserName":"[email protected]","StartTimestamp":1698932494,"AgentIdString":"fffffffff33333"}}`,
+		`{"metadata":{"customerIDString":"abcabcabc22222","offset":8695285,"eventType":"RemoteResponseSessionStartEvent","eventCreationTime":1698932494000,"version":"1.0"},"event":{"SessionId":"1111-fffff-4bb4-99c1-74c13cfc3e5a","HostnameField":"UKCHUDL00206","UserName":"[email protected]","StartTimestamp":1698932494,"AgentIdString":"fffffffff33333"}}`,
+		`{"metadata":{"customerIDString":"abcabcabc22223","offset":8695286,"eventType":"RemoteResponseSessionStartEvent","eventCreationTime":1698932494000,"version":"1.0"},"event":{"SessionId":"1111-fffff-4bb4-99c1-74c13cfc3e5a","HostnameField":"UKCHUDL00206","UserName":"[email protected]","StartTimestamp":1698932494,"AgentIdString":"fffffffff33333"}}`,
+	}


Either read this from a file with flag-provided path, or use embed to get the text from a text file so that changes in the future are simpler.

1. Update golang version. 2. Use caller before calee. 3. Replace logs with log files.

efd6 · 2024-11-26T23:26:16Z

packages/crowdstrike/changelog.yml

+    - description: Update the minimum kibana version to 8.16.0.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/11773


I don't think we need to include this.

efd6 · 2024-11-26T23:28:41Z

packages/crowdstrike/data_stream/falcon/_dev/deploy/docker/streaming-mock-service/main.go

Please run go fmt.

efd6 · 2024-11-26T23:30:32Z

packages/crowdstrike/data_stream/falcon/_dev/deploy/docker/streaming-mock-service/main.go

+func main() {
+	flag.Parse()
+	// Setup routes
+	http.HandleFunc("/oauth2/token", mockTokenHandler)                // Token endpoint


Suggested change

http.HandleFunc("/oauth2/token", mockTokenHandler) // Token endpoint

http.HandleFunc("POST /oauth2/token", mockTokenHandler)

make use of the METHOD pattern feature of the mux

the code is adequately self-commenting

efd6 · 2024-11-26T23:31:27Z

packages/crowdstrike/data_stream/falcon/_dev/deploy/docker/streaming-mock-service/main.go

+	// Only allow POST method
+	if r.Method != http.MethodPost {
+		http.Error(w, "Invalid request method", http.StatusMethodNotAllowed)
+		return
+	}


Can be dropped when using the METHOD pattern feature of the mux.

efd6 · 2024-11-26T23:33:01Z

packages/crowdstrike/data_stream/falcon/_dev/deploy/docker/streaming-mock-service/main.go

+	chunk := s.data[s.index]
+	s.index++
+	copy(p, chunk)


This is not safe if len(p) < len(chunk).

1. Remove kibana version from changelog. 2. Utilize pattern feature of the MUX. 3. Add len check of chunk safety check.

efd6 · 2024-11-27T07:16:00Z

packages/crowdstrike/data_stream/falcon/_dev/deploy/docker/streaming-mock-service/main.go

+	if len(chunk) > len(p) {
+		p = append(p, make([]byte, len(chunk)-len(p))...)
+	}


You cannot do this.

efd6 · 2024-11-27T07:16:33Z

It's still not clear to me why you can't use stream. Can you show me how you had it configure where it did not work?

mohitjha-elastic · 2024-11-27T08:47:22Z

It's still not clear to me why you can't use stream. Can you show me how you had it configure where it did not work?

I have sent you the full configuration file in DM that I tried earlier to use the stream.

efd6

Full patch for mock changes sent offline.

efd6 · 2024-11-29T01:25:07Z

packages/crowdstrike/data_stream/falcon/_dev/deploy/docker/streaming-mock-service/main.go

+func resourceHandler(w http.ResponseWriter, r *http.Request) {
+	log.Printf("Received request: Method=%s, URL=%s", r.Method, r.URL.String())
+	log.Printf("Headers: %+v", r.Header)
+	if *datafeedCalled {


This is racy in the context of L82.

efd6 · 2024-11-29T01:26:20Z

packages/crowdstrike/data_stream/falcon/agent/stream/streaming.yml.hbs

+    "events": {
+          "message":  body.encode_json(),
+    }


Suggested change

"events": {

"message": body.encode_json(),

}

?"cursor": body.?metadata.optMap(m, {"offset": m.offset}),

"events": [{

"message": body.encode_json(),

}],

Apply the patch file sent by Dan. 1. Update the system test, use stream instead of custom mock. 2. Add offset in curson in data collection.

efd6 · 2024-11-29T06:48:57Z

/test

elasticmachine · 2024-11-29T07:15:25Z

💚 Build Succeeded

Buildkite Build
Commit: a3d39f8

History

💚 Build #18521 succeeded e44c3fd

elastic-sonarqube · 2024-11-29T07:15:31Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

Add support of crowdstrike event stream via streaming.

edfa58a

mohitjha-elastic requested a review from a team as a code owner November 19, 2024 12:27

Update the changelog entry.

2679eec

andrewkroh added Crest enhancement New feature or request Integration:crowdstrike CrowdStrike Team:Security-Service Integrations Security Service Integrations Team [elastic/security-service-integrations] labels Nov 19, 2024

efd6 reviewed Nov 19, 2024

View reviewed changes

Resolve review comments by @efd6.

e44c3fd

Add new line at last in input file. Remove the byte conversion.

mohitjha-elastic requested a review from efd6 November 20, 2024 11:06

efd6 reviewed Nov 20, 2024

View reviewed changes

Resolve review comments by @efd6

4d5507c

Add system test.

mohitjha-elastic requested a review from efd6 November 25, 2024 10:37

efd6 reviewed Nov 25, 2024

View reviewed changes

mohitjha-elastic added 2 commits November 26, 2024 20:05

Resolve review comments by @efd6

018d081

1. Update golang version. 2. Use caller before calee. 3. Replace logs with log files.

Resolved merged conflicts

246fea3

mohitjha-elastic requested a review from efd6 November 26, 2024 14:42

efd6 reviewed Nov 26, 2024

View reviewed changes

Resolve comments by @efd6

f42a1c4

1. Remove kibana version from changelog. 2. Utilize pattern feature of the MUX. 3. Add len check of chunk safety check.

mohitjha-elastic requested a review from efd6 November 27, 2024 05:55

efd6 reviewed Nov 27, 2024

View reviewed changes

efd6 reviewed Nov 29, 2024

View reviewed changes

Resolve review comments by @efd6.

a3d39f8

Apply the patch file sent by Dan. 1. Update the system test, use stream instead of custom mock. 2. Add offset in curson in data collection.

mohitjha-elastic requested a review from efd6 November 29, 2024 06:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CrowdStrike] Add Support of Crowdstrike Event Stream #11773

[CrowdStrike] Add Support of Crowdstrike Event Stream #11773

mohitjha-elastic commented Nov 19, 2024 •

edited

Loading

elasticmachine commented Nov 19, 2024

efd6 left a comment

efd6 Nov 19, 2024

efd6 Nov 19, 2024

mohitjha-elastic commented Nov 20, 2024

efd6 commented Nov 20, 2024

efd6 commented Nov 20, 2024

efd6 left a comment

elastic-vault-github-plugin-prod bot commented Nov 20, 2024

efd6 Nov 25, 2024

mohitjha-elastic Nov 26, 2024

efd6 Nov 26, 2024

efd6 Nov 25, 2024

efd6 Nov 25, 2024

efd6 Nov 25, 2024

efd6 Nov 26, 2024

efd6 Nov 26, 2024

efd6 Nov 26, 2024

efd6 Nov 26, 2024

efd6 Nov 26, 2024

efd6 Nov 27, 2024

efd6 commented Nov 27, 2024

mohitjha-elastic commented Nov 27, 2024

efd6 left a comment

efd6 Nov 29, 2024

efd6 Nov 29, 2024

efd6 commented Nov 29, 2024

elasticmachine commented Nov 29, 2024

elastic-sonarqube bot commented Nov 29, 2024

	bytes(state.response).decode_json().as(body,{
	state.response.decode_json().as(body,{

	http.HandleFunc("/oauth2/token", mockTokenHandler) // Token endpoint
	http.HandleFunc("POST /oauth2/token", mockTokenHandler)

[CrowdStrike] Add Support of Crowdstrike Event Stream #11773

Are you sure you want to change the base?

[CrowdStrike] Add Support of Crowdstrike Event Stream #11773

Conversation

mohitjha-elastic commented Nov 19, 2024 • edited Loading

Proposed Commit Message

Checklist

Screenshots

elasticmachine commented Nov 19, 2024

efd6 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mohitjha-elastic commented Nov 20, 2024

efd6 commented Nov 20, 2024

efd6 commented Nov 20, 2024

efd6 left a comment

Choose a reason for hiding this comment

elastic-vault-github-plugin-prod bot commented Nov 20, 2024

🚀 Benchmarks report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

efd6 commented Nov 27, 2024

mohitjha-elastic commented Nov 27, 2024

efd6 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

efd6 commented Nov 29, 2024

elasticmachine commented Nov 29, 2024

💚 Build Succeeded

History

elastic-sonarqube bot commented Nov 29, 2024

Quality Gate passed

mohitjha-elastic commented Nov 19, 2024 •

edited

Loading