Use Stream Names for Content Packs

[ryan-carroll-graylog]

Adds ability to create content packs with entities that reference Streams by title, to enable re-upload on another system where a stream with that title exists.

When creating a content pack, if a selected entity references a stream and the specified stream entity is not selected, a new Stream Reference entity type containing the Stream name will be used in the content pack.

When installing the content pack, the Stream Reference will be resolved to an existing Stream on the system with the exported name.

Description

Creates a new Stream reference content pack entity to be used when a stream dependent entity is exported without the stream itself. Also ads a new Stream reference facade class to cover the specific handling of these Stream reference entities during export/import etc.

Implementation notes:
I went with the new stream reference entity approach instead of using stream titles in place of the current stream IDs with versioning, because the latter required refactoring in every content pack entity class that uses streams in order to resolve back and forth between native entity stream IDs and content pack entity stream titles. Additionally this required adding special handling in the top level ContentPackService because of the way we resolve the entity dependency graphs linked by IDs and to omit decency streams but not streams that were selected by the user.

Open Questions:

• If an entity with a Stream Reference dependency can't be resolved to one Stream upon installation, what should the behavior be?
  • Fail installation entirely
  • Omit stream from entity and proceed

TODO:

• Add/update unit tests
• Update Docs

Motivation and Context

Closes: #16885

How Has This Been Tested?

Locally in dev environment.

Testing notes:

• Should be able to create and re-upload Content Packs that contain Stream referencing entities (E.x. Dashboards, Even Definitions), with and without the Streams and have them resolve the stream successfully upon import.
• Should be able to successfully install an Illuminate pack using this new Stream reference entity.
  • To aid in this I've created a version of the Illuminate 4.0.0 bundle that has the "Microsoft Windows Security Spotlight" with Stream scoped Dashboards: https://start.1password.com/open/i?a=UJL6RJPHXJAG5I6GONDPUM4W2I&h=graylog.1password.com&i=zbnncxtxnax3r2hdwpnuyqpnjy&v=yy2feawerg6dps5o2w7ah53wei
  • To fully test this the Streams (and their indices) "Windows Event Log Messages" and "Windows Security Event Log Messages" should be deleted before activating the "Microsoft Windows Security" and "Microsoft Windows Security Spotlight" packs.
  • In all cases the Streams should be successfully associated with the relevant Content Pack entities without creating duplicates.

Screenshots (if appropriate):

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Refactoring (non-breaking change)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes.

[ryan-carroll-graylog]

@kingzacko1 @danotorrey opening this up for initial reviews as the main functionality works (just for content packs with filter/aggregation Event Definitions).

I need to implement this for all stream dependent content pack entities but wanted to get y'all's feedback on the approach. After experimenting with a few options this was the cleanest way I could come up with to get this done.

Need to also verify with the SecCon team that this satisfies their need but based on the Engineering week discussion it seems like it should.

[danotorrey]

@ryan-carroll-graylog Apologies, I did not have a chance to review this today, but I will do that as soon as I am back next week.

[kingzacko1]

Creating a new revision of the content packs with stream_title entities doesn't seem to work. The stream_title entity no longer exists, is not available to be added, and the new revision cannot be installed after download. @ryan-carroll-graylog

[ryan-carroll-graylog]

Creating a new revision of the content packs with stream_title entities doesn't seem to work. The stream_title entity no longer exists, is not available to be added, and the new revision cannot be installed after download. @ryan-carroll-graylog

Thanks for catching this @kingzacko1. I had originally tried to avoid exposing the Stream title entities to the UI for the list of available entities (just to limit possible confusion), but this is clearly a use case where it's needed.

Looks like the stream titles need to be there to be included in what the front end passes back for existing entities to be included in the revision.

With the latest commit the revisions should work correctly. Stream title entities are now included in the available entity list, but otherwise the behavior is the same whether or not a user explicitly includes a dependent stream title.

[danotorrey]

@ryan-carroll-graylog I have been reviewing and testing, and this is looking really solid. This looks like a great design. And, it fits well backwards compatibility-wise, since the export of an entity attached to a stream was not previously possible unless the stream was associated. This is a really nice way to gain that capability.

My testing of importing content packs with the stream reference all were successful.

I hit one snag though: I exported a content pack with a dashboard including selecting a widget's stream to include it in the content pack too (to test the existing supported path of including both). It worked successfully on the first import to another Graylog system, but for some reason, after uninstalling it and attempting to reinstall, I am hitting this error. I've not figured out why that is happening.

graylog2-server/graylog2-server/src/main/java/org/graylog2/contentpacks/model/entities/WidgetEntity.java

Line 162 in ad8564f

throw new ContentPackException("Missing Stream for widget entity");

ERROR: org.graylog2.shared.rest.exceptionmappers.AnyExceptionClassMapper - Unhandled exception in REST resource
org.graylog2.contentpacks.exceptions.ContentPackException: Failed to install content pack <4d08689c-03fa-4a75-9534-0d4573a94815/1>
	at org.graylog2.contentpacks.ContentPackService.installContentPack(ContentPackService.java:161) ~[classes/:?]
	at org.graylog2.contentpacks.ContentPackService.installContentPack(ContentPackService.java:102) ~[classes/:?]
	at org.graylog2.rest.resources.system.contentpacks.ContentPackResource.installContentPack(ContentPackResource.java:295) ~[classes/:?]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	...
Caused by: org.graylog2.contentpacks.exceptions.ContentPackException: Stream with title <Illuminate:Cisco Device Messages> does not exist!
	at org.graylog2.contentpacks.facades.StreamReferenceFacade.findExisting(StreamReferenceFacade.java:153) ~[classes/:?]
	at org.graylog2.contentpacks.facades.StreamReferenceFacade.findExisting(StreamReferenceFacade.java:123) ~[classes/:?]
	at org.graylog2.contentpacks.ContentPackService.installContentPack(ContentPackService.java:134) ~[classes/:?]
	... 29 more

Here's the content pack:
content-pack-ebe297d5-469c-4c5c-a978-0a1ca7d558ce-1.json

I was also testing with installing/uninstalling this content pack that only has the dashboard and the stream reference. Not sure it that somehow impacted the issue.
content-pack-4d08689c-03fa-4a75-9534-0d4573a94815-1.json

[danotorrey]

Great idea to add the default method here. One potential improvement suggestion: Instead of throwing the unsupported exception, could it be implemented using the existing loadAll() method and a title filter? Definitely not as efficient as the Mongo implementation, but perhaps could be used as a backup.

[ryan-carroll-graylog]

Good call! No reason not to have the naive implementation as the default. Just updated the function as suggested.

[ryan-carroll-graylog]

I hit one snag though:

@danotorrey I think I've tracked this down to a separate potential bug in master. I created a separate PR with some more details as I'm thinking it's best to tackle separately but would love your feedback: #17370

[danotorrey]

@danotorrey I think I've tracked this down to a separate potential bug in master. I created a separate PR with some more details as I'm thinking it's best to tackle separately but would love your feedback: #17370

Great work @ryan-carroll-graylog! I will take a look on Monday morn.

[danotorrey]

@ryan-carroll-graylog I tested every combination I could think of, and all is testing successfully with latest code. All LGTM and tests successfully. Very well done working through this! I am sure folks will really enjoy using this new functionality in Illuminate.

[kingzacko1]

Nice work on all of this @ryan-carroll-graylog!

[htolic]

Hi everyone here,
I think this change introduced regression on /api/authz/shares/entities/{entityGRN}/prepare API endpoint. If I click Share button on any event definition at /alerts/definitions the error is: {"type":"ApiError","message":"type <stream_title> does not exist"}

[ryan-carroll-graylog]

Hi everyone here, I think this change introduced regression on /api/authz/shares/entities/{entityGRN}/prepare API endpoint. If I click Share button on any event definition at /alerts/definitions the error is: {"type":"ApiError","message":"type <stream_title> does not exist"}

Thank you @htolic!

I am also able to reproduce this with any Stream scoped Event Definition. I'm investigating further now and will open an issue.

[ryan-carroll-graylog]

Hi everyone here, I think this change introduced regression on /api/authz/shares/entities/{entityGRN}/prepare API endpoint. If I click Share button on any event definition at /alerts/definitions the error is: {"type":"ApiError","message":"type <stream_title> does not exist"}

Thank you @htolic!

I am also able to reproduce this with any Stream scoped Event Definition. I'm investigating further now and will open an issue.

Here's the opened issue: #17882

In the mean time a workaround to share entities with users is:

1. Remove any Streams from entity that needs to be shared
2. Share the entity (and dependent Stream)
3. Add Stream back to the entity