Realm export normalization

[sonOfRa]

What this PR does / why we need it:
When trying to move an existing keycloak installation with no configuration management to keycloak-config-cli, it can be hard to figure out which parts of the realm export are actually needed to reproduce the state when re-importing the configuration.

This PR can serve as a (partial) solution to this problem, by taking a realm export from a certain version of Keycloak, comparing it against a reference export of a "barebones" Keycloak realm from the same version, and then creating a yml (or JSON) file that only contains changes that are actually required.

This PR will not be a complete solution that works for all cases, but it will supply an at least somewhat minimized view of clients and some related settings, which can then easily be merged with a full export in order to get a smaller view of the file. Support for additional parts (identity providers, authentication flows, etc) can be added in later steps to improve this export.

This PR includes a "mode switch" that allows users to switch between IMPORT and NORMALIZE modes using the property run.operation=NORMALIZE. IMPORT is the default, and is used when no option is supplied by the user.

This PR includes an upgrade to Spring Boot 2.7.5, which contains a fix for spring-projects/spring-boot#32559. The @DefaultValue annotation is used to move configuration from the default application.properties file directly to the code. This is needed so that only the needed configurations for the chosen mode (IMPORT or NORMALIZE) are activated.

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #799

Special notes for your reviewer:
A custom properties application-normalize-dev.properties file for testing the export functionality was added. Activating this profile enables normalization mode, and expects an input realm to be present at the top-level exports directory

exports
├── in
│   └── realm.json
└── out
    └── demo1234.yaml

The file in/realm.json will be imported, and the file out/demo1234.yaml is created by the process, where the realm name of the input realm is demo1234

Run the application with the normalize-dev profile active to normalize a given realm.

The configuration parameter normalization.output-format can be used to switch between YAML (the default, if no option is given) and JSON output.

PR Readiness Checklist:

Complete these before marking the PR as ready to review:

• the CHANGELOG.md release notes have been updated to reflect any significant (and particularly user-facing) changes introduced by this PR

[codecov]

Codecov Report

Attention: Patch coverage is 20.45728% with 661 lines in your changes missing coverage. Please review.

Project coverage is 83.08%. Comparing base (dbb8950) to head (93c5fbc).
Report is 219 commits behind head on main.

❗ Current head 93c5fbc differs from pull request most recent head fe9de52

Please upload reports for the commit fe9de52 to get more accurate results.

Files	Patch %	Lines
...ycloak/config/provider/KeycloakExportProvider.java	7.69%	84 Missing ⚠️
...ervice/normalize/AuthFlowNormalizationService.java	6.94%	67 Missing ⚠️
...g/service/normalize/GroupNormalizationService.java	8.33%	66 Missing ⚠️
...ig/service/normalize/RoleNormalizationService.java	7.46%	62 Missing ⚠️
...g/service/normalize/RealmNormalizationService.java	24.28%	53 Missing ⚠️
...ormalize/IdentityProviderNormalizationService.java	8.92%	51 Missing ⚠️
.../service/normalize/ClientNormalizationService.java	12.72%	48 Missing ⚠️
...ce/normalize/ScopeMappingNormalizationService.java	8.51%	43 Missing ⚠️
...ice/normalize/ClientScopeNormalizationService.java	11.36%	39 Missing ⚠️
...loak/config/KeycloakConfigNormalizationRunner.java	20.93%	34 Missing ⚠️
... and 10 more

Additional details and impacted files

@@              Coverage Diff              @@
##               main     #818       +/-   ##
=============================================
- Coverage     95.46%   83.08%   -12.39%     
- Complexity     1302     1339       +37     
=============================================
  Files            76       96       +20     
  Lines          4213     5037      +824     
  Branches        467      621      +154     
=============================================
+ Hits           4022     4185      +163     
- Misses           94      755      +661     
  Partials         97       97               

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[manuschillerdev]

Would it be possible via some kind of diffing to keep params for fields with env var substitutions?

Example:

{
  "realm": "my-realm",
  "smtpServer": {
    "replyToDisplayName": "$(env:SMTP_REPLY_TO_DISPLAYNAME)",
    "port": "$(env:SMTP_PORT)",
    "host": "$(env:SMTP_HOST)",
    "replyTo": "$(env:SMTP_REPLY_TO)",
    "from": "$(env:SMTP_FROM)",
    "fromDisplayName": "$(env:SMTP_FROM_DISPLAY_NAME)",
    "envelopeFrom": "$(env:SMTP_ENVELOPE_FROM)"
  },

When I import this realm, make changes and export the realm again I'd like to keep the params for SMTP instead of having to replace the actual values from the export with the parameters again.

I have no idea what that means regarding the implementation. Just as an idea to make working with the huge JSON-Files easier :)

[sonOfRa]

@manuschillerdev I think that might be a valuable future extension of this feature!

In theory, one could provide a map of "reverse injections" where you specify something like "When the field realm.smtpServer.host has the same value as the resolved environment variable "SMTP_HOST", then replace the value with $(env:SMTP_HOST). This seems to me like a valuable addition to this feature in order to make the process of going from "I have a keycloak instance that has been configured via admin-ui" to "I have a relatively minimal yaml file that represents my config state"

However, I think I'd like to get the base idea of this PR (minimize a realm.json acquired from a full Keycloak realm export) done first, and then build additional features on top of that

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
24 Code Smells

No Coverage information
0.5% Duplication

[francis-pouatcha]

@jonasvoelcker what do you think. Can we look deeper into this feature?

[jonasvoelcker]

Hi @francis-pouatcha,

as mentioned in one of the other merge requests I am currently not available for any work on community things. Our project requires other things and additionally I am about to get a second child in a few days so I can't invest any freetime 😉

Best Regards
Jonas

[sonOfRa]

I currently cannot spare time to work on this PR further; I switched to another employer and we're not using the feature here. But if any of the people trying to use the feature are willing to take over development on this PR, feel free to do so; I place the changes I've made here in the public domain and anyone is free to use them to continue development of this feature with my changes as a base

[srose]

Maybe talk about the topic in Vienna tomorrow?

[thomasdarimont]

I just gave this a spin and aligned the changes from this branch with the latest keycloak-config-cli version in main
https://github.com/thomasdarimont/keycloak-config-cli/tree/feature/realm-normalization
I integrated all commits from this PR with one merge commit with proper config resolutions. I had to do make some changes to adapt the code for the newer Spring Boot version 3.x.
See: https://github.com/adorsys/keycloak-config-cli/compare/main...thomasdarimont:keycloak-config-cli:feature/realm-normalization?expand=1

I did only minimal changes to get it working again. I think this needs some refactoring (config property sharing, use profiles for conditional beans instead of propertys, revise naming, etc.) and I think we could (and should) extend this to support a "realm export" from an "live" realm.

Note that the normalization does not support some features of KC24/25, e.g. organizations etc.

The build works again I could successfully "normalize" a realm export generated from KC 25.0.5.
See: https://gist.github.com/thomasdarimont/7c434fe6dcf20fa28a06cc38e15d6105

[francis-pouatcha]

@thomasdarimont would you send a pull request to sonOfRa:feature/export-diff-yaml or to adorsys:main?

[thomasdarimont]

@francis-pouatcha I rebased this branch and adapted it to the current main. I had to squash most commits to ease the rebase. I added @sonOfRa as coauthor of the new PR.

See: #1207