Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Normalized Export #799

Open
mlhartme opened this issue Oct 14, 2022 · 6 comments · May be fixed by #1207
Open

Normalized Export #799

mlhartme opened this issue Oct 14, 2022 · 6 comments · May be fixed by #1207

Comments

@mlhartme
Copy link

mlhartme commented Oct 14, 2022

Problem Statement

(I'm not sure if this feature request is best placed here or in keycloak itself ... I'll first try here ...)

We use keycloak-config-cli to properly define the configuration of our Keycloak server in my-realm.json. That's great!

But we use manual configuration as well, usually because we need try and error to find the proper configuration.
To get manual changes into my-realm.json (and thus get a well-defined state again), we diff keycloaks export (kcadm.sh create realms/my-realm/partial-export) before and after the manual change.

Problem with that is: the diff gets big, mostly because ordering in some of keycloaks arrays doesn't matter, and keycloak returns entries in different order. And because we regularly delete and create the realm from my-realm.json to play save - which causes Keycloak to choose new id s.

Proposed Solution

Is it possible to provide normalized realm exports that:

  • sorts arrays where ordering doesn't matter (e.g. protocolMappers and org.keycloak.keys.KeyProvider)
  • normalize id s by replacing the uuids with a numbers: first with 1, then 2, etc

We currently use jq to manually do some normalization (see below). That helps us a lot, but it feels like a poor solution, especially replacing all id s with XXX ...

Best regards, Michael

#!/bin/sh
if [ "$#" -ne 1 ] ; then
echo "Normalize arrays and ids"
echo "usage: normalize.sh <src.json>"
exit 1
fi
set -e

json=$(cat "$1")
json=$(echo "${json}" | jq '.clientScopes = (.clientScopes | sort_by(.name))')
json=$(echo "${json}" | jq '(.. | select(.protocolMappers?) | .protocolMappers) |= sort_by(.name)')
json=$(echo "${json}" | jq '(.components | .["org.keycloak.keys.KeyProvider"]) |= sort_by(.name)')
json=$(echo "${json}" | jq '(.components | .["org.keycloak.storage.UserStorageProvider"] | .[] | .subComponents | .["org.keycloak.storage.ldap.mappers.LDAPStorageMapper"]) |= sort_by(.name)')
json=$(echo "${json}" | jq '(.components | .["org.keycloak.services.clientregistration.policy.ClientRegistrationPolicy"]) |= sort_by(.name, .subType)')
json=$(echo "${json}" | jq '(.. | select(.["allowed-protocol-mapper-types"]?) | .["allowed-protocol-mapper-types"] ) |= sort')
json=$(echo "${json}" | jq '(.. | select(.id?) | .id) |= "XXX" ')
echo "${json}" | jq --sort-keys

Environment

  • Keycloak Version: 19.0.3
  • keycloak-config-cli Version: 5.3.1
  • Java Version: 17
@sonOfRa
Copy link

sonOfRa commented Nov 24, 2022

I've started a draft implementation of this feature over at #818.

The general idea is to use Javers to diff the objects, comparing them to a "baseline" realm export for any given Keycloak version, and then using that diff to produce a normalized/minimized yaml file that can later be imported using keycloak-config-cli.

Any inputs on particular choices or ideas on how to do it better are much appreciated!

@katoquro
Copy link

katoquro commented Mar 7, 2023

Hello!
@sonOfRa, how is it going with normalization? I have a similar requirement: show diff before deployment of the new configuration. I've started but I've found this issue and think that it would be better to wait for your implementation and then tune it for my case.

@sonOfRa
Copy link

sonOfRa commented Mar 8, 2023

Hi!

I'm making progress, but it's relatively slow going. There's a few edge cases that pop up every now and then when trying the current state out with different realm configurations. Most of the important features of the keycloak config are already present in the normalized config now, but one big block that is still missing is the "components": [] block, which is a quite complex type that isn't very constrained, and as such a bit difficult to "normalize" without accidentally breaking things.

@katoquro
Copy link

katoquro commented Mar 8, 2023

I see. components are like Map<String, Object> 😄. Currently, I think that we can just make a deep ordering for the keys of that maps and it may cover most cases. At least, it will be reproducible.

@SebastianKunz
Copy link

I am also interested in this feature. @sonOfRa can you elaborate what the state of progress is? Are there any blockers for you?

@dominiktopp
Copy link

dominiktopp commented Oct 12, 2023

We had a similar problem. Our solution was to replace ids (search by regex, replace with XXX) and use a JSON diff tool like https://www.jsondiff.com/. That reduces the problem to the order of elements in arrays but this is not very hard to check by hand.
After a few roundtrips we had enough experience and examples in our realm.json to do most of the configuration directly in the realm.json.

But for the migration from "configuration by admin ui" to keycloak-config-cli a tool like #818 would have been great 😄

thomasdarimont added a commit to thomasdarimont/keycloak-config-cli that referenced this issue Nov 22, 2024
This is a squash of the commits from multiple repositories:
Original PR by sonOfRa: adorsys#818
Adaptations by thomasdarimont: https://github.com/thomasdarimont/keycloak-config-cli/tree/feature/realm-normalization

Fixes adorsys#799

Co-authored-by: Simon Levermann <[email protected]>
Signed-off-by: Thomas Darimont <[email protected]>
thomasdarimont added a commit to thomasdarimont/keycloak-config-cli that referenced this issue Nov 22, 2024
This is a squash of the commits from multiple repositories:
Original PR by sonOfRa: adorsys#818
Adaptations by thomasdarimont: https://github.com/thomasdarimont/keycloak-config-cli/tree/feature/realm-normalization

Fixes adorsys#799

Co-authored-by: Simon Levermann <[email protected]>
Signed-off-by: Thomas Darimont <[email protected]>
@thomasdarimont thomasdarimont linked a pull request Nov 22, 2024 that will close this issue
1 task
@Motouom Motouom moved this from Todo to Review in os-competence-center-board Nov 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
5 participants