Refactor ESLogMessage to not define fields upfront #46702

pgomulka · 2019-09-13T12:06:53Z

The current behaviour expect all additional fields to be defined in ESLogMessage. This makes adding new custom ESLogMessages harder.
This refactoring is changing this behaviour to only define fields in a properties file when overriding a standard field for instance message. All other custom fields do not need to be defined upfront in property file.
As an example an AllocationService log statement Cluster health status changed from [{}] to [{}] (reason: [{}] will have additional 3 fields in json logs.

{
  "type": "server",
  "timestamp": "2019-09-27T13:22:19,660+02:00",
  "level": "INFO",
  "component": "o.e.c.r.a.AllocationService",
  "cluster.name": "distribution_run",
  "node.name": "node-0",
  "message": "Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[twitter45][1]]]).",
  "cluster.uuid": "1WUx41iLT06b2VXRAgm3nw",
  "node.id": "4lYUM1SDSx2HwHFKB_SJWQ",
  "reason": "shards started [[twitter45][1]]",
  "current.health": "YELLOW",
  "previous.health": "RED"
}

relates #46119

elasticmachine · 2019-09-13T12:06:55Z

Pinging @elastic/es-core-infra

pgomulka · 2019-09-13T13:45:58Z

@pgomulka test

pgomulka · 2019-09-27T12:16:16Z

@elasticmachine update branch

…asticsearch into feature/logging-parametrised

alpar-t · 2019-09-30T13:00:04Z

server/src/main/java/org/elasticsearch/common/logging/ESLogMessage.java

@@ -72,4 +81,9 @@ public static String asJsonArray(Stream<String> stream) {
            .map(ESLogMessage::inQuotes)
            .collect(Collectors.joining(", ")) + "]";
    }
+
+    public static ESLogMessage message(String messagePattern, Object... args){


Since we eventually collect the arguments using .with( we could probably structure this in a way that avoids repeating them. It's probably worth it even if we end up using a builder.
I'm thinking something similar to: https://github.com/logstash/logstash-logback-encoder#event-specific-custom-fields

alpar-t · 2019-09-30T13:02:02Z

server/src/main/java/org/elasticsearch/common/logging/ESLogMessage.java

@@ -72,4 +81,9 @@ public static String asJsonArray(Stream<String> stream) {
            .map(ESLogMessage::inQuotes)
            .collect(Collectors.joining(", ")) + "]";
    }
+
+    public static ESLogMessage message(String messagePattern, Object... args){


We have tooling to check that loggers are used correctly which I think is now circumvent-able with this method.
Most probably the tooling will have to be updated regardless of how this API is implemented.

you probably mean ESLoggerUsageTests ? I think it should be fine as it was already extended for subclass usages. But I will have a look to make sure it passes fine

Yes I do mean the ESLoggerUsageChecker I don't know enough about it's implementation to be able to tell if it will work with the builders or not.

@pgomulka Could we maybe add an explicit test to ESLoggerUsageChecker that uses ESLogMessage.of just to be clear that this works ?

qa/logging-config/src/test/java/org/elasticsearch/common/logging/JsonLoggerTests.java

pgomulka · 2019-10-07T07:34:19Z

@elasticmachine update branch

pgomulka · 2019-10-07T10:40:53Z

@elasticmachine run elasticsearch-ci/2

server/src/main/java/org/elasticsearch/common/logging/DeprecatedMessage.java

server/src/main/java/org/elasticsearch/common/logging/ESLogMessage.java

alpar-t · 2019-10-07T13:23:54Z

server/src/main/java/org/elasticsearch/common/logging/ESLogMessage.java

+            sb.append(Chars.DQUOTE).append(':').append(Chars.DQUOTE);
+            start = sb.length();
+            sb.append(getIndexedReadOnlyStringMap().getValueAt(i).toString());
+//            ParameterFormatter.recursiveDeepToString(getIndexedReadOnlyStringMap().getValueAt(i), sb, null);


should we remove this commented code ?

…asticsearch into feature/logging-parametrised

pgomulka · 2019-10-08T12:08:55Z

@elasticmachine run elasticsearch-ci/2

pgomulka · 2019-10-08T12:10:46Z

@elasticmachine run elasticsearch-ci/packaging-sample-matrix

alpar-t

LGTM

pgomulka · 2019-10-09T14:14:05Z

@elasticmachine run elasticsearch-ci/2

pgomulka · 2019-10-09T14:14:12Z

@elasticmachine run elasticsearch-ci/packaging-sample

Slow log's routing value when null was causing ESLogMessage.asJson method to throw Null Pointer Exception. This should be fixed in ESLogMessage as well as prevent passing that key at all. This only happens in 8 because of previous refactoring #46702

refactor ESLogMessage and overriding fields

cb24964

pgomulka added the :Core/Infra/Logging Log management and logging utilities label Sep 13, 2019

pgomulka self-assigned this Sep 13, 2019

pgomulka changed the title ~~Refactor ESLogMessage to not defining fields upfront~~ WIP Refactor ESLogMessage to not defining fields upfront Sep 13, 2019

pgomulka added the WIP label Sep 13, 2019

pgomulka added 2 commits September 13, 2019 15:16

rename esmessagefields to overrideFields

6b450a9

javadoc

b23ffb1

pgomulka changed the title ~~WIP Refactor ESLogMessage to not defining fields upfront~~ Refactor ESLogMessage to not defining fields upfront Sep 13, 2019

pgomulka changed the title ~~Refactor ESLogMessage to not defining fields upfront~~ Refactor ESLogMessage to not define fields upfront Sep 13, 2019

pgomulka removed the WIP label Sep 13, 2019

pgomulka added 2 commits September 27, 2019 13:29

simplify custom message class creation

5605955

line length

74cb563

elasticmachine and others added 3 commits September 27, 2019 05:16

Merge branch 'master' into feature/logging-parametrised

ad55b7b

small cleanup and rename of ESJsonField

ba1bf6f

Merge branch 'feature/logging-parametrised' of github.com:pgomulka/el…

d9bfd89

…asticsearch into feature/logging-parametrised

alpar-t reviewed Sep 30, 2019

View reviewed changes

pgomulka added the v8.0.0 label Sep 30, 2019

pgomulka added 2 commits October 2, 2019 15:04

working with builder

bf0d670

parametrized map - basing on ecs

ae1bc99

pgomulka commented Oct 2, 2019

View reviewed changes

qa/logging-config/src/test/java/org/elasticsearch/common/logging/JsonLoggerTests.java Show resolved Hide resolved

pgomulka added 3 commits October 4, 2019 14:42

builder

9c38db3

refactoring and cleanup

9f307c8

cleanup

2025181

Merge branch 'master' into feature/logging-parametrised

747c220

pgomulka requested a review from alpar-t October 7, 2019 10:43

alpar-t reviewed Oct 7, 2019

View reviewed changes

server/src/main/java/org/elasticsearch/common/logging/DeprecatedMessage.java Show resolved Hide resolved

alpar-t reviewed Oct 7, 2019

View reviewed changes

server/src/main/java/org/elasticsearch/common/logging/ESLogMessage.java Show resolved Hide resolved

alpar-t reviewed Oct 7, 2019

View reviewed changes

pgomulka added 3 commits October 8, 2019 12:06

logger checker

40cd0ef

Merge branch 'feature/logging-parametrised' of github.com:pgomulka/el…

7bf85c8

…asticsearch into feature/logging-parametrised

Merge branch 'master' into feature/logging-parametrised

c4d7cd4

alpar-t approved these changes Oct 9, 2019

View reviewed changes

pgomulka merged commit a4a7967 into elastic:master Oct 10, 2019

pgomulka mentioned this pull request Nov 14, 2019

Logging improvements #49087

Closed

14 tasks

pgomulka mentioned this pull request Mar 18, 2020

Fix NPE when logging null values in JSON #53715

Merged

jakelandis added v8.0.0-alpha1 >refactoring and removed v8.0.0 labels Jul 26, 2021

ChrisHegarty unassigned pgomulka Oct 29, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor ESLogMessage to not define fields upfront #46702

Refactor ESLogMessage to not define fields upfront #46702

pgomulka commented Sep 13, 2019 •

edited

Loading

elasticmachine commented Sep 13, 2019

pgomulka commented Sep 13, 2019

pgomulka commented Sep 27, 2019

alpar-t Sep 30, 2019

alpar-t Sep 30, 2019

pgomulka Oct 3, 2019

alpar-t Oct 3, 2019

alpar-t Oct 7, 2019

pgomulka commented Oct 7, 2019

pgomulka commented Oct 7, 2019

alpar-t Oct 7, 2019

pgomulka commented Oct 8, 2019

pgomulka commented Oct 8, 2019

alpar-t left a comment

pgomulka commented Oct 9, 2019

pgomulka commented Oct 9, 2019

Refactor ESLogMessage to not define fields upfront #46702

Refactor ESLogMessage to not define fields upfront #46702

Conversation

pgomulka commented Sep 13, 2019 • edited Loading

elasticmachine commented Sep 13, 2019

pgomulka commented Sep 13, 2019

pgomulka commented Sep 27, 2019

alpar-t Sep 30, 2019

Choose a reason for hiding this comment

alpar-t Sep 30, 2019

Choose a reason for hiding this comment

pgomulka Oct 3, 2019

Choose a reason for hiding this comment

alpar-t Oct 3, 2019

Choose a reason for hiding this comment

alpar-t Oct 7, 2019

Choose a reason for hiding this comment

pgomulka commented Oct 7, 2019

pgomulka commented Oct 7, 2019

alpar-t Oct 7, 2019

Choose a reason for hiding this comment

pgomulka commented Oct 8, 2019

pgomulka commented Oct 8, 2019

alpar-t left a comment

Choose a reason for hiding this comment

pgomulka commented Oct 9, 2019

pgomulka commented Oct 9, 2019

pgomulka commented Sep 13, 2019 •

edited

Loading