Adds log4j-docgen module

[ppkarwasz]

Adds a log4j-docgen module whose purpose is to automatically generate the documentation of Log4j components based on annotations in the source code and Javadoc comments.

This initial version:

• defines a /META-INF/log4j/plugins.xml plugin descriptor, which is similar in nature to Maven's /META-INF/maven/plugin.xml and uses the same model generator (Modello),
• provides a SchemaGenerator that produces a single XML schema from a collection of plugins.xml files.

apache/logging-log4j2#1955

[vy]

@ppkarwasz, I find it very difficult to understand what this PR does exactly and, maybe more importantly, what it is supposed to deliver. Could you provide some review kit, please? In particular, questions I have:

• How is this tool supposed to be used/run?
• What are its inputs and outputs?
• How does it all work? Modello, SchemaGenerator, configuration.xml, plugins.xml, etc.

I guess, since you are deep in the problem, these might sound obvious to you. But I can assure you that the rest of us have no clue.

[ppkarwasz]

@vy,

Here are some suggestions on how to review this PR.

How to review this PR

This PR contains multiple contents that might require review:

1. The contents of a /META-INF/log4j/plugins.xml file are transformed into Java classes using Modello. These classes are described in a file that is also called plugins.xml: src/main/mdo/plugins.xml

   The model should be minimal and the attributes should be rationally named (naming is not my strong suite). An example of a /META-INF/log4j/plugins.xml file is given here: test/resources/META-INF/log4j/plugins.xml

2. From the model I generate an XSD. The SchemaGenerator accepts multiple plugins.xml files to be merged together. There is actually always an internal file that is added: src/main/resources/org/apache/logging/log4j/docgen/internal/configuration.xml

   The generated XSD (cf. src/test/resources/xsd/log4j.xsd) should be powerful enough to easily write a configuration file. Due to the usage of <group> each plugin element contains a <sequence> of sub-elements, i.e. order matter. If you have a solution that accepts elements in any order let me know (all can not contain a group).

3. There is no CLI interface in this module. I'll wrap it up in a Maven plugin when everything is ready (the transformation from plugins.xml to XSD and Asciidoc and generation of the plugins.xml file itself). SchemaGeneratorTest provides an example how all the elements will be bound together.

[vy]

I have some minor comments, but the overall looks good to me. Great job! 💯

Sanity check

To check if I get the wiring right, I share my notes below.

1. A to-be-implemented PluginScanner generates plugins.xml (similar to the one in test/resources/META-INF/log4j/plugins.xml) from log4j-core.jar, log4j-layout-template-json.jar, etc.
2. Modello, provided with src/main/mdo/plugins.xml, generates DTOs and an XML reader/writer.
3. We use Modello-generated DTOs and the XML reader to read plugins.xml.
4. Using DTOs populated from the XML, we manually generate XSD ourselves.

Correct?

plugins.xml file name confusion

There are so many plugins.xml and it requires quite some stretch to wrap my mind around them. I would appreciate it if we can use more self-explanatory file names:

• src/main/mdo/plugins.xml ⇒ src/main/mdo/plugins-model.xml
• src/test/resources/META-INF/log4j/plugins.xml ⇒ sample-plugins.xml
• src/test/resources/xsd/log4j.xsd ⇒ sample-plugins.xsd
• src/main/resources/.../configuration.xml ⇒ src/.../supplement-plugins.xml

src/site

src/site is hand-crafted as a showcase, right?

XML groups

I think the problem you're describing is as follows:

  <complexType name="org.apache.logging.log4j.core.config.LoggerConfig">
    <sequence>
      <group ref="log4j:org.apache.logging.log4j.core.config.AppenderRef.group" minOccurs="0" maxOccurs="unbounded"/>
      <group ref="log4j:org.apache.logging.log4j.core.Filter.group" minOccurs="0"/>
    </sequence>
    <attribute name="name" type="string"/>
  </complexType>

Naively thinking, wouldn't omitting the sequence work? That is,

  <complexType name="org.apache.logging.log4j.core.config.LoggerConfig">
    <group ref="log4j:org.apache.logging.log4j.core.config.AppenderRef.group" minOccurs="0" maxOccurs="unbounded"/>
    <group ref="log4j:org.apache.logging.log4j.core.Filter.group" minOccurs="0"/>
    <attribute name="name" type="string"/>
  </complexType>

[vy]

Attribute-vs-element distinction feels to me an implementation detail. That is, from the pov of the user, they are all configuration properties of the plugin. As a matter of fact, JSON-, YAML-, and .properties-formatted configurations cannot even accommodate the attribute concept, for them, every property is an element. Do we really need two different types, i.e., attribute & element? Can't we use a single property/attribute/element/etc. to refer the both?

[ppkarwasz]

Good point,

From a JSON and YAML perspective the distinction is not as evident ("elements" are properties with an object value), but I would keep the distinction for the user: "attributes" are injected by name, whereas "elements" are injected by type. Knowing that simplifies configuration considerably.

I will look at this again to see if I can simplify things.

[ppkarwasz]

Hi @vy,

To check if I get the wiring right, I share my notes below.

1. A to-be-implemented `PluginScanner` generates `plugins.xml` (similar to the one in `test/resources/META-INF/log4j/plugins.xml`) from `log4j-core.jar`, `log4j-layout-template-json.jar`, etc.

2. Modello, provided with `src/main/mdo/plugins.xml`, generates DTOs and an XML reader/writer.

3. We use Modello-generated DTOs and the XML reader to read `plugins.xml`.

4. Using DTOs populated from the XML, we manually generate XSD ourselves.

Correct?

Exactly

plugins.xml file name confusion

I corrected it, thanks.

src/site

src/site is hand-crafted as a showcase, right?

Yes, I will use it later to test the generation of AsciiDoc. The idea is to know the rough structure of the documentation, so that we know what elements we need.

In the showcase documentation attributes and elements are in separate sections for example, so I would like to distinguish the two types.

XML groups

I think the problem you're describing is as follows:

  <complexType name="org.apache.logging.log4j.core.config.LoggerConfig">
    <sequence>
      <group ref="log4j:org.apache.logging.log4j.core.config.AppenderRef.group" minOccurs="0" maxOccurs="unbounded"/>
      <group ref="log4j:org.apache.logging.log4j.core.Filter.group" minOccurs="0"/>
    </sequence>
    <attribute name="name" type="string"/>
  </complexType>

Naively thinking, wouldn't omitting the sequence work? That is,

  <complexType name="org.apache.logging.log4j.core.config.LoggerConfig">
    <group ref="log4j:org.apache.logging.log4j.core.config.AppenderRef.group" minOccurs="0" maxOccurs="unbounded"/>
    <group ref="log4j:org.apache.logging.log4j.core.Filter.group" minOccurs="0"/>
    <attribute name="name" type="string"/>
  </complexType>

Unfortunately the XMLSchema XSD does not allow that. I think we'll need to decide whether writing elements in an arbitrary order is important to users. We can assume for now it isn't and improve the schema later.

[ppkarwasz]

I am closing this, since the documentation generator needs some refactoring to account for the generation of human documentation. This will be resubmitted in three more digestible parts:

• the common structure and metadata model c1d23ab
• Add an XML schema generator #92
• Add FreeMarker documentation generator #93