XsData ignores some choice elements

[martinwiesmann]

XsData seems to have a problem in recognising some choice elements.
I think this is a bug in XsData, since PyXB used to pick up on all choice elements.

But maybe there is a workaround to this problem, that I could apply. But I haven't found it yet.

For example, a xsd like this:

    <xs:complexType name="rightSide">
        <xs:annotation>
            <xs:documentation> The right side of a conditionOperator can be
                either a literal value or a data model item (specified by a
                path) </xs:documentation>
        </xs:annotation>
        <xs:choice>
            <xs:annotation>
                <xs:documentation> No right side if operator is unary.
                </xs:documentation>
            </xs:annotation>
            <xs:element maxOccurs="1" minOccurs="0" name="Item" type="item"/>
            <xs:element maxOccurs="1" minOccurs="0" name="Literal" type="xs:string"/>
        </xs:choice>
    </xs:complexType>

produces this class:

@dataclass
class RightSide:
    """
    The right side of a conditionOperator can be either a literal value or a
    data model item (specified by a path)
    """
    class Meta:
        name = "rightSide"

    Item: Optional[Item] = field(
        default=None,
        metadata={
            "type": "Element",
            "namespace": "",
        }
    )
    Literal: Optional[str] = field(
        default=None,
        metadata={
            "type": "Element",
            "namespace": "",
        }
    )

This class definition does not include any hint that only one of these elements are allowed. The correct class should look more like this:

@dataclass
class RightSide:
    """
    The right side of a conditionOperator can be either a literal value or a
    data model item (specified by a path)
    """
    class Meta:
        name = "rightSide"

    choice: Optional[object] = field(
        default_factory=list,
        metadata={
            "type": "Elements",
            "choices": (
                {
                    "name": "Item",
                    "type": Item,
                    "namespace": "",
                },
                {
                    "name": "Literal",
                    "type": str,
                    "namespace": "",
                },
    )

This xsd however produces a correct dataclass

    <xs:complexType name="genericKVParam">
        <xs:annotation>
            <xs:documentation> Implementation of a generic Key/Value parameter.
            </xs:documentation>
        </xs:annotation>
        <xs:sequence>
            <xs:element name="Key" type="xs:string"/>
            <xs:element name="Description" type="xs:string" minOccurs="0"/>
            <xs:choice>
                <xs:element name="StringValue" type="xs:string"/>
                <xs:element name="BoolValue" type="xs:boolean"/>
                <xs:element name="IntValue" type="xs:int"/>
                <xs:element name="LongValue" type="xs:long"/>
                <xs:element name="DoubleValue" type="xs:double"/>
                <xs:element name="Files" type="dss:dataContainer" minOccurs="1"
                    maxOccurs="unbounded"/>
            </xs:choice>
            <!--			Unit is voluntarily not taken from bas/utd because it would break
            the flexibility of this generic KV parameters implementation -->
            <xs:element name="Unit" type="xs:string" minOccurs="0"/>
        </xs:sequence>
    </xs:complexType>

produces this class:

@dataclass
class GenericKVParam:
    """
    Implementation of a generic Key/Value parameter.
    """
    class Meta:
        name = "genericKVParam"

    Key: Optional[str] = field(
        default=None,
        metadata={
            "type": "Element",
            "namespace": "",
            "required": True,
        }
    )
    Description: Optional[str] = field(
        default=None,
        metadata={
            "type": "Element",
            "namespace": "",
        }
    )
    choice: List[object] = field(
        default_factory=list,
        metadata={
            "type": "Elements",
            "choices": (
                {
                    "name": "StringValue",
                    "type": str,
                    "namespace": "",
                },
                {
                    "name": "BoolValue",
                    "type": bool,
                    "namespace": "",
                },
                {
                    "name": "IntValue",
                    "type": int,
                    "namespace": "",
                },
                {
                    "name": "LongValue",
                    "type": int,
                    "namespace": "",
                },
                {
                    "name": "DoubleValue",
                    "type": float,
                    "namespace": "",
                },
                {
                    "name": "Files",
                    "type": DataContainer,
                    "namespace": "",
                },
            ),
        }
    )
    Unit: Optional[str] = field(
        default=None,
        metadata={
            "type": "Element",
            "namespace": "",
        }
    )

which is correct.

Another example that fails:

Xsd:

    <xs:complexType name="functions">
        <xs:choice>
            <xs:element name="Statistics" type="statisticFunctions"/>
            <xs:element name="Transforms" type="transformFunctions"/>
            <xs:element name="Mathematics" type="mathematicFunctions"/>
            <xs:element name="ReadFits" type="astroPyPath"/>
        </xs:choice>
    </xs:complexType>

produces this class:

@dataclass
class Functions:
    class Meta:
        name = "functions"

    Statistics: Optional[StatisticFunctions] = field(
        default=None,
        metadata={
            "type": "Element",
            "namespace": "",
        }
    )
    Transforms: Optional[TransformFunctions] = field(
        default=None,
        metadata={
            "type": "Element",
            "namespace": "",
        }
    )
    Mathematics: Optional[MathematicFunctions] = field(
        default=None,
        metadata={
            "type": "Element",
            "namespace": "",
        }
    )
    ReadFits: Optional[str] = field(
        default=None,
        metadata={
            "type": "Element",
            "namespace": "",
            "pattern": r"hdul(\[.+\]).(data|header)(\[([\w-]+|[:\d,]+)\])+",
        }
    )

which does not include the choice part.

[martinwiesmann]

I did some digging, and my results are probably not the final results. My findings are that at least one of the elements within the xs:choice element must have maxOccurs="unbounded". Otherwise the choice element is ignored.

The other problem I realised is that the 'choices' tag in the metadata of a dataclass does not have any of the options indicated there. E.g. the maxOccurs="unbounded" is not obvious, and if I include a minOccurs, that won't appear there neither. Possibly other options are also omitted.

[martinwiesmann]

Examples:

    <xs:complexType name="functions">
        <xs:choice>
            <xs:element name="Statistics" type="statisticFunctions"/>
            <xs:element name="Transforms" type="transformFunctions"/>
            <xs:element name="Mathematics" type="mathematicFunctions"/>
            <xs:element name="ReadFits" type="astroPyPath" minOccurs="1"
                maxOccurs="unbounded"/>
        </xs:choice>
    </xs:complexType>
    
    
    <xs:complexType name="functions2">
        <xs:choice>
            <xs:element name="Statistics" type="statisticFunctions"/>
            <xs:element name="Transforms" type="transformFunctions"/>
            <xs:element name="Mathematics" type="mathematicFunctions"/>
            <xs:element name="ReadFits" type="astroPyPath" maxOccurs="1"/>
        </xs:choice>
    </xs:complexType>
    
    
    <xs:complexType name="functions3">
        <xs:choice>
            <xs:element name="Statistics" type="statisticFunctions"/>
            <xs:element name="Transforms" type="transformFunctions"/>
            <xs:element name="Mathematics" type="mathematicFunctions"/>
            <xs:element name="ReadFits" type="astroPyPath" maxOccurs="unbounded"/>
        </xs:choice>
    </xs:complexType>
    
    
    <xs:complexType name="functions4">
        <xs:choice>
            <xs:element name="Statistics" type="statisticFunctions" minOccurs="1"
                maxOccurs="unbounded"/>
            <xs:element name="Transforms" type="transformFunctions"/>
            <xs:element name="Mathematics" type="mathematicFunctions"/>
            <xs:element name="ReadFits" type="astroPyPath"/>
        </xs:choice>
    </xs:complexType>

Results:

@dataclass
class Functions:
    class Meta:
        name = "functions"

    choice: List[object] = field(
        default_factory=list,
        metadata={
            "type": "Elements",
            "choices": (
                {
                    "name": "Statistics",
                    "type": StatisticFunctions,
                    "namespace": "",
                },
                {
                    "name": "Transforms",
                    "type": TransformFunctions,
                    "namespace": "",
                },
                {
                    "name": "Mathematics",
                    "type": MathematicFunctions,
                    "namespace": "",
                },
                {
                    "name": "ReadFits",
                    "type": str,
                    "namespace": "",
                    "pattern": r"hdul(\[.+\]).(data|header)(\[([\w-]+|[:\d,]+)\])+",
                },
            ),
        }
    )


@dataclass
class Functions2:
    class Meta:
        name = "functions2"

    Statistics: Optional[StatisticFunctions] = field(
        default=None,
        metadata={
            "type": "Element",
            "namespace": "",
        }
    )
    Transforms: Optional[TransformFunctions] = field(
        default=None,
        metadata={
            "type": "Element",
            "namespace": "",
        }
    )
    Mathematics: Optional[MathematicFunctions] = field(
        default=None,
        metadata={
            "type": "Element",
            "namespace": "",
        }
    )
    ReadFits: Optional[str] = field(
        default=None,
        metadata={
            "type": "Element",
            "namespace": "",
            "pattern": r"hdul(\[.+\]).(data|header)(\[([\w-]+|[:\d,]+)\])+",
        }
    )


@dataclass
class Functions3:
    class Meta:
        name = "functions3"

    choice: List[object] = field(
        default_factory=list,
        metadata={
            "type": "Elements",
            "choices": (
                {
                    "name": "Statistics",
                    "type": StatisticFunctions,
                    "namespace": "",
                },
                {
                    "name": "Transforms",
                    "type": TransformFunctions,
                    "namespace": "",
                },
                {
                    "name": "Mathematics",
                    "type": MathematicFunctions,
                    "namespace": "",
                },
                {
                    "name": "ReadFits",
                    "type": str,
                    "namespace": "",
                    "pattern": r"hdul(\[.+\]).(data|header)(\[([\w-]+|[:\d,]+)\])+",
                },
            ),
        }
    )


@dataclass
class Functions4:
    class Meta:
        name = "functions4"

    choice: List[object] = field(
        default_factory=list,
        metadata={
            "type": "Elements",
            "choices": (
                {
                    "name": "Statistics",
                    "type": StatisticFunctions,
                    "namespace": "",
                },
                {
                    "name": "Transforms",
                    "type": TransformFunctions,
                    "namespace": "",
                },
                {
                    "name": "Mathematics",
                    "type": MathematicFunctions,
                    "namespace": "",
                },
                {
                    "name": "ReadFits",
                    "type": str,
                    "namespace": "",
                    "pattern": r"hdul(\[.+\]).(data|header)(\[([\w-]+|[:\d,]+)\])+",
                },
            ),
        }
    )

[tefra]

I can't remember why this condition was added to be honest, it looks like a remnant from the very first compound fields implementation.

https://github.com/tefra/xsdata/blob/master/xsdata/codegen/handlers/create_compound_fields.py#L32

I 'll check again and remove it. Thank you @martinwiesmann for your awesome analysis

[martinwiesmann]

Great, thanks. Great job in finding the problematic line so quickly.

Unfortunately, there remains another problem: If for some reason there is actually a 'max_occurs=unbounded' as a restriction to a choice-element (which is the case in our project), then this is not reflected in the python dataclass, see above class 'GenericKVParam' choice-element 'Files'. I haven't checked any other restrictions (e.g. minInclusive etc), but it would be nice to have this information also in the dataclass.

Thanks for your great work.

[martinwiesmann]

It seems that most restrictions are in the dataclass, but minOccurs and maxOccurs are not. See the example:

    <xs:simpleType name="astroPyPath">
        <xs:restriction base="xs:string">
            <xs:pattern value="hdul(\[.+\]).(data|header)(\[([\w-]+|[:\d,]+)\])+"/>
        </xs:restriction>
    </xs:simpleType>

    <xs:simpleType name="someNumber">
        <xs:restriction base="xs:float">
            <xs:minExclusive value="10"/>
            <xs:maxInclusive value="100"/>
        </xs:restriction>
    </xs:simpleType>

    <xs:complexType name="functions">
        <xs:choice>
            <xs:element name="Statistics" type="xs:string" minOccurs="4" maxOccurs="12"/>
            <xs:element name="Transforms" type="someNumber"/>
            <xs:element name="Mathematics" type="mathematicFunctions"/>
            <xs:element name="ReadFits" type="astroPyPath" minOccurs="1" maxOccurs="unbounded"/>
        </xs:choice>
    </xs:complexType>

becomes:

@dataclass
class Functions:
    class Meta:
        name = "functions"

    choice: List[object] = field(
        default_factory=list,
        metadata={
            "type": "Elements",
            "choices": (
                {
                    "name": "Statistics",
                    "type": str,
                    "namespace": "",
                },
                {
                    "name": "Transforms",
                    "type": float,
                    "namespace": "",
                    "min_exclusive": 10.0,
                    "max_inclusive": 100.0,
                },
                {
                    "name": "Mathematics",
                    "type": MathematicFunctions,
                    "namespace": "",
                },
                {
                    "name": "ReadFits",
                    "type": str,
                    "namespace": "",
                    "pattern": r"hdul(\[.+\]).(data|header)(\[([\w-]+|[:\d,]+)\])+",
                },
            ),
        }
    )

[tefra]

Hi @martinwiesmann the fix for the first issue is on master, give it a try and let me know how it works for you.

This one was tough to crack but I am happy that you brought this up, it involved fixing some early patches early on to make things to just work. Thank you for reporting!

I split the second one here #748.

[loopj]

I just tested this with my schema and it works great. <xs:choice maxOccurs="1"> now generates choices correctly in my models. Thanks!

[martinwiesmann]

Thank you @tefra
I am unfortunately swamped with other projects, so that I couldn't properly test the updated version. But it looks good so far.