Preserve yaml tags (e.g. to support CloudFormation templates)

[jdojff]

Hey, sorry if I missed similar issue or a solution for this, but I failed to find such.

In your web (https://get-ytt.io) it's written: Template any YAML content, including, <...> AWS CloudFormation configurations, <...>. But it seems you can't.

Please see some random template and resulting yaml. Is there anything I could do not to change the original template syntax (with !Ref, !If, etc)?

Input:

Resources:
  Ec2Instance:
    Type: 'AWS::EC2::Instance'
    Properties:
      SecurityGroups:
        - !Ref InstanceSecurityGroup
        - MyExistingSecurityGroup
      KeyName: !Ref KeyName
      ImageId: ami-7a11e213
  InstanceSecurityGroup:
    Type: 'AWS::EC2::SecurityGroup'
    Properties:
      GroupDescription: Enable SSH access via port 22
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: '22'
          ToPort: '22'
          CidrIp: 0.0.0.0/0
  NewVolume:
    Type: "AWS::EC2::Volume"
    Properties: 
      Size: 
        !If [CreateLargeSize, 100, 10]
      AvailabilityZone: !GetAtt: Ec2Instance.AvailabilityZone
    DeletionPolicy: Snapshot

Output:

Resources:
  Ec2Instance:
    Type: AWS::EC2::Instance
    Properties:
      SecurityGroups:
      - InstanceSecurityGroup
      - MyExistingSecurityGroup
      KeyName: KeyName
      ImageId: ami-7a11e213
  InstanceSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Enable SSH access via port 22
      SecurityGroupIngress:
      - IpProtocol: tcp
        FromPort: "22"
        ToPort: "22"
        CidrIp: 0.0.0.0/0
  NewVolume:
    Type: AWS::EC2::Volume
    Properties:
      Size:
      - CreateLargeSize
      - 100
      - 10
      AvailabilityZone: Ec2Instance.AvailabilityZone
    DeletionPolicy: Snapshot

[cppforlife]

im guessing you are asking if it's possible to convert cloudformation dsl into ytt. the answer is no.

what we mean by "template any yaml content" is if you want to add some conditional logic or insert some variable values throughtout yaml, ytt can do that. (but it is out of scope for ytt to take place of cloudformation dsl as that would mean it would have to be executing in context of aws live objects to be able to assemble things together).

example:

config.yml

#@ load("@ytt:data", "data")

Resources:
  Ec2Instance:
    Type: 'AWS::EC2::Instance'
    Properties:
      SecurityGroups:
        - !Ref InstanceSecurityGroup
        - MyExistingSecurityGroup
      KeyName: !Ref KeyName
      ImageId: #@ data.values.image_id
  InstanceSecurityGroup:
    Type: 'AWS::EC2::SecurityGroup'
    Properties:
      GroupDescription: Enable SSH access via port 22
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: '22'
          ToPort: '22'
          CidrIp: 0.0.0.0/0
  #@ if/end data.values.include_volume:
  NewVolume:
    Type: "AWS::EC2::Volume"
    Properties: 
      Size: 
        !If [CreateLargeSize, 100, 10]
      AvailabilityZone: !GetAtt: Ec2Instance.AvailabilityZone
    DeletionPolicy: Snapshot

values.yml

#@data/values
---
image_id: ami-7a11e213
include_volume: true

you could run ytt -f config.yml -f values.yml -v image_id=other-ami-id

hopefully that answers your question.

[jdojff]

Hey, thank you for the answer.
Variable insertions are enough for me, same as in your example.
What I want is not to spoil the existing CF DSL when applying changes to original YAML.
Please take a look here:

SecurityGroups:
  - !Ref InstanceSecurityGroup
  - MyExistingSecurityGroup

becomes

SecurityGroups:
  - InstanceSecurityGroup
  - MyExistingSecurityGroup

or, this

Properties: 
  Size: 
    !If [CreateLargeSize, 100, 10]

becomes this.

Properties:
  Size:
    - CreateLargeSize
    - 100
    - 10

I expect the original DSL kept as it is, to remain valid CloudFormation template. If it's not possible with ytt, maybe you have other suggestions?
Thanks!

[cppforlife]

ah i see what you saying. you want to preserve yaml tags (like !Ref and !If) in resulting file after it runs through ytt. ytt currently doesnt support custom yaml tags hence it doesnt preserve them. thats something that would be interesting to add but im not sure about the amount of effort as it related to underlying yaml library.

one option you can use is more verbose cloudformation format like so:

#@ load("@ytt:data", "data")
Resources:
  Ec2Instance:
    Type: 'AWS::EC2::Instance'
    Properties:
      SecurityGroups:
        - {Ref: InstanceSecurityGroup}
        - MyExistingSecurityGroup
      KeyName: {Ref: KeyName}
      ImageId: #@ data.values.image_id

and then use -ojson as output type.

If it's not possible with ytt, maybe you have other suggestions?

im not sure which tools preserve yaml tags, so i dont have any good recommendation.

[brandonkal]

@jdojff Have you seen emscripten emrichen which uses custom yaml-tags extensively for templating? It may be a better fit for your use case. I haven't used it and would prefer ytt to support custom yaml tags in the long run.

[roaddemon]

I stumbled on this, and with some help on the #k14s slack channel have been able to replace all shorthand CloudFormation functions calls with standard form ones without any curly brace formatting or json output.

Here is a working DynamoDb example. First format doesn't work with ytt. Second does (replacing !Ref and !FindInMap shorthand functions)

Parameters:
  EnvName:
    Type: String
Mappings: 
  EnvMap: 
    Dev: 
      UsersTable: "Users-Dev"
      ConnectionsTable: "Connections-Dev"
    Prod: 
      UsersTable: "Users-Prod"
      ConnectionsTable: "Connections-Prod"
Resources: 
  UsersTable:
    Type: AWS::DynamoDB::Table
    Properties:
      AttributeDefinitions:
      - AttributeName: "connectionId"
        AttributeType: "S"
      KeySchema:
      - AttributeName: "connectionId"
        KeyType: "HASH"
      ProvisionedThroughput:
        ReadCapacityUnits: 5
        WriteCapacityUnits: 5
      TableName: !FindInMap
        - EnvMap
        - !Ref 'EnvName'
        - UsersTable

I replaced those with the standard form and it worked fine:

Parameters:
  EnvName:
    Type: String
Mappings: 
  EnvMap: 
    Dev: 
      UsersTable: "Users-Dev"
      ConnectionsTable: "Connections-Dev"
    Prod: 
      UsersTable: "Users-Prod"
      ConnectionsTable: "Connections-Prod"
Resources: 
  UsersTable:
    Type: AWS::DynamoDB::Table
    Properties:
      AttributeDefinitions:
      - AttributeName: "connectionId"
        AttributeType: "S"
      KeySchema:
      - AttributeName: "connectionId"
        KeyType: "HASH"
      ProvisionedThroughput:
        ReadCapacityUnits: 5
        WriteCapacityUnits: 5
      TableName: 
        Fn::FindInMap:
        - EnvMap
        - Ref: 'EnvName'
        - UsersTable

[pivotaljohn]

This issue has gone stale. Looks like there's a reasonable workaround. If we have renewed interest in support YAML tags through the whole processing, we can re-open.

[OskiKervinen-MF]

The work-around helps only for the specific case of AWS templates, where an equivalent longer syntax is available. In my use-case (GitLab CI yaml files !reference), there is no work-around for ytt eating the tag. Tags are a normal part of standard YAML, it's quite limiting for ytt to be utterly incompatible with them. Is there any way to force the output to contain a tag?

[Art3mK]

Oh man, just stepped on a !reference and ytt mine for GitLab pipelines as well.

so, in ytt template file I'm doing:

...
- \!reference [.snippets, script]
...

and then in parent pipeline

sed -i 's/\\!reference/!reference/g' gitlab-ci.yml

after I generate child pipelines configs with ytt :megusta: