Add new lense for Teleport service

[smgrol]

Hi!

This lense read teleport configuration file https://goteleport.com/ .

The existing yaml lens does not work correctly with this type of configuration file, so we wrote a separate parser for a specific service.

[georgehansper]

It looks like the teleport.yaml file is a straight-forward Yaml file.
As you noticed, the existing Yaml lens handles only a small subset of Yaml.

This lens attempts to at least handle 2 levels of mappings (also known as hashes or dictionaries), plus sequences (ordered lists)

Unfortunately, although the lens passes the tests supplied, it does not achieve its desired goals.

For example, taking a sample from the test_teleport.aug comments as a teleport.yaml file:

teleport:
  nodename: nodename
  auth_servers:
    - host:port

Gives the following output from augtool:

> augtool print /files/etc/teleport.yaml
/files/etc/teleport.yaml
/files/etc/teleport.yaml/teleport
/files/etc/teleport.yaml/teleport/nodename = "nodename"
/files/etc/teleport.yaml/auth_servers

Notice that the value "host:port" is missing from the output

Also, the lens fails a basic test where a single key-value is created

> augtool
augtool> print /files/etc/teleport.yaml
augtool> set /files/etc/teleport.yaml/teleport/nodename 'nodename'
augtool> save
error: Failed to execute command
saving failed (run 'errors' for details)
augtool> 

Also note that, at the current version, Augeas does not handle recursion within a lens definition.

[smgrol]

Also, the lens fails a basic test where a single key-value is created

This lense only for read, not for update/create values.

We need reading this configuration file without updating them. Because we are use osquery and osquery parses configuration files with augeas, we wrote this lense for this goal..

For update this configuration files must be used specific playbooks with corporate templates.

[georgehansper]

I can see how this may be useful for osquery, and that it solves a particular problem related to your setup

Augeas does provide for users to create their own custom lenses, and these can be read automatically by augeas by setting the environment variable AUGEAS_LENS_LIB

For example:

mkdir -p /usr/local/share/augeas
cp teleport.aug /usr/local/share/augeas
export AUGEAS_LENS_LIB=/usr/local/share/augeas
augtool --load-file /etc/teleport.yaml print /files/etc/teleport.yaml

The environment variable AUGEAS_LENS_LIB applies for both augtool and any program linked to the library libaugeas.so

The allows users to have custom special-purpose lenses which will not be overwritten by a package update.

In this case, I would suggest that this is a better solution, until a more complete Yaml lens is developed.

[smgrol]

@georgehansper thank you!

I am trying write a complete Yaml parser. True, it won't be easy.

So, all lenses must work with the SET request?

[georgehansper]

I think it is important to keep in mind that Augeas is actually a file editor.

It is intended to be able to make changes to config files, both by changing existing values and adding additional content.

By and large, the existing lenses achieve this.

Although Augeas is somewhat "aware" of the syntax of the file, it is not designed to be a config checker or linter.

Lenses convert the file's native syntax into a "tree" representation.

The set command operates on the tree, and more often than not, the set command will succeed, even if it makes the resulting tree "invalid" for the underlying lens.

So to answer your question, it's not the "set" function that must succeed, it is the subsequent "save" command.
If you have Augease 1.13.0 or later, the "preview" does the same thing as "save", but only for a single file (path), and "preview" does not write to the filesystem.

A good criteria for success here would be the following should succeed:

• create the desired file with the correct syntax for the application
• augtool print /files/etc/path/to/file
• Using the paths and values, presented, create an augtool script file consisting of a series of "set" commands (in the order printed)
• Remove the original file
• Run
  • augtool --autosave --noload --file path_to_augtool_script

The original config file should be re-created, and be syntactically correct.
Likely the resulting file will not be exactly the same, because augeas uses the "default" values from the lens for inserting spaces and the like.

[smgrol]

Hello @georgehansper ! Thanks for the clarification.

I am updated lense for read a simple/classic yaml files. These file name is like simplevars/simplelines lenses.
Existing YAML parser very specific and not used often. But I don't changing existing lense because I am afraid that something is go wrong with parsing files with these YAML parser.

My parser works with simple 'key: value' params, indented 'key: value', sequence and comments. In test file I am add multiple config format.

I tested with your criteria:

1. create file for test /etc/test3.yaml

foo: bar
pleh: help
stuff:
  foo: bar
  bar: foo

2. augtool print /files/etc/test3.yaml

/files/etc/test3.yaml
/files/etc/test3.yaml/foo = "bar"
/files/etc/test3.yaml/pleh = "help"
/files/etc/test3.yaml/stuff
/files/etc/test3.yaml/stuff/foo = "bar"
/files/etc/test3.yaml/stuff/bar = "foo"

3. create augtool_script

set /files/etc/test3.yaml
set /files/etc/test3.yaml/foo  "bar"
set /files/etc/test3.yaml/pleh  "help"
set /files/etc/test3.yaml/stuff
set /files/etc/test3.yaml/stuff/foo  "bar"
set /files/etc/test3.yaml/stuff/bar  "foo"

4. remove /etc/test3.yaml
5. run augtool --autosave --noload --file augtool_script
6. file /etc/test3.yaml created with content , but there is no spaces.

foo: bar
pleh: help
stuff:
foo: bar
bar: foo

8. Additional:
   augtool set /files/etc/test3.yaml/foo 'new_foo'
   return updated value foo

foo: new_foo
pleh: help
stuff:
foo: bar
bar: foo

So, YAML format is not simple for parse all available type of key:values . For example, a pyyaml project - big project for analyze yaml files.

Although my work covers a small percentage of possible scripts for writing the YAML format, nevertheless it is most often found in the configuration files of many services. And these can be useful to many people.

I'm waiting for a review. Thank you.

[georgehansper]

Thanks for having a go at this. Writing lenses is tricky at the best of times, and I have to admit that it is not something I have done much of to date.

In step 6 of your test, the indentation is lost on the values of stuff
This changes the meaning of the YAML. If you load the resulting file into augtool, you would get a different result

This means that after a "save", the resulting file would become corrupted.
This is not acceptable, because it would create enormous frustration to other users, who would not be expecting this behaviour

I can see that you have made use of the existing Util and Rx modules. Unfortunately, this is why the indentation is lost during the save operation.

The definition of Util.indent is:

  let indent = del /[ \t]*/ ""

Note the arg "". This tells Augeas: "when creating a new entry, put an empty string at this location."

So Util.indent is part of the problem

I think we can do better than this, by telling Augeas to insert some spaces.

I have been experimenting with the following code, which shows some promise:

let indent1 = del "  " "  "
let indent2 = del "    " "    "
let eol = /[ \t]*\n/
let nl = del "\n" "\n"
let trim_nl = del /[ \t]*\n/ "\n"

let empty_line = del /^\n/ ""
let keyname = /[a-zA-Z0-9_][-a-zA-Z0-9_]*/
let rest_of_line = /[^\n]+/
let colon = del /: */ ":"
let colon_sp = del /: */ ": "

let scalar = [ key keyname . colon_sp . store rest_of_line . nl ]
let comment = [ label "#comment" . del "#" "#" . store /.*/ . nl ]

let level1_dict = [ key keyname . colon . trim_nl .
   ( [ indent1 .  key keyname  . colon_sp . store rest_of_line . nl ] |
     [ indent1 .  key keyname  . colon . trim_nl .
         ( [ indent2 . key keyname . colon_sp . store  rest_of_line . nl ] )*
     ]
   )*
  ]
(* View: lns *)
let lns = ( comment | scalar | level1_dict | empty_line )*

Using the following test-data:

# ignore this
scalar1: value1
scalar2: value2
dict1:
  key1: value_1
  key2: value_2
  dict2:
    keylevel2: value-level2

It loads as expected, and if I run the following commands:

set /files/etc/fake_yaml.yaml/dict1/key3 "value (appended)"
save

It successfully saves the file with key3 at the desired indentation (ie the same as key2), as the last line of the file
It also keeps the indentation of the other keys

It is by no means a full Yaml parser, but it is certainly an improvement