Creating a lens step by step

After you've played a bit with Augeas, or even used it with the default lenses it comes with, you probably want to begin writing your own lenses to support more configuration files.

This page is a HowTo to help you create an Augeas lens. In this example, we will create a lens for the apt/preferences file, which looks like this:

Explanation: Backport packages are never prioritary
Package: *
Pin: release a=backports
Pin-Priority: 100
 
Explanation: My packages are the most prioritary
Package: *
Pin: release l=Raphink, version=3.0
Pin-Priority: 700

The main specifics of this format are:

Comments are not allowed, except in the Explanation fields
Records are separated by one or more empty lines
Fields are key/value pairs separated by ":"

Writing a test file

Since our goal is to be able to parse an apt/preferences file properly, I suggest we begin with a test file. This test file will be saved as /usr/share/augeas/lenses/tests/test_aptpreferences.aug. This will allow us to do a test-driven development, as soon as our lens will actually compile.

An Augeas test file is an Augeas module which uses an existing lens to test it against a given configuration string. The Language Overview explains how unit tests work.

In our case, it could look like this:

module Test_aptpreferences = 

let conf ="Explanation: Backport packages are never prioritary
Package: *
Pin: release a=backports
Pin-Priority: 100
 
Explanation: My packages are the most prioritary
Package: *
Pin: release l=Raphink, v=3.0
Pin-Priority: 700
"
 
test AptPreferences.lns get conf =
  { "1"
     { "Explanation"  = "Backport packages are never prioritary" }
     { "Package"      = "*" }
     { "Pin"          = "release"
         { "a" = "backports" } }
     { "Pin-Priority" = "100" } }
  {}
  { "2"
     { "Explanation"  = "My packages are the most prioritary" }
     { "Package"      = "*" }
     { "Pin"          = "release"
         { "l" = "Raphink" }
         { "v" = "3.0"     } }
     { "Pin-Priority" = "700" } }

This is the step where you will choose the way you will translate your file into the Augeas tree. In this case, I chose to represent records as sequences. This choice is reasonable here because my file only has records or empty lines, and nothing else, so my records can be simply listed numerically under the file root node. Inside each node, each main field will be a node with a value. In the case of the "Pin" node, we need one or more sub nodes to store the tests (e.g. a=backports or l=Raphink).

Notes:

Newlines are important in your configuration string! In this example, I didn't want a newline in the beginning of the file, and I wanted one in the end.
Notice the "{}" between the two records; this is necessary because Augeas will notice the empty line, even if it doesn't display it.
Do not forget to close the brackets properly when you specify the nodes!

Let's try to launch augparse on our test file:

$ augparse tests/test_aptpreferences.aug
tests/test_aptpreferences.aug:14.9-.27:Could not load module AptPreferences for Aptpreferences.lns
tests/test_aptpreferences.aug:14.9-.27:Undefined variable AptPreferences.lns
tests/test_aptpreferences.aug: error: Loading failed

It doesn't work... which is quite logical, since the AptPreferences module doesn't exist yet! So let's move to the lens development now.

Writing the lens

Setting the layout

To begin with, let us first set a layout for our lens. The lens file will be saved as /usr/share/augeas/lenses/aptpreferences.aug:

(* Apt/preferences module for Augeas          *)
 
module AptPreferences =
autoload xfm
 
let lns = 

let filter = incl "/etc/apt/preferences"
               . Util.stdexcl
 
let xfm = transform lns filter

Note that comments are in the form of (* something *).

Development choices

Now there are two main ways to get started with development:

from big to small
from small to big

Let me explain:

You could begin by describing how your file is made up generally, and then get into the details as you go, or by first describing all the little primitives of your file, and then build up the various kinds of records on top of them until you describe the whole file.

In this example, I chose the first option since the records are easy to define. In this other example, the lens is created from bottom to top.

Defining the sub-lenses

Since we chose to begin with the big picture, let's think about the whole file. The question is: `What is an apt/preferences file? What is it made of?'. The answer is quite easy: it is made of records and empty lines, any number of times. Alright, let's write that down:

(* Define lens *)
let lns = ( record | empty )*

Note: This definition means that our file could begin or end with any number of empty lines.

Now, what is an empty line? It is a line that contains only spaces. Let's write that down:

(* Define empty line *)
let empty = [ del /[ \t]*\n/ "" ]

This means that an empty line is made of any number of spaces or tabs, followed by a new line. "del" indicates that all these characters should be ignored in the tree, and "" indicates that the default value for an empty line is "".

Next question: `What is a record'? A record is a set of entries, nothing more:

(* Define record *)
let record = [ seq "record" . entries+ ]

These entries can be key/value entries (in the case of "Explanation", "Package", and "Pin-Priority") or special entries (the case of "Pin"). Let's write it this way for now:

let entries = explanation 
            | package
            | pin_priority
            | pin

Simple entries

Let's define explanation:

let explanation = [ key "Explanation" . Util.del_str ":" . store /[^\n]*/ . Util.del_str "\n" ]

An explanation entry begins with an "Explanation" word, which is our key, followed by a colon to be ignored, followed by anything but a newline, until the end of the line, which is to be ignored. "[" and "]" indicate that explanation is a lens, which will store a node in the Augeas tree.

Let's take some time to think for a minute. All our entries will use 'Util.del_str ":"', 'store /[^\n]*/' and 'Util.del_str "\n"', so it would be smart to define shortcuts for these.

(* Define useful primitives *)
let colon        = Util.del_str ":"
let eol          = Util.del_str "\n"
let value_to_eol = store /[^\n]*/

Alright... but how about spaces? Util.del_str ":" is the equivalent of del /:/ ":", but in our case, we could actually have spaces following the colon. Do we want to remove them altogether, or to keep them in the value? All the same, we could have spaces between the value and the newline... Let's fix this:

let colon        = del /:[ \t]*/ ": "
let eol          = del /[ \t]*\n/ "\n"

But now we have a problem: value_to_eol could begin or end with spaces, which would conflict with colon and eol. So we also have to modify value_to_eol:

let value_to_eol = store /[^ \t\n][^\n]*[^ \t\n]/

This would work, but only if my value is at least 2 characters long! So here is another way to write it:

let value_to_eol = store ( /[^\n]*/ - /([ \t][^\n]*|[^\n]*[ \t])/ )

Here we're saying: value_to_eol is anything but a newline, except if it begins or ends with a space or a tab.

In fact, a good practice here is to use Util.eol for end of lines:

let eol = Util.eol

You might even want to support DOS-style newlines, in which case you can use Util.doseol:

let eol = Util.doseol

Ok, now we're happy with our primivites, let's rewrite explanation:

let explanation = [ key "Explanation" . colon . value_to_eol . eol ]

It's easier to read, isn't it? If we move on to the next field, we'll get:

let package = [ key "Package" . colon . value_to_eol . eol ]

Wait a minute, this is quite the same! This is where functions can make our lives easier. We will write a simple_entry function as follows:

let simple_entry (kw:string) = [ key kw . colon . value_to_eol . eol ]

(kw:string) is a variable for the function. kw is the name of the variable and string is the type. Now explanation and package would look like this:

let explanation = simple_entry "Explanation"
let package     = simple_entry "Package"

These are a bit too simple to be declared individually. It's not really necessary anymore. So we could just redefine entries as:

let entries = simple_entry "Explanation" | simple_entry "Package" | simple_entry "Pin-Priority" | pin

The "Pin" entry

Now that we have defined simple entries, it's time to confront the "Pin" entry, which is a bit harder.

What is a "Pin" entry made of? It is made of a value, followed by any number of key/value pairs, separated by "=".

Let's write that down:

let pin = [ key "Pin" . colon . value_to_spc . Sep.space . key_value+ . eol ]

Sep.space is a useful definition from the Sep module, which is equivalent to del /[ \t]*/ " ". We need to define the value_to_spc and key_value primitives:

 let value_to_spc = store Rx.no_spaces
 let key_value    = [ key /[^= \/\t\n]+/ . Sep.equal . value_to_spc ]

where Rx.no_spaces is a definition from the Rx (regexps) module, and Sep.equal is taken from the Sep module.

Note that each key_value is a lens inside the pin lens, so it will create a new sub-node inside the '''Pin''' node.

The big picture

Let's see our file now:

(* Apt/preferences module for Augeas          *)
 
module AptPreferences =
autoload xfm
 
 
(* Define useful primitives *)
let colon        = del /:[ \t]*/ ": "
let eol          = Util.eol
let value_to_eol = store ( /[^\n]*/ - /([ \t][^\n]*|[^\n]*[ \t])/ )
let value_to_spc = store Rx.no_spaces
let key_value    = [ key /[^= \t\n]+/ . Sep.equal . value_to_spc ]
 
(* Define empty *)
let empty = Util.empty
 
(* Define record *)

let simple_entry (kw:string) = [ key kw . colon . value_to_eol . eol ]
let pin = [ key "Pin" . colon . value_to_spc . Sep.space . key_value+ . eol ]
 
let entries = simple_entry "Explanation"
            | simple_entry "Package"
            | simple_entry "Pin-Priority"
            | pin
 
let record = [ seq "record" . entries+ ]
 
(* Define lens *)
let lns = ( record | empty )*
 
let filter = incl "/etc/apt/preferences"
           . Util.stdexcl
 
let xfm = transform lns filter

Take some time to re-read it and understand the various parts of it.

Debug, debug...

OK, we wrote a lens... but does it compile?

does it compile now ?

$ augparse aptpreferences.aug
aptpreferences.aug:18.3-.76:Failed to compile key_value
aptpreferences.aug:18.24-.40:exception: The key regexp /[^=
]+/ matches a '/'
 
aptpreferences.aug:28.3-31.13:Failed to compile entries
aptpreferences.aug:25.36-.71:exception: ambiguous concatenation
      'Explanation: \n' can be split into
      'Explanation:|=| \n'
 
     and
      'Explanation: |=|\n'
 
    First lens: aptpreferences.aug:25.36-.65
    Second lens: aptpreferences.aug:15.22-.41
 
aptpreferences.aug: error: Loading failed

Unfortunately, no! But hey, it's OK! I don't know any good program without a few bugs!

So now let's try to understand these errors and fix them!

Refining pin keys

The first error has to do with the key in key_value, which is not precise enough. One way around is to specify it, since we know the values it could take:

let key_value (kw:string)    = [ key kw . Sep.equal . value_to_spc ]
let pin_key = key_value "a"
            | key_value "c"
            | key_value "l"
            | key_value "o"
            | key_value "v"

let pin = [ key "Pin" . colon . value_to_spc . Sep.space . pin_key+ . eol ]

This way we ensure that the key is one of the specified values.

value_to_eol

Let's switch to the second error. Augeas tells us that a conffile with 'Explanation: \n' could be split in two ways, using the lenses on lines 15 or 25. Let's look it up!

Line 15 is:

let eol = Util.eol

which is equivalent to del /[ \t]*\n/ "\n".

Line 25 is:

let simple_entry (kw:string) = [ key kw . colon . value_to_eol . eol ]

The error suggests that value_to_eol could be a simple space. In this case, Augeas wouldn't know if the space belongs to value_to_eol or to eol. It seems our value_to_eol definition doesn't really work. An easy way around would be to consider that eol is after all Util.del_str "\n", but the format allows to have spaces in the end of the line, so it could happen, and we wouldn't like to get the spaces inside our values.

Instead, we could rewrite value_to_eol as :

let value_to_eol = store /([^ \t\n].*[^ \t\n]|[^ \t\n])/

or use the Rx.space_in definition, which is equivalent:

let value_to_eol = store Rx.space_in

Note that . is the same as [^\n]. In this new regexp, we accept either any string of 2 characters minimum without a newline that doesn't begin or end with a space or tab, or a single character that is not a space, a tab, or a newline.

pin_key spaces

If we run the parser again, we get a new error:

aptpreferences.aug:33.68-.77:exception: ambiguous iteration
      'v=\002\001\001' can be split into
      'v=\002\001|=|\001'

     and
      'v=\002\001|=|\001'
 
    Iterated lens: aptpreferences.aug:27.18-31.24

Ah, we made a mistake in the pin entry again!

let pin = [ key "Pin" . colon . value_to_spc . Sep.space . pin_key+ . eol ]

Where are the separators between the various possible pin_key entries? The format allows to have several pin_key key/value pairs, separated by a comma. So here we go:

let pin = [ key "Pin" . colon . value_to_spc . Sep.space . pin_key . ( del /,[ \t]*/ ", " . pin_key )*. eol ]

The error is still here though, it just looks a bit different:

aptpreferences.aug:27.79-.113:exception: ambiguous iteration
      ',v=\002\001\001\001' can be split into
      ',v=\002\001|=|\001\001'
 
     and
      ',v=\002\001|=|\001\001'
 
    Iterated lens: aptpreferences.aug:27.81-.110

It is currently possible for a value_to_spc to contain a ,, so Augeas wouldn't know if the , is part of value_to_spc or if it separates pin_key entries, thus we change value_to_spc into:

let value_to_spc = store /[^, \t\n]+/

Now we get:

aptpreferences.aug:37.13-.32:exception: ambiguous iteration
      'Pin-Priority:\001\nPin:\001\001\001\n\001' can be split into
      'Pin-Priority:\001\n|=|Pin:\001\001\001\n\001'
 
     and
      'Pin-Priority:\001\nPin:\001|=|\001\001\n\001'

Ah, we forgot another important point! Records are separated by newlines, so a record is actually:

let record = [ seq "record" . entries+ . eol ]

Now the lens compiles without an error. It's time to test it!

The debugged lens

(* Apt/preferences module for Augeas          *)
 
module AptPreferences =
autoload xfm
  
(* Define useful primitives *)
let colon        = del /:[ \t]*/ ": "
let eol          = Util.eol
let value_to_eol = store Rx.space_in
let value_to_spc = store /[^, \t\n]+/
 
(* Define empty *)
let empty = Util.empty
 
(* Define record *) 
let simple_entry (kw:string) = [ key kw . colon . value_to_eol . eol ]
 
let key_value (kw:string)    = [ key kw . Sep.equal . value_to_spc ]
let pin_key = key_value "a"
            | key_value "c"
            | key_value "l"
            | key_value "o"
            | key_value "v"
 
let pin = [ key "Pin" . colon . value_to_spc . Sep.space . pin_key . ( del /,[ \t]*/ ", " . pin_key )*. eol ]
 
let entries = simple_entry "Explanation"
            | simple_entry "Package"
            | simple_entry "Pin-Priority"
            | pin
 
let record = [ seq "record" . entries+ . eol ]
 
(* Define lens *)
let lns = ( record | empty )*
 
let filter = incl "/etc/apt/preferences"
           . Util.stdexcl
 
let xfm = transform lns filter

Testing the lens

Now is the time to run our test!

$ augparse tests/test_aptpreferences.aug
Test run encountered exception:
tests/test_aptpreferences.aug:14.9-.36:exception: Short iteration
    Error encountered here (107 characters into string)
    <ackports\nPin-Priority: 100\n\n|=|Explanation: My packages are>
 
    Tree generated so far:
    /1
/1/Explanation = "Backport packages are never prioritary"
/1/Package = "*"
/1/Pin = "release"
/1/Pin/a = "backports"
/1/Pin-Priority = "100"

 
tests/test_aptpreferences.aug: error: Loading failed

Err... there's still something wrong! We could go back to the last fix we made to understand it. We added an eol in the end of a record. That fixed the lens, but it doesn't reflect reality. In reality, the last record doesn't necessarily end with a newline. Furthermore, the line that separates two records is not really an eol, but an empty line instead.

So there we go:

let record = [ seq "record" . entries+ ]
let lns = empty* . ( record . empty )* . record?

The record itself doesn't end with an eol anymore. Instead, the lns takes care of the separation. A lns is now an optional amount of empty lines in the beginning, followed by an optional series of records and empty lines, optionally followed by a single record, in case there is no empty line in the end of the file.

Let's check the lens again:

$ augparse aptpreferences.aug

And now the test:

$ augparse tests/test_aptpreferences.aug

It also works!

To infinity and beyond!

This lens is far from being perfect. It is merely a suggestion of how to parse and represent this configuration file.

Here are a few suggestions if you feel like going further on it:

the key_value function could benefit from another argument, typed as a regex, to refine the contents of each field. The same could be done for simple_entry and value_to_spc used in the pin declaration.
instead of mapping pin keys directly under the '''Pin''' tree ( e.g. /files/etc/apt/preferences/1/Pin/a = "backports" ), it could be interesting to put another level which might help listing the keys ( e.g. /files/etc/apt/preferences/1/Pin/key[1] = a; /files/etc/apt/preferences/1/Pin/key[1]/value = "backports" )

And there's always more to improve...

Provide feedback

Saved searches

Use saved searches to filter your results more quickly