lux.html

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
  <meta charset="UTF-8"/>
  <title>Lux - LUcid eXpect scripting</title>
</head>
<body>
<h1>Lux - LUcid eXpect scripting</h1>

<p>Version 2.9.1 - 2023-10-25</p>

<ul>
<li><a href="#../README">Introduction</a></li>
<li><a href="#main_concepts">Concepts</a></li>
<li><a href="#../tutorial/README">Tutorial</a></li>
<li><a href="#script_syntax">Script syntax</a></li>
<li><a href="#cmd_line_opts">Command line options</a></li>
<li><a href="#config_params">Configuration parameters</a></li>
<li><a href="#logs">Logs</a></li>
<li><a href="#debug_cmds">Debugger for Lux scripts</a></li>
<li><a href="#examples">Examples</a></li>
<li><a href="#hardening">Hardening test cases</a></li>
<li><a href="#warnings">Warnings</a></li>
<li><a href="#../INSTALL">Installation</a></li>
<li><a href="#../AUTHORS">Original author</a></li>
<li><a href="#references">References</a></li>
</ul>

<p><a name="../README"/></p>

<h1>Introduction</h1>

<p>Lux (LUcid eXpect scripting) is a test automation framework with
Expect style execution of commands. See <a href="http://www.nist.gov/el/msid/expect.cfm" title="Expect homepage">Expect</a> for more info about
the origin.</p>

<p>With Lux it is possible to</p>

<ul>
<li>simplify automated testing</li>
<li>control interactive programs by sending textual input to them and
using <a href="http://www.erlang.org/doc/man/re.html#regexp_syntax" title="Erlang style regular expressions (re)">regular expression</a>s to ensure that their output matches the
expectations</li>
<li>perform detailed post mortem analyzis of test suite results</li>
<li>interactively debug and trace single test cases</li>
<li>get editor support for editing scripts by using the <a href="http://www.gnu.org/software/emacs" title="Emacs text editor">Emacs</a> mode</li>
</ul>

<p>The tool is written in <a href="http://www.erlang.org/" title="Erlang programming language">Erlang/OTP</a> and requires its runtime
environment.</p>

<p>See the file <strong>lux.html</strong> for the full documentation or view it online
on <a href="https://github.com/hawk/lux/blob/master/doc/lux.md">GitHub</a>.</p>

<h1>Video walkthru of the tutorial</h1>

<p>This talk about Lux was presented at the Erlang User Conference 2019:</p>

<p>[https://www.youtube.com/watch?v=Nu15YOpmCKQ]</p>

<h2>A sample script</h2>

<p>Here is an example of a test script. It starts couple of concurrent
shells, sends text to them with the <code>!</code> command and matches expected
output with <code>?</code>.</p>

<p>Snippet from the enclosed <code>.../lux/examples/intro.lux</code> file:</p>

<blockquote>
<pre><code>[doc Test of single and multi line regular expressions]

# Assign a global variable which is accessible in all shells
[global file=removeme.txt]

# Start a shell
[shell single]
  # Send text to the active shell
  !echo foo
  # Match output from the active shell
  # The terminal echoes all input and here we match on the echoed input
  ?echo foo

# Start yet another shell (and make it the active one)
[shell multi]
  # Create a file where bar and baz happens to be indented
  # Variables are
  !echo "foo"      &gt; $file
  !echo "    bar" &gt;&gt; $file
  !echo "  baz"   &gt;&gt; $file
  !echo "fum"     &gt;&gt; $file

  # Single line matches
  !cat $file
  ?foo
  ?bar
  # Don't bother of matching baz. All output between bar and fum is skipped.
  ?fum
  # Match the predefined shell prompt
  ?SH-PROMPT:

  # Multi line match. The first double quote char defines the first
  # column of the regexp. The indentation of bar and baz is significant.
  !cat $file
  """?
  foo
      bar
    baz
  fum
  SH-PROMPT:
  """

# Switch back to the first shell
[shell single]
  # Match the actual output from the echo command
  ?^foo

# Cleanup side effects. The cleanup section is always executed,
# regardless of the script succeeds or fails
[cleanup]
  !rm -f $file
  ?SH-PROMPT:
  # Match command exit status. Observe the double dollar sign which
  # escapes the dollar sign, implying "echo ==$$?==" to be sent to
  # the shell.
  !echo ==$$?==
  ?^==0==
</code></pre>
</blockquote>

<h2>How to run the script</h2>

<p>Run a single script like this:</p>

<p>Evaluate <code>lux examples/intro.lux</code></p>

<blockquote>
<pre><code>.../lux&gt; lux examples/intro.lux
summary log       : /Users/hmattsso/dev/lux/lux_logs/run_2022_06_27_19_40_24_793549/lux_summary.log
test case         : examples/intro.lux
progress          : ..:...:.:.:..:.:.:.....:...:.:.:.:..:..:.:..:.:..:.:.:..:.:.:..:..:.:.:..:.:...:..:.:.:....c:......:..:.:..:.:..:..:.:.:..:.:..:.
result            : SUCCESS
successful        : 1
summary           : SUCCESS
file:///Users/hmattsso/dev/lux/lux_logs/run_2022_06_27_19_40_24_793549/lux_summary.log.html
.../lux&gt; echo $?
0
</code></pre>
</blockquote>

<p>In this run we got a (brief) progress report of the test case on
stdout and a link to a summary log containing (lots of) details.</p>

<h2>How to assemble the history of multiple runs</h2>

<p>In a nightly build environment it might be difficult to pinpoint when
a certain test case/suite started to fail. This process is greatly
simplified by running <code>lux</code> with the <code>--history</code> option as it will
assemble all test results as a timeline (interleaved with change-set
identities if provided with <code>--revision</code>).</p>

<p>Evaluate <code>lux --revision svn_4711 --run jenkins_17 examples</code></p>

<p>Evaluate <code>lux --revision svn_4712 --run jenkins_20 examples/intro.lux</code></p>

<p>Evaluate <code>lux --revision svn_4712 --run jenkins_20 examples/fail.lux</code></p>

<p>Evaluate <code>lux --revision svn_4715 --run jenkins_22 examples</code></p>

<p>Evaluate <code>lux --history .</code></p>

<blockquote>
<pre><code>.../lux&gt; lux --history .
Cwd: /Users/hmattsso/dev/lux
Invoke: /Users/hmattsso/dev/lux/bin/lux --history .
Assembling history of logs from...
./lux_history.cache (678 bytes)
&lt;WARNING&gt; Cache file is incompatible with previous run ./lux_history.cache: ignoring cache
. eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee.eee.........eeeeee......................................................eeeeee...........................eeeee.........eeeeee.........eeeeee....eeeee.eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee..eeeeeeeeeeeeeeeeeeeeeee.......eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee
Wrote 9907 bytes in run cache to file ./lux_history.cache
Analyzed 123 test runs with 554 test cases (0 errors)...ok
file:///Users/hmattsso/dev/lux/lux_history.html
.../lux&gt; echo $?
0
</code></pre>
</blockquote>

<p><a name="main_concepts"/></p>

<h2>Concepts</h2>

<p>A Lux script may succeed or fail, meaning that the system under
test is either conforming to or diverging from the expected
behavior. A Lux script may also end with an <strong>error</strong>, if the
Lux engine encountered some problem, and could not determine
whether the system under test conforms or not. A syntax error in
the Lux script is an error, not a failure.</p>

<p>A Lux script consists of a sequence of instructions, mostly <code>send</code>
and <code>expect</code> operations, read in sequence from top to bottom. The test
case <strong>succeed</strong>s if all statements in the script are executed (or if an
optional success criteria matched). The test case <strong>fail</strong>s if there
is an <code>expect</code> operation not matching within a given time (or if an
optional failure criteria matches).</p>

<p>Each test case should not depend on other test cases being executed
before (for example preparing something) or after (to clean up).
The <strong>cleanup</strong> procedure is an integral part of each test case.</p>

<p>Each <code>send</code> and <code>expect</code> operation is directed to one particular shell
(the <strong>active shell</strong>). Input is sent to the <strong>stdin</strong> stream of the
shell. The <strong>stdout</strong> and <strong>stderr</strong> streams of the shell are combined
to one single output stream. A Lux script can start and control
many concurrent shells, but at any time point only one is the active
shell which can evaulate new Lux statements.</p>

<p>It is possible to reference <strong>variables</strong> in the <code>send</code>, <code>expect</code>,
<code>my</code>, <code>local</code>, <code>global</code> and <code>config</code> statements, using the standard
shell <code>$var</code> or <code>${var}</code> notation. Note that the substitution is
performed by the Lux engine and not by the process running within
the shell. The variables are initially set to all <strong>environment
variables</strong>. But new settings can be added by using <code>[my var=val]</code>,
<code>[local var=val]</code> and <code>[global var=val]</code>. Each such variable setting
overrides existing settings of the same variable.</p>

<p>If no variable substitution should take place, the dollar sign must be
<em>escape</em>d with yet another dollar sign (<code>$</code>). For example, if the
actual value of an environment variable named <code>var</code> should be read
from the Bourne shell, the <code>$var</code> string cannot be substituted to
another value by the Lux engine. The string <code>$var</code> must be sent
literally to the shell even if the Lux engine happens to have
have a variable registered with that name. In order to achieve this
the <code>$var</code> must be escaped as <code>$$var</code>.</p>

<p>The <a href="http://www.erlang.org/doc/man/re.html#regexp_syntax" title="Erlang style regular expressions (re)">regular expression</a>s in <code>expect</code> statements may contain
<strong>sub-patterns</strong>. The values matching captured sub-patterns may be
accessed in the following variable assignments. For example if the
statement <code>?begin (.*) middle (.*) end</code> matches the actual shell output
<code>begin abc middle def end</code>. The captured sub-patterns can be accessed
as numbered variables: <code>[local foo=the vals are $1 and $2]</code> which will
assign the variable <code>foo</code> to the value "<code>the vals are abc and def</code>".
As in any language <a href="http://www.erlang.org/doc/man/re.html#regexp_syntax" title="Erlang style regular expressions (re)">regular expression</a>s contains keywords. If the
output of a shell contains such a keyword and we want to match that,
the keyword must be escaped with a backslash. Example of such keywords
are any of the characters <code>^$.?+*()[]{}|</code>.</p>

<p>The variable substitution mechanism makes it also possible to reuse
(parts of) generic scripts in several test cases. The (main) script
for these test cases may assign different values to variables and then
include a generic script that makes use of these variables.</p>

<p>See the documentation about <a href="http://www.erlang.org/doc/man/re.html#regexp_syntax" title="Erlang style regular expressions (re)">regular expression</a>s in Erlang for
details about the regular expression dialect used in Lux. It is
an extended subset of <a href="http://www.pcre.org/" title="PCRE - Perl Compatible Regular Expressions">PCRE</a>.
<a name="../tutorial/README"/></p>

<h1>Tutorial</h1>

<h2>Installation</h2>

<p>Read the file <strong>.../lux/tutorial/INSTALL.md</strong> or view it online on
<a href="https://github.com/hawk/lux/blob/euc/tutorial/INSTALL.md">GitHub</a>
about how to do a install LUX, build and test the chatty app.</p>

<blockquote>
<pre><code>cd .../lux/tutorial/chatty
make build
</code></pre>
</blockquote>

<h2>How do we test a simple chat server?</h2>

<p>Imagine a scenario where we start a server and two clients. The
clients cannot connect until the server is up and running. When text
is entered in one client it must be verified that it is displayed in
the other client(s). You can start the system with these commands,
using three different shells:</p>

<blockquote>
<pre><code>cd chatty/test/intro
erl -pa ../../../chatty/ebin -noshell -sname mytopic -s chatty server
erl -pa ../../../chatty/ebin -noshell -sname cons    -s chatty client mytopic
erl -pa ../../../chatty/ebin -noshell -sname hawk    -s chatty client mytopic
</code></pre>
</blockquote>

<h2>Walkthru the test cases and emphasize on their differences</h2>

<p>Now when you are familiar with the system, how would you write an
automated test case for it? That is a stable test without race
conditions.</p>

<p>Walkthru these test cases below and emphasize on their differences.
Hopefully the test code is self-explanatory.</p>

<p>Evaluate <code>cd tutorial/chatty/test/intro &amp;&amp; lux .</code></p>

<blockquote>
<pre><code>.../lux&gt; cd tutorial/chatty/test/intro &amp;&amp; lux .
summary log       : /Users/hmattsso/dev/lux/tutorial/chatty/test/intro/lux_logs/run_2022_06_27_19_40_56_509994/lux_summary.log
test case         : a_simple_server.lux
progress          : ..:...:.:.:.:..:.:..:.:.:.:.:..:.:....:.:.:..:.:.:..:..:.:..:.:..:.:....
result            : SUCCESS
test case         : async_startup_fail.lux
progress          : ..:...:.:.:.:...:.:.:.:.:.:.:.:.:.:....:.:.:..:.:..:.:..:..:.:.:..:.:.Will fail due to startup race cond.:.:.:..:.:.:.:.:.:.:.:.:.:.:.25????25..
result            : FAIL at line 25 in shell hawk
expected*
Trying to join the mytopic chat room...
Welcome to the chat room mytopic!a!!
Enter text and press enter. Exit chat with \^d.

hawk&gt;
actual match_timeout
erl -pa ../../../chatty/ebin -sname hawk -noshell -s chatty client myt opic
Trying to join the mytopic chat room...
&lt;ERROR&gt; Failed to join 'mytopic@HMATTSSO-M-74JD'. Is the server started?
{"init terminating in do_boot",shutdown}
init terminating in do_boot (shutdown)
SH-PROMPT:
diff
+ erl -pa ../../../chatty/ebin -sname hawk -noshell -s chatty client myt 
+ opic
  Trying to join the mytopic chat room...
- Welcome to the chat room mytopic!a!!
- Enter text and press enter. Exit chat with \^d.
- 
- hawk&gt;
+ &lt;ERROR&gt; Failed to join 'mytopic@HMATTSSO-M-74JD'. Is the server started?
+ {"init terminating in do_boot",shutdown}
+ init terminating in do_boot (shutdown)
+ SH-PROMPT:

test case         : sync_startup.lux
progress          : ..:...:.:.:.:..:..:.:.:.:.:.:.:.:.:.:.:.:.:..:.:.:.:....:..:..:.:..:..:.:.:....:.:..:..:.:..:.:..:.:.:.:.:.:.:.:.:.:.:.:....:...:.:.:..:.:..:.:.:.:.:.:.:.:.:.:.::......:.:.:.....:............
result            : SUCCESS
test case         : sync_startup_cleanup.lux
progress          : ()..:...:.:.:.:.:....:.:.:.:.:.:.:.:.:.:..:.:.:.:....:..:.:..:.:..:..:.:.:.:....:...:.:.:.:...:.:.:.:.:.:.:.:.:.:.:.:.:.:.:....:..:..:.:.:...:.:.:.:.:.:..:.:....:.:.:.::..c..........:..:..:.:.(..:.:.:.:.:.)(..:.:.:.:.)((.:..:.:.:.:.:.:.:.:.:.:.:.:.:.)(.:.:..:..))((..:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.)(.:.:..:..))
result            : SUCCESS
successful        : 3
failed            : 1
async_startup_fail.lux:25 - match_timeout
summary           : FAIL
file:///Users/hmattsso/dev/lux/tutorial/chatty/test/intro/lux_logs/run_2022_06_27_19_40_56_509994/lux_summary.log.html
.../lux&gt; echo $?
1
</code></pre>
</blockquote>

<p>Snippet from the enclosed <code>.../lux/tutorial/chatty/test/intro/a_simple_server.lux</code> file:</p>

<blockquote>
<pre><code>[doc Demo a simple single shell test case]

# Start a shell
[shell server]
  # Send text to the active shell
  !erl -sname server -pa ../../../chatty/ebin
  # Match output from the active shell
  ?Erlang/OTP
  ?Eshell
  ?&gt; 

  !chatty:server().
  ?Starting server
  ?&gt; 

  !halt(3).
  ?SH-PROMPT:

  !echo "===$?==="
  ?===3===
  ?SH-PROMPT:
</code></pre>
</blockquote>

<p>Snippet from the enclosed <code>.../lux/tutorial/chatty/test/intro/async_startup_fail.lux</code> file:</p>

<blockquote>
<pre><code>[doc Demo too fast startup]

# Assign a global variable which is accessible in all shells
[global topic=mytopic]
[global ebin=../../../chatty/ebin]

[shell server]
  !erl -pa $ebin -sname $topic -s chatty server
  ?Starting server

[shell hawk]
  !export ERL_CRASH_DUMP_BYTES=0
  ?SH-PROMPT:
  !erl -pa $ebin -sname hawk -noshell -s chatty client $topic

  [progress Will fail due to startup race cond]
  # Multi line match. The first double quote char defines the first
  # column of the regexp.
  """?
  Trying to join the $topic chat room...
  Welcome to the chat room $topic!a!!
  Enter text and press enter. Exit chat with \^d.

  hawk&gt;
  """
</code></pre>
</blockquote>

<p>Snippet from the enclosed <code>.../lux/tutorial/chatty/test/intro/sync_startup.lux</code> file:</p>

<blockquote>
<pre><code>[doc Demo start sync]

[global topic=mytopic]
[global ebin=../../../chatty/ebin]

[shell server]
  !erl -pa $ebin -sname $topic -s chatty server
  ?Starting server

  # Match sub-expressions
  ?Trying to open log file (.*)\.\.\.ok.
  [global logfile=$1]

# Start another shell
[shell server-log]
  # Match in log file
  !tail -F $logfile
  ?Server started

[shell hawk]
  !erl -pa $ebin -sname hawk -noshell -s chatty client $topic
  # Match with variable
  """?
  Trying to join the $topic chat room...
  Welcome to the chat room $topic!!!
  Enter text and press enter. Exit chat with \^d.

  hawk&gt;
  """

[shell cons]
  !erl -pa $ebin -sname cons -noshell -s chatty client $topic
  # Verbatim match
  """??
  Trying to join the $topic chat room...
  Welcome to the chat room $topic!!!
  Enter text and press enter. Exit chat with ^d.

  cons&gt;
  """

# Switch active shell
[shell hawk]
  ?cons: Client joined
  !ping
  ?hawk&gt;

[shell server-log]
  ?Client hawk said ping

[shell cons]
  ?hawk: ping
</code></pre>
</blockquote>

<p>Snippet from the enclosed <code>.../lux/tutorial/chatty/test/intro/sync_startup_cleanup.lux</code> file:</p>

<blockquote>
<pre><code>[doc Demo cleanup]

[include ../../../support/luxinc/macros.luxinc]

[global topic=mytopic]
[global ebin=../../../chatty/ebin]

[shell server]
  # Set fail pattern for shell
  -[Ee][Rr][Rr][Oo][Rr]]
  !erl -pa $ebin -sname $topic -s chatty server
  ?Starting server

  ?Trying to open log file (.*)\.\.\.ok.
  [global logfile=$1]

[shell server-log]
  !tail -F $logfile
  ?Server started

[shell hawk]
  !erl -pa $ebin -sname hawk -noshell -s chatty client $topic
  """??
  Trying to join the $topic chat room...
  Welcome to the chat room $topic!!!
  Enter text and press enter. Exit chat with ^d.

  hawk&gt;
  """

[shell cons]
  # Use interactive Erlang shell
  !erl -pa $ebin -sname cons
  ?Erlang/OTP
  ?Eshell
  ?cons@
  !chatty:client(['${topic}']).
  """??
  Trying to join the $topic chat room...
  Welcome to the chat room $topic!!!
  Enter text and press enter. Exit chat with ^d.

  cons&gt;
  """

[cleanup]
  # Kill lingering processes
  [invoke eval_any "pkill -f beam.*chatty.*client"]
  [invoke eval_any "pkill -f beam.*chatty.*server"]

  # Save log file
  [invoke eval "mkdir -p ${LUX_EXTRA_LOGS}"]
  [invoke eval "cp $logfile ${LUX_EXTRA_LOGS}/"]
</code></pre>
</blockquote>

<h2>Post mortem analysis</h2>

<p>Walkthru the different logs from the latest test run. They are found
at <code>lux_logs/latest_run</code>. With this command you get a list of all logs:</p>

<p>Evaluate <code>cd tutorial/chatty/test/intro &amp;&amp; ls -ld lux_logs/latest_run</code></p>

<blockquote>
<pre><code>.../lux&gt; cd tutorial/chatty/test/intro &amp;&amp; ls -ld lux_logs/latest_run
lrwxr-xr-x 1 hmattsso staff 30 Jun 27 21:40 lux_logs/latest_run -&gt; run_2022_06_27_19_40_56_509994
.../lux&gt; echo $?
0
</code></pre>
</blockquote>

<p>Evaluate <code>cd tutorial/chatty/test/intro &amp;&amp; find -L lux_logs/latest_run</code></p>

<blockquote>
<pre><code>.../lux&gt; cd tutorial/chatty/test/intro &amp;&amp; find -L lux_logs/latest_run
lux_logs/latest_run
lux_logs/latest_run/sync_startup.lux.event.log
lux_logs/latest_run/sync_startup.lux.event.log.html
lux_logs/latest_run/sync_startup_cleanup.lux.cleanup.stdin.log
lux_logs/latest_run/a_simple_server.lux.server.stdout.log
lux_logs/latest_run/sync_startup.lux.hawk.stdout.log
lux_logs/latest_run/async_startup_fail.lux.hawk.stdin.log
lux_logs/latest_run/async_startup_fail.lux.event.log.html
lux_logs/latest_run/async_startup_fail.lux.hawk.stdout.log
lux_logs/latest_run/async_startup_fail.lux.event.log.csv
lux_logs/latest_run/sync_startup_cleanup.lux.server.stdout.log
lux_logs/latest_run/sync_startup.lux.hawk.stdin.log
lux_logs/latest_run/lux_summary.log.html
lux_logs/latest_run/async_startup_fail.lux.config.log
lux_logs/latest_run/sync_startup_cleanup.lux.cons.stdin.log
lux_logs/latest_run/sync_startup_cleanup.lux.server-log.stdout.log
lux_logs/latest_run/sync_startup_cleanup.lux.extra.logs
lux_logs/latest_run/sync_startup_cleanup.lux.extra.logs/chatty_mytopic.log
lux_logs/latest_run/sync_startup.lux.orig
lux_logs/latest_run/sync_startup_cleanup.lux.event.log.csv
lux_logs/latest_run/sync_startup_cleanup.lux.cleanup.stdout.log
lux_logs/latest_run/sync_startup_cleanup.lux.event.log
lux_logs/latest_run/sync_startup.lux.config.log
lux_logs/latest_run/sync_startup_cleanup.lux.hawk.stdout.log
lux_logs/latest_run/sync_startup.lux.server-log.stdin.log
lux_logs/latest_run/a_simple_server.lux.event.log.html
lux_logs/latest_run/async_startup_fail.lux.server.stdout.log
lux_logs/latest_run/sync_startup.lux.server.stdin.log
lux_logs/latest_run/a_simple_server.lux.orig
lux_logs/latest_run/async_startup_fail.lux.event.log
lux_logs/latest_run/sync_startup_cleanup.lux.orig
lux_logs/latest_run/sync_startup_cleanup.lux.hawk.stdin.log
lux_logs/latest_run/sync_startup_cleanup.lux.server.stdin.log
lux_logs/latest_run/sync_startup_cleanup.lux.server-log.stdin.log
lux_logs/latest_run/a_simple_server.lux.event.log.csv
lux_logs/latest_run/sync_startup_cleanup.lux.config.log
lux_logs/latest_run/Users
lux_logs/latest_run/Users/hmattsso
lux_logs/latest_run/Users/hmattsso/dev
lux_logs/latest_run/Users/hmattsso/dev/lux
lux_logs/latest_run/Users/hmattsso/dev/lux/tutorial
lux_logs/latest_run/Users/hmattsso/dev/lux/tutorial/support
lux_logs/latest_run/Users/hmattsso/dev/lux/tutorial/support/luxinc
lux_logs/latest_run/Users/hmattsso/dev/lux/tutorial/support/luxinc/macros.luxinc.orig
lux_logs/latest_run/lux_config.log
lux_logs/latest_run/async_startup_fail.lux.server.stdin.log
lux_logs/latest_run/sync_startup.lux.cons.stdout.log
lux_logs/latest_run/sync_startup.lux.server-log.stdout.log
lux_logs/latest_run/sync_startup_cleanup.lux.event.log.html
lux_logs/latest_run/lux_result.log
lux_logs/latest_run/async_startup_fail.lux.orig
lux_logs/latest_run/lux_summary.log
lux_logs/latest_run/sync_startup.lux.server.stdout.log
lux_logs/latest_run/a_simple_server.lux.config.log
lux_logs/latest_run/sync_startup.lux.event.log.csv
lux_logs/latest_run/sync_startup_cleanup.lux.cons.stdout.log
lux_logs/latest_run/a_simple_server.lux.event.log
lux_logs/latest_run/a_simple_server.lux.server.stdin.log
lux_logs/latest_run/lux.tap
lux_logs/latest_run/sync_startup.lux.cons.stdin.log
.../lux&gt; echo $?
0
</code></pre>
</blockquote>

<p>Some logs are common for all test cases in a test suite:</p>

<ul>
<li>Summary log - a summary of the outcome of the test suite</li>
<li>Config log - actual configuration for the run</li>
<li>Annotated summary log (HTML) - pretty printed asummary log</li>
</ul>

<p>while others are per test case:</p>

<ul>
<li>Event log - a trace of internal lux events</li>
<li>Extra logs - user defined logs/files worth to save after the run</li>
<li>Config log - test case specific configuration</li>
<li>Statistics - low level info about actual duration of timers</li>
<li>TAP log - summary log on TAP format</li>
<li>JUnit log - summary log on JUnit format</li>
<li>Annotated event log (HTML) - pretty printed event log with links to other logs</li>
</ul>

<p>and yet some are per shell in the test case:</p>

<ul>
<li>Shell stdin log(s) - bytes sent to stdin of the shell</li>
<li>Shell stdout log(s) - bytes received from stdout (and stderr) of the shell</li>
</ul>

<h2>Debugging</h2>

<p>There are various ways of debugging test cases. The simplest way is to
use the <code>--progress=verbose</code> flag or <code>-v</code> for short:</p>

<blockquote>
<pre><code>lux -v a_simple_server.lux
</code></pre>
</blockquote>

<p>Evaluate <code>cd tutorial/chatty/test/intro &amp;&amp; lux -v a_simple_server.lux</code></p>

<blockquote>
<pre><code>.../lux&gt; cd tutorial/chatty/test/intro &amp;&amp; lux -v a_simple_server.lux
summary log       : /Users/hmattsso/dev/lux/tutorial/chatty/test/intro/lux_logs/run_2022_06_27_19_41_21_142972/lux_summary.log
test case         : a_simple_server.lux
event log         : 0.8
/Users/hmattsso/dev/lux/tutorial/chatty/test/intro/a_simple_server.lux
21:41:21.276605 lux(0): start_time "2022-06-27 21:41:21.229784"
21:41:21.276924 lux(0): suite_timeout infinity
21:41:21.277050 lux(0): case_timeout 300000000 micros left (300 seconds * 1.000 multiplier)
21:41:21.279648 lux(1): doc "Demo a simple single shell test case"
21:41:21.284303 server(4): start "/Users/hmattsso/dev/lux/priv/bin/runpty /bin/sh -i"
21:41:21.285051 server(4): expected* ".+"
21:41:21.285051 server(4): timer started (10 seconds * 1.000 multiplier)
21:41:21.295985 server(4): recv "\e[?1034hsh-3.2$ "
21:41:21.296145 server(4): timer canceled (after 10925 microseconds)
21:41:21.296239 server(4): match "\e[?1034hsh-3.2$ "
21:41:21.296239 server(4): rest ""
21:41:21.296511 server(4): send "export PS1=SH-PROMPT:
  "
21:41:21.296715 server(4): expected* "^SH-PROMPT:"
21:41:21.296715 server(4): timer started (10 seconds * 1.000 multiplier)
21:41:21.296907 server(4): recv "export PS1"
21:41:21.297045 server(4): recv "=SH-PROM"
21:41:21.297188 server(4): recv "PT:
  "
21:41:21.297317 server(4): recv "SH-PROMPT:"
21:41:21.297423 server(4): timer canceled (after 571 microseconds)
21:41:21.297501 server(4): skip "export PS1=SH-PROMPT:
  "
21:41:21.297501 server(4): match "SH-PROMPT:"
21:41:21.297501 server(4): rest ""
21:41:21.297771 server(6): send "erl -sname server -pa ../../../chatty/ebin
  "
21:41:21.297990 server(6): recv "e"
21:41:21.298092 server(6): recv "rl"
21:41:21.298191 server(8): expected* "Erlang/OTP"
21:41:21.298191 server(8): timer started (10 seconds * 1.000 multiplier)
21:41:21.298331 server(8): recv " -sname server -pa "
21:41:21.298438 server(8): recv "../../"
21:41:21.298523 server(8): recv "../ch"
21:41:21.298593 server(8): recv "atty"
21:41:21.298685 server(8): recv "/ebin"
21:41:21.298769 server(8): recv "
  "
21:41:21.574667 server(8): recv "Erlang/OTP 24 [erts-12.3.1] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [jit] [dtrace]

  "
21:41:21.574789 server(8): timer canceled (after 276496 microseconds)
21:41:21.574870 server(8): skip "erl -sname server -pa ../../../chatty/ebin
  "
21:41:21.574870 server(8): match "Erlang/OTP"
21:41:21.574870 server(8): rest " 24 [erts-12.3.1] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [jit] [dtrace]

  "
21:41:21.575228 server(9): expected* "Eshell"
21:41:21.575228 server(9): timer started (10 seconds * 1.000 multiplier)
21:41:21.696138 server(9): recv "Eshell V12.3.1  (abort with ^G)
  (server@HMATTSSO-M-74JD)1&gt; "
21:41:21.696316 server(9): timer canceled (after 120975 microseconds)
21:41:21.696480 server(9): skip " 24 [erts-12.3.1] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [jit] [dtrace]

  "
21:41:21.696480 server(9): match "Eshell"
21:41:21.696480 server(9): rest " V12.3.1  (abort with ^G)
  (server@HMATTSSO-M-74JD)1&gt; "
21:41:21.696850 server(10): expected* "&gt; "
21:41:21.696850 server(10): timer started (10 seconds * 1.000 multiplier)
21:41:21.697041 server(10): timer canceled (after 8 microseconds)
21:41:21.697146 server(10): skip " V12.3.1  (abort with ^G)
  (server@HMATTSSO-M-74JD)1"
21:41:21.697146 server(10): match "&gt; "
21:41:21.697146 server(10): rest ""
21:41:21.697509 server(12): send "chatty:server().
  "
21:41:21.697713 server(13): expected* "Starting server"
21:41:21.697713 server(13): timer started (10 seconds * 1.000 multiplier)
21:41:21.719395 server(13): recv "cha"
21:41:21.719581 server(13): recv "tty:server().
  "
21:41:21.725196 server(13): recv "Starting server server...
  "
21:41:21.725304 server(13): timer canceled (after 27456 microseconds)
21:41:21.725423 server(13): skip "chatty:server().
  "
21:41:21.725423 server(13): match "Starting server"
21:41:21.725423 server(13): rest " server...
  "
21:41:21.725737 server(14): expected* "&gt; "
21:41:21.725737 server(14): timer started (10 seconds * 1.000 multiplier)
21:41:24.726659 server(14): recv "Trying to open log file chatty_server.log..."
21:41:24.731218 server(14): recv "ok.
  "
21:41:24.731442 server(14): recv "&lt;0.87.0&gt;
  (server@HMATTSSO-M-74JD)2&gt; "
21:41:24.731635 server(14): timer canceled (after 3005772 microseconds)
21:41:24.731790 server(14): skip " server...
  Trying to open log file chatty_server.log...ok.
  &lt;0.87.0&gt;
  (server@HMATTSSO-M-74JD)2"
21:41:24.731790 server(14): match "&gt; "
21:41:24.731790 server(14): rest ""
21:41:24.732272 server(16): send "halt(3).
  "
21:41:24.732629 server(17): expected* "SH-PROMPT:"
21:41:24.732629 server(17): timer started (10 seconds * 1.000 multiplier)
21:41:24.732921 server(17): recv "halt(3).
  "
21:41:24.737803 server(17): recv "SH-PROMPT:"
21:41:24.737974 server(17): timer canceled (after 5137 microseconds)
21:41:24.738088 server(17): skip "halt(3).
  "
21:41:24.738088 server(17): match "SH-PROMPT:"
21:41:24.738088 server(17): rest ""
21:41:24.738467 server(19): send "echo "===$?==="
  "
21:41:24.738768 server(19): recv "ec"
21:41:24.738987 server(19): recv "ho "===$?==="
21:41:24.739133 server(20): expected* "===3==="
21:41:24.739133 server(20): timer started (10 seconds * 1.000 multiplier)
21:41:24.739344 server(20): recv ""
  ===3===
  SH-PROMPT:"
21:41:24.739453 server(20): timer canceled (after 160 microseconds)
21:41:24.739541 server(20): skip "echo "===$?==="
  "
21:41:24.739541 server(20): match "===3==="
21:41:24.739541 server(20): rest "
  SH-PROMPT:"
21:41:24.739753 server(21): expected* "SH-PROMPT:"
21:41:24.739753 server(21): timer started (10 seconds * 1.000 multiplier)
21:41:24.739865 server(21): timer canceled (after 8 microseconds)
21:41:24.739939 server(21): skip "
  "
21:41:24.739939 server(21): match "SH-PROMPT:"
21:41:24.739939 server(21): rest ""
21:41:24.740129 server(22): no_cleanup
21:41:24.750479 server(22): inactivate after zombify
21:41:24.761906 server(22): stop shutdown
21:41:24.761906 server(22): where "22"
21:41:24.761906 server(22): stack "a_simple_server.lux:22" no_cleanup 
21:41:24.762584 lux(0): case_timeout 296514000 micros left (296 seconds * 1.000 multiplier)
21:41:24.762834 lux(0): suite_timeout infinity
21:41:24.763034 lux(0): end_time "2022-06-27 21:41:24.763022"
result            : SUCCESS
successful        : 1
summary           : SUCCESS
file:///Users/hmattsso/dev/lux/tutorial/chatty/test/intro/lux_logs/run_2022_06_27_19_41_21_142972/lux_summary.log.html
.../lux&gt; echo $?
0
</code></pre>
</blockquote>

<p>The shell stdin log is also quite useful when trying to reproduce a
run of a test case.</p>

<ul>
<li>Start multiple terminalks and create shells manually</li>
<li>Copy and paste from stdin logs to the shells</li>
</ul>

<p>Evaluate <code>cd tutorial/chatty/test/intro &amp;&amp; cat lux_logs/latest_run/a_simple_server.lux.server.stdin.log</code></p>

<blockquote>
<pre><code>.../lux&gt; cd tutorial/chatty/test/intro &amp;&amp; cat lux_logs/latest_run/a_simple_server.lux.server.stdin.log
export PS1=SH-PROMPT:
erl -sname server -pa ../../../chatty/ebin
chatty:server().
halt(3).
echo "===$?==="
.../lux&gt; echo $?
0
</code></pre>
</blockquote>

<p>Evaluate <code>cd tutorial/chatty/test/intro &amp;&amp; cat lux_logs/latest_run/a_simple_server.lux.server.stdout.log</code></p>

<blockquote>
<pre><code>.../lux&gt; cd tutorial/chatty/test/intro &amp;&amp; cat lux_logs/latest_run/a_simple_server.lux.server.stdout.log
[?1034hsh-3.2$ export PS1=SH-PROMPT:
SH-PROMPT:erl -sname server -pa ../../../chatty/ebin
Erlang/OTP 24 [erts-12.3.1] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [jit] [dtrace]

Eshell V12.3.1  (abort with ^G)
(server@HMATTSSO-M-74JD)1&gt; chatty:server().
Starting server server...
Trying to open log file chatty_server.log...ok.
&lt;0.87.0&gt;
(server@HMATTSSO-M-74JD)2&gt; halt(3).
SH-PROMPT:echo "===$?==="
===3===
.../lux&gt; echo $?
SH-PROMPT:0
</code></pre>
</blockquote>

<p>Lux has a built-in debugger. It is always present, ready to read
commands from stdin. For example the command <code>tail</code> or <code>t</code> for short
can be used to interactively display the various logs while the test
is running. By default it displays the last 10 lines of the event
log. So when the test program is waiting for output you can use <code>tail</code>
to see what is going on.</p>

<blockquote>
<pre><code>lux --timeout=30000 async_startup.lux
t
</code></pre>
</blockquote>

<p>Just pressing enter without any command will simply repeat the
previous command. Some commands behaves slightly different when they
are repeated. The <code>tail</code> command for example displays more and more
for each time. It displays 10 lines, 20, lines, 30 lines, ...</p>

<p>The test script may also be attached before line 1 with <code>--debug</code> or
<code>-d</code> for short:</p>

<blockquote>
<pre><code>lux -d a_simple_server.lux
</code></pre>
</blockquote>

<p>There you can explore the available commands using the built-in <code>help</code>
command. Try these commands out and see what happens:</p>

<p>Evaluate <code>cd tutorial/chatty/test/intro &amp;&amp; lux a_simple_server.delux</code></p>

<p>Snippet from the enclosed <code>.../lux/tutorial/chatty/test/intro/lux_logs/latest_run/a_simple_server.delux.debug.stdin.log</code> file:</p>

<blockquote>
<pre><code>export PS1=SH-PROMPT:
rm -rf tmp_logs
lux -d --log_dir=tmp_logs a_simple_server.lux
c 15
shell server
!im().
?
n
t
help quit
c
</code></pre>
</blockquote>

<p>Snippet from the enclosed <code>.../lux/tutorial/chatty/test/intro/lux_logs/latest_run/a_simple_server.delux.debug.stdout.log</code> file:</p>

<blockquote>
<pre><code>[?1034hsh-3.2$ export PS1=SH-PROMPT:
SH-PROMPT:rm -rf tmp_logs
SH-PROMPT:lux -d --log_dir=tmp_logs a_simple_server.lux
summary log       : /Users/hmattsso/dev/lux/tutorial/chatty/test/intro/tmp_logs/lux_summary.log

test case         : a_simple_server.lux
progress          : 
Break at "a_simple_server.lux:1"

File a_simple_server.lux:
1&gt; [doc Demo a simple single shell test case]
2: 
3: # Start a shell
4: [shell server]
5:     # Send text to the active shell
6:     !erl -sname server -pa ../../../chatty/ebin
7:     # Match output from the active shell
8:     ?Erlang/OTP
9:     ?Eshell
10:     ?&gt; 

Debugger for lux. Try help or continue.
c 15

Set temporary breakpoint at "a_simple_server.lux:15"

Continue to run from "a_simple_server.lux:1"
..:..:.:..:..:.:..:.:.:..:.:....:.:.:..:.:.:.
Break at "a_simple_server.lux:15"

File a_simple_server.lux:
13:     ?Starting server
14:     ?&gt; 
15&gt; 
16:     !halt(3).
17:     ?SH-PROMPT:
18: 
19:     !echo "===$?==="
20:     ?===3===
21:     ?SH-PROMPT:
22: 
shell server
22: 

Connect to shell "server" in background mode.
!im().

Send data to shell "server".

server(recv): im().
server(recv): 
?

Reset output buffer for shell "server".
n

File a_simple_server.lux:
16&gt;     !halt(3).
t
Log files at tmp_logs/.:

*  1 lux_config.log
*  2 lux_summary.log.tmp
*  3 lux_result.log
*  4 a_simple_server.lux.config.log
*  5 a_simple_server.lux.event.log
*  6 a_simple_server.lux.server.stdin.log
*  7 a_simple_server.lux.server.stdout.log

Last 10 (65..74) lines of log file: a_simple_server.lux.event.log

21:41:29.433328 server(14): recv "ok.\r\n&lt;0.87.0&gt;\r\n"
21:41:29.433638 server(14): recv "(server@HMATTSSO-M-74JD)2&gt; "
21:41:29.433835 server(14): timer canceled (after 3004863 microseconds)
21:41:29.433956 server(14): skip " server...\r\nTrying to open log file chatty_server.log...ok.\r\n&lt;0.87.0&gt;\r\n(server@HMATTSSO-M-74JD)2"
21:41:29.433956 server(14): match "&gt; "
21:41:29.433956 server(14): rest ""
21:41:29.441252 server(14): send "im().\n"
21:41:29.942268 server(14): recv "im().\r\n"
21:41:29.954063 server(14): reset "7 bytes wasted"
21:41:29.954063 server(14): output reset 7 bytes

help quit

quit \[scope\]
--------------

Quit a single test case or the entire test suite
in a controlled manner. Runs cleanup if applicable.

**Parameters:**  

* scope - scope of exit; enum(case|suite)  

c

Continue to run from "a_simple_server.lux:16"

server(recv): &lt;0.90.0&gt;
server(recv): 

server(recv): (server@HMATTSSO-M-74JD)3&gt; halt(3).
server(recv): 

server(recv): SH-PROMPT:

server(recv): e

server(recv): cho "===$?==="
server(recv): ===3===
server(recv): SH-PROMPT:
.
Cleanup. Turn existing shells into zombies.

Disconnect from shell "server".
.
result            : SUCCESS

successful        : 1
summary           : SUCCESS

file:///Users/hmattsso/dev/lux/tutorial/chatty/test/intro/tmp_logs/lux_summary.log.html
SH-PROMPT:
</code></pre>
</blockquote>

<h2>Infra-structure support</h2>

<p>In a hetrogenous test environment with various types of machines
possibly with different architectures and hardware, it may be
necessary to have machine dependent configuration settings. This can
be achieved by using <code>.luxcfg</code> files. Look in the
<code>lux_logs/latest_run/lux_config.log</code> file to figure out the
architecture and name your <code>.luxcfg</code> file accordingly. It is also
possible to have a host specific configuration file or rely on the
default configuration in the file named <code>luxcfg</code>.</p>

<p>Typical things that may vary from machine to machine is shell settings
and test cases which only should be run on certain architectures. If
some machine is very slow the <code>multiplier</code> can be set to something
else than 1000 which is the default. The match timeout (in seconds) is
multiplied with this setting to compute the actual timeout to get
milliseconds which is used internally.</p>

<p>Here you can find a couple of architecture specific examples:</p>

<p>Evaluate <code>cd tutorial &amp;&amp; find support/luxcfg</code></p>

<blockquote>
<pre><code>.../lux&gt; cd tutorial &amp;&amp; find support/luxcfg
support/luxcfg
support/luxcfg/luxcfg
support/luxcfg/NetBSD-macppc.luxcfg
support/luxcfg/SunOS-i86pc.luxcfg
.../lux&gt; echo $?
0
</code></pre>
</blockquote>

<p>Here are some examples of how test cases can be skipped or marked as
unstable when architecture or host specific variables are set (or not
set).</p>

<p>Snippet from the enclosed <code>.../lux/tutorial/chatty/test/infra/skip.lux</code> file:</p>

<blockquote>
<pre><code>[doc Demonstrate a skipped test]

[config skip=PATH]
</code></pre>
</blockquote>

<p>Snippet from the enclosed <code>.../lux/tutorial/chatty/test/infra/unstable.lux</code> file:</p>

<blockquote>
<pre><code>[doc Demonstrate an unstable test which is run but do not clutter the results]

[config unstable_unless=TEST_DEVELOP]

[shell date]
  # Ensure a quick fail if it fails
  [timeout 2]
  -[2-4]
  !date +%S
  ?SH-PROMPT
</code></pre>
</blockquote>

<p>For more complex test cases there may be a need to have a build step
before running the test case(s). One way of solving this is to use
<code>lux --mode=list_dir</code> to find the directories which contain <code>.lux</code>
files, and simply run make on those directories. A simple example of
this can be found in this makefile:</p>

<p>Snippet from the enclosed <code>.../lux/tutorial/chatty/test/Makefile</code> file:</p>

<blockquote>
<pre><code>LUXDIRS=$(filter-out .,$(shell lux --mode=list_dir *))

.PHONY: all build test history clean info

all build test history:
@for d in $(LUXDIRS); do \
   if test -f $$d/Makefile ; then \
      $(MAKE) -C $$d $@ || exit $?; \
   fi; \
done

clean:
rm -rf lux_logs lux_history* *~
@for d in $(LUXDIRS); do \
   if test -f $$d/Makefile ; then \
      (cd $$d &amp;&amp; $(MAKE) $@) ; \
   fi; \
done
info:
@echo "LUXDIRS=$(LUXDIRS)"
</code></pre>
</blockquote>

<p>History of test run results</p>

<p>Snippet from the enclosed <code>.../lux/tutorial/chatty/test/infra/Makefile</code> file:</p>

<blockquote>
<pre><code>.PHONY: all build test history clean info

TIMESTAMP=$(shell date +"%F_%T")
GITHASH=$(shell git rev-parse --verify --short HEAD)
TOPDIR=$(shell pwd | sed -e 's/tutorial.*/tutorial/')

all: build test

build:

test:
lux .

history:
lux --history . lux_logs
@echo
@echo open lux_history.html

clean:
rm -rf lux_logs history_demo*_logs erl_crash.dump *~

info:
@echo "TOPDIR=$(TOPDIR)"
@echo "TIMESTAMP=$(TIMESTAMP)"
@echo "GITHASH=$(GITHASH)"

############################################################
# Internal history demo targets
############################################################

.PHONY: history_demo history_demo_single history_demo_multi_host history_demo_multi_branch history_demo_success history_demo_warning history_demo_empty

history_demo: history_demo_single history_demo_multi_host history_demo_multi_branch history_demo_success history_demo_warning history_demo_empty
ls -1 history_demo_*/lux_history.html | sed -e 's/^/open /g'

history_demo_single:
rm -rf ${@}; \
for i in 1 2 3 4 5 6 7 8 9; do \
  opts="--revision=$(TIMESTAMP)_$$i_$(GITHASH) --suite=demo --config_dir=$(TOPDIR)/support/luxcfg"; \
  lux $$opts --hostname=sunny  --config_name=SunOS-i86pc  --log_dir=${@}/run_logs/run_sunny_$$i .; \
done; \
lux --history ${@} ${@}/run_logs

history_demo_multi_host:
rm -rf ${@}; \
for i in 1 2 3 4 5 6 7 8 9; do \
  opts="--revision=$(TIMESTAMP)_$$i_$(GITHASH) --suite=demo --config_dir=$(TOPDIR)/support/luxcfg"; \
  lux $$opts --hostname=sunny  --config_name=SunOS-i86pc   --log_dir=${@}/run_logs/run_sunny_$$i  .; \
  lux $$opts --hostname=cloudy --config_name=SunOS-i86pc   --log_dir=${@}/run_logs/run_cloudy_$$i .; \
  lux $$opts --hostname=netty  --config_name=NetBSD-macppc --log_dir=${@}/run_logs/run_netty_$$i  .; \
done; \
lux --history ${@} ${@}/run_logs

history_demo_multi_branch:
rm -rf ${@}; \
branches="chatty-1.0 chatty-2.0"; \
histargs=${@}; \
for b in $$branches; do \
  for i in 1 2 3 4 5 6 7 8 9; do \
    opts="--revision=$(TIMESTAMP)_$$i_$(GITHASH) --suite=demo --config_dir=$(TOPDIR)/support/luxcfg"; \
    lux $$opts --hostname=sunny  --config_name=SunOS-i86pc    --log_dir=${@}/run_logs/$$b/run_sunny_$$i  .; \
    lux $$opts --hostname=cloudy --config_name=SunOS-i86pc    --log_dir=${@}/run_logs/$$b/run_cloudy_$$i .; \
    lux $$opts --hostname=netty  --config_name=NetBSD-macppc  --log_dir=${@}/run_logs/$$b/run_netty_$$i  .; \
  done; \
  histargs="$$histargs $$b:::${@}/run_logs/$$b"; \
done; \
lux --history $$histargs

history_demo_success:
rm -rf ${@}; \
for i in 1 2 3 4 5 6 7 8 9; do \
  opts="--revision=$(TIMESTAMP)_$$i_$(GITHASH) --suite=demo --config_dir=$(TOPDIR)/support/luxcfg"; \
  lux $$opts --hostname=sunny  --config_name=SunOS-i86pc  --log_dir=${@}/run_logs/run_sunny_$$i success.lux; \
done; \
lux --history ${@} ${@}/run_logs

history_demo_warning:
rm -rf ${@}; \
for i in 1 2 3 4 5 6 7 8 9; do \
  opts="--revision=$(TIMESTAMP)_$$i_$(GITHASH) --suite=demo --config_dir=$(TOPDIR)/support/luxcfg"; \
  lux $$opts --hostname=sunny  --config_name=SunOS-i86pc  --log_dir=${@}/run_logs/run_sunny_$$i success.lux warning.lux; \
done; \
lux --history ${@} ${@}/run_logs

history_demo_empty:
rm -rf ${@}; \
lux --history ${@} ${@}/run_logs
</code></pre>
</blockquote>

<p>Evaluate <code>cd tutorial/chatty/test/infra &amp;&amp; make history_demo_multi_host</code></p>

<p>Evaluate <code>cd tutorial/chatty/test/infra &amp;&amp; rm -f history_demo_multi_host/lux_history*</code></p>

<p>Evaluate <code>cd tutorial/chatty/test/infra &amp;&amp; lux --history history_demo_multi_host history_demo_multi_host/run_logs</code></p>

<blockquote>
<pre><code>.../lux&gt; cd tutorial/chatty/test/infra &amp;&amp; lux --history history_demo_multi_host history_demo_multi_host/run_logs
Cwd: /Users/hmattsso/dev/lux/tutorial/chatty/test/infra
Invoke: /Users/hmattsso/dev/lux/bin/lux --history history_demo_multi_host history_demo_multi_host/run_logs
Assembling history of logs from...
history_demo_multi_host/run_logs ...........................
Wrote 2499 bytes in run cache to file history_demo_multi_host/lux_history.cache
Analyzed 27 test runs with 135 test cases (0 errors)...ok
file:///Users/hmattsso/dev/lux/tutorial/chatty/test/infra/history_demo_multi_host/lux_history.html
.../lux&gt; echo $?
0
</code></pre>
</blockquote>

<p>Walkthru history<em>demo</em>multi<em>host/lux</em>history.html</p>

<ul>
<li>Overview
<ul>
<li>Per architecture (config)</li>
<li>Per host</li>
<li>Still failing test cases</li>
</ul></li>
</ul>

<p>Evaluate <code>cd tutorial/chatty/test/infra &amp;&amp; make history_demo_multi_branch</code></p>

<p>Walkthru history<em>demo</em>multi<em>branch/lux</em>history.html</p>

<pre><code>- Compare branches
</code></pre>

<p>Jenkins</p>

<ul>
<li>Automated tests</li>
<li>Display Jenkins test results as LUX history for non-LUX tests</li>
</ul>

<h2>More concepts</h2>

<ul>
<li>Fail pattern</li>
<li>Loops
<ul>
<li>Foreach</li>
<li>Break pattern</li>
</ul></li>
<li>Macros</li>
<li>Variable scope
<ul>
<li>Environment
<ul>
<li>Initial values</li>
<li>Require</li>
<li>Local within one shell</li>
<li>Global for all shells</li>
<li>Statement block (my)</li>
<li>Sub expression</li>
</ul></li>
</ul></li>
<li>Regexp match vs verbatim match</li>
<li>Match on permutations</li>
<li>Shell config
<ul>
<li>Pseudo terminal (PTY)</li>
<li>Normalized prompt</li>
<li>Using other types of shells</li>
</ul></li>
<li>Use the power of various interactive languages</li>
<li>Using LUX as an all purpose scripting language</li>
</ul>

<h2>Implementation</h2>

<p>Why is Erlang a good fit? Primary due to its</p>

<ul>
<li>Concurrency</li>
<li>Port programs</li>
<li>Built-in regular expresssions (re)</li>
<li>Timers</li>
</ul>

<p>Lux is written as an escript which can be installed as stand-alone
(including the Erlang runtime). Reltool is used for this.</p>

<p>The test cases in a suite are executed in sequence where a new
interpreter process is started for each test script. The script is
interpreted statement for statement.</p>

<p>When a new Lux shell is to be started a new process is spawned. That
process runs the Bourne shell as a port program and acts as a man in
the middle between the interpreter and the port program.</p>

<p>In fact it is not that simple. To make the Bourne shell believe it is
executed in an interactive terminal there is actually one more man in
the middle. The <code>runpty</code> is a small C program which manipulates the
terminal settings for the pseudo TTY. When it has done that and setup
sockets between the parent and child process it will fork the Bourne
shell.</p>

<p>Input data strings from the script is sent as is to the stdin of the
port. The terminal is setup to echo the input to stdout.</p>

<p>The stderr is redirected to stdout. The terminal will normalise the
output from the Bourne shell (stdout and stderr) to make each end of
line a carriage return followed by a line feed. The output from the
port is buffered.</p>

<p>When the script expects a regexp to match the buffer a timer is
started. And the buffer is matched against the regexp when the buffer
is updated. If the buffer does not match the regexp when the timer
times out the script will fail.</p>

<p>If the test script has a cleanup section, the cleanup is run as yet
another Lux shell.</p>

<h2>Lessons learned</h2>

<ul>
<li>Expect like testing requires a different mindset (find sync points in streams of data)</li>
<li>Testability is a vital property of products, observability</li>
<li>Effective post mortem analysis of test runs is a big time saver</li>
<li>Test cases (as well as test tools) does also needs to be debugged</li>
</ul>

<h2>More info</h2>

<ul>
<li><p>Download from https://github.com/hawk/lux (Apache license)</p></li>
<li><p>See the file <strong>../lux.html</strong> for the full documentation or view it online
on <a href="https://github.com/hawk/lux/blob/euc/doc/lux.md">GitHub</a>.</p></li>
</ul>

<h2>Maintenance of LUX itself</h2>

<ul>
<li>Run LUX in Erlang debugger</li>
<li>Use Erlang trace
<ul>
<li>Interactive display</li>
<li>Display filtered Erlang trace</li>
</ul></li>
<li>Use Event Tracer</li>
<li>Use xref</li>
<li>Use reltool</li>
<li>Install as stand-alone incl Erlang runtime</li>
<li>Documentation
<ul>
<li>Markdown</li>
<li>Generated from example runs</li>
<li>Generated from built-in debugger help</li>
<li>Generated from .md.src files</li>
</ul></li>
<li>Test of LUX itself</li>
</ul>

<h2>Tail-f/Cisco</h2>

<ul>
<li>Widely used for testing of Tail-f products</li>
<li>Automated test environment using Jenkins
<ul>
<li>~4500 Lux test cases per run</li>
<li>distributed over ~150 Docker containers</li>
</ul></li>
<li>ConfD
<ul>
<li>Device configuration</li>
<li>Model driven configuration management framework for a network element</li>
<li>Render northbound interfaces such as CLI, Netconf, SNMP, Rest, RestConf</li>
<li>Tracable internal interfaces</li>
</ul></li>
<li>NSO
<ul>
<li>Orchestrator for a massive number of (hetrogenous) network elements</li>
<li>Same standardised northbound interfaces as Confd</li>
<li>Standard interfaces southbound combined with</li>
<li>100+ adaptors for network elements lacking standard interfaces
<a name="script_syntax"/></li>
</ul></li>
</ul>

<h1>Script syntax</h1>

<p>The Lux script syntax is as follows. The <strong>first non whitespace</strong>
character on each line determines how it will be processed. Lines
beginning with a <code>#</code> are comments. It is recommended to use
indentation and comments to make the scripts more readable. The <strong>Lux
mode for <a href="http://www.gnu.org/software/emacs" title="Emacs text editor">Emacs</a></strong> (<code>lux/emacs/lux-mode.el</code>) is quite useful as it
simplifies the indentation and makes scripts more easy to read with
coloring for different types of language constructs.</p>

<p>Lines beginning with <code>"""Char</code> are <strong>multi-line quotes</strong>. The quote
ends with the next line beginning with <code>"""</code>. The opening quote and
closing quote must be in the same column of the script. The char right
after the first <code>"""</code> determines how the multi-line quote will be
interpreted. The char is interpreted as a statement just like any of
the single line statement characters (so it can be e.g. <code>?</code>, <code>!</code>, <code>~</code>,
<code>#</code>, <code>-</code>, etc).</p>

<p>When multi-line quotes are indented the leading whitespaces are
stripped from the quoted lines, up to but not including the column
of the double quote character, or to the first non-whitespace
character, whichever occurs first. In this process, a tab character
is treated as 8 space characters.</p>

<p>A backslash at end of line implies line continuation and not a
newline. This is syntactic sugar which makes it possible to split
a long line into several shorter ones. Leading whitespaces on the
following line are ignored. If the intention is to keep the backslash
at the end of the line, this can be achieved with two backslashes.</p>

<h2>Interacting with a shell</h2>

<p><strong>#String</strong> <br />
Inline style comment. The <code>#</code> must be the first non-whitespace
character on the line.</p>

<p><strong>!String</strong>  </p>

<p>A <code>send</code> operation. Sends a <code>String</code> to the <code>stdin</code> of the active
shell. Adds a <code>LF</code> (line feed) at the end of the string. <code>String</code>
may contain references to variables using <code>$Var</code> or <code>${Var}</code>.</p>

<p><strong>~String</strong> <br />
Same as <code>!String</code>, but it does NOT add a <code>LF</code> at the end.</p>

<p><strong>???Verbatim</strong> <br />
An <code>expect</code> operation which waits for a given sequence of characters
to appear on the shell output (either <code>stdout</code> or <code>stderr</code>). All
characters in the <code>Verbatim</code> string are matched literally. This means
that even characters like <code>\</code> (backslash), <code>$</code> (dollar) etc. are
matched explicitly.</p>

<p>If no matching output does appear within the timeout period, the test
case is considered as failed. See the <code>--timeout</code> option. See also the
<code>--flush_timeout</code> and <code>--poll_timeout</code> configuration parameters about
customizing the <code>?</code> behavior.</p>

<p><strong>??Template</strong> <br />
Like <code>?Verbatim</code>, but variables are also substituted.</p>

<p><strong>?Regexp</strong> <br />
Like <code>??Template</code>, but matches a <a href="http://www.erlang.org/doc/man/re.html#regexp_syntax" title="Erlang style regular expressions (re)">regular expression</a> after the
variable substitution. If the shell output is expected to contain a
regexp keyword, such as <code>^$.?+*()[]{}|</code>, the keyword must be escaped
with a backslash.</p>

<p><strong>?</strong> <br />
Flush the output streams (<code>stdout</code>, <code>stderr</code>). Already received output
is discarded. Avoid this (mis)feature. At a first look it seems more
useful than it is. It often causes unexpected race patterns.</p>

<p><strong>?+Regexp</strong> <br />
Like <code>?Regexp</code>, but has no immediate effect. It is used when the
order of the output is undeterministic. Assume a case where the
strings A, B and C occurs in the output but the order of them is
unknown. Then we need to match all permutations of the strings.
Such as ABC, ACB, BAC, BCA, CAB and CBA. It can be achieved by
the relatively simple regexp <code>?(ABC)|(ACB)|(BAC)|(BCA)|CAB)|(CBA)</code>.
But with larger regexps, possibly spanning multiple lines, it
can be quite complex to just write the regexps. Performing the
post mortem analysis to determine which sub-pattern that is
matching which part of the output will be even worse. In the
following example <code>?+</code> is used to register a sub-pattern and <code>?</code>
evaluates the permutations of all sub-patterns (including the one
specified with <code>?</code>).</p>

<pre><code>?+A
?+B
?C
</code></pre>

<p>will render matching of all permutatations of A, B and C. Note the
usage of <code>?</code>. <code>?+</code> is always used in conjunction with <code>?</code>. Never <code>??</code>
nor <code>???</code>. It is the <code>?</code> command which triggers the actual regexp
match.</p>

<p><strong>-</strong> <br />
<strong>-Regexp</strong> <br />
Sets the failure pattern for a shell to a regular expression (see
<a href="http://www.erlang.org/doc/man/re.html#regexp_syntax" title="Erlang style regular expressions (re)">regular expression</a>). It is typically used to match error
messages. If the given <code>Regexp</code> matches, the test case is considered
to have failed (no further processing of the script will be performed
besides the <code>cleanup</code>). If no <code>Regexp</code> is given, the failure pattern
is reset (cleared).</p>

<p>The failure pattern is primarily searched for when the script
explicitly is expecting some output. That is when a command like <code>?</code>,
<code>??</code> or <code>???</code> is evaluated. It is also searched for when a shell
cannot produce more output, for example when a shell exits or when
there are no more commands to evaluate.</p>

<p>See also the configuration parameter <code>--pattern_mode</code> and the command
<code>[pattern_mode]</code>.</p>

<p><strong>+</strong> <br />
<strong>+Regexp</strong> <br />
Sets the success pattern for a shell to a regular expression (see
<a href="http://www.erlang.org/doc/man/re.html#regexp_syntax" title="Erlang style regular expressions (re)">regular expression</a>). If the given <code>Regexp</code> matches, the test case
is considered to be a success (no further processing of the script
will be performed besides the <code>cleanup</code>). If no <code>Regexp</code> is given, the
success pattern is reset (cleared).</p>

<p>The success pattern is primarily searched for when the script
explicitly is expecting some output. That is when a command like <code>?</code>,
<code>??</code> or <code>???</code> is evaluated. It is also searched for when a shell
cannot produce more output, for example when a shell exits or when
there are no more commands to evaluate.</p>

<p>See also the configuration parameter <code>--pattern_mode</code> and the command
<code>[pattern_mode]</code>.</p>

<p><strong>@</strong> <br />
<strong>@Regexp</strong> <br />
Sets a loop break pattern for a shell to a regular expression (see
<a href="http://www.erlang.org/doc/man/re.html#regexp_syntax" title="Erlang style regular expressions (re)">regular expression</a>). This statement is only valid in loops. It
is typically used to match output from a poll like command which is
executed over and over again and after a while the command causes some
output that will match the break pattern.  When the given <code>Regexp</code>
matches, the loop (and all nested loops) is immediately exited and the
execution continues with the first statement after the loop.</p>

<p>The break pattern is only searched for when the script explicitly is
expecting some output. That is when a command like <code>?</code>, <code>??</code> or <code>???</code>
is evaluated. It may be a prompt or whatever, indicating that the poll
like command has produced all output that may match the break pattern.</p>

<p>A loop with a break pattern can only exit by a successful match of the
break pattern. If the loop exits anyway it will cause the test case to
fail. Unless the loop break pattern is reset (cleared).</p>

<p><strong>[endshell Regexp]</strong>
An <code>expect</code> operation like <code>?</code>, but it waits for the <code>stdout</code> stream
of the shell to be closed. This means the shell has terminated. The
<code>Regexp</code> matches the exit status from the shell, such as <code>[endshell
0]</code> or <code>[endshell .*]</code>. Be careful with using <code>[endshell .*]</code> as it
may cause crashing shells to not be noticed. Use [endshell 0] instead.</p>

<p>Note that the "exit" command in a Bourne shell may return a non-zero
status code if the latest executed command returned a non-zero. Use
"exit 0" if that is an issue.</p>

<h3>Meta statements</h3>

<p><strong>[</strong> <br />
Indicates the beginning of a meta statement. Meta statements are ended
on the same line with a <code>]</code>.</p>

<p><strong>[newshell Name]</strong> <br />
Creates a new shell named <code>Name</code>.</p>

<p>By default a <code>/bin/sh</code> shell (Bourne shell) is started. See
the <code>--shell_wrapper</code>, <code>--shell_cmd</code> and <code>--shell_arg</code> configuration
parameters. The current working directory of a newly started shell is
the same as the dirname of the script file. The <strong>environment
variable</strong> <code>LUX_SHELLNAME</code> is set to <code>Name</code>. The shell prompt variable
<code>PS1</code> is set to <code>SH-PROMPT:</code> and the first printout of the prompt is
automatically matched in an expect like manner in order to ensure that
the shell is ready for input. The <code>Name</code> may contain variables. Shell
names beginning with <code>lux</code>, <code>cleanup</code> and <code>post_cleanup</code> are reserved
for internal purposes. The <strong>environment variable</strong> <code>LUX_START_REASON</code>
is initially set to <code>normal</code>. See also <code>[cleanup]</code>.</p>

<p><strong>[shell]</strong> <br />
<strong>[shell Name]</strong> <br />
Switches to the named shell, to make it active. If <code>Name</code> is omitted,
the active shell is deactivated. This implies no shell to be activated.</p>

<p>If <code>--newshell</code> mode is not activated, the command may also be used
to create a new shell named <code>Name</code>. See the configuration parameter
<code>--newshell</code>.</p>

<p><strong>[cleanup]</strong> <br />
is the cleanup marker. If the script is prematurely aborted due to
a failure (or due to a matching success pattern) the remaining
statements in the file are normally skipped. But if the there is a
cleanup marker after the failing line (and this is the only
cleanup marker), the lines after the cleanup marker will also be
run in order to enable a controlled cleanup of leftovers. Such as
killing processes, removing files etc. When the cleanup marker is
evaluated, the running shells will be set into a non accessible mode
(<strong>zombie mode</strong>) and their failure and success patterns will be
reset (cleared). This means that output received by zombie shells
during the cleanup is not matched against failure or success
patterns. A brand new shell (called something beginning with
<code>cleanup</code>) will also be started. If the cleanup code causes a failure
the remaining statements (on that level) will be skipped.</p>

<p>The <strong>environment variable</strong> <code>LUX_START_REASON</code> is set to <code>normal</code>
in most shells, but if the cleanup is run due to premature failure or
premature success it will be set to <code>fail</code> or <code>success</code> respectively.
This can for example be used if you want to save the contents of
error logs, core dumps etc. in case of failure. Textual logs can
simply be written to <code>stdout</code> in order to be easily accessible in
the post mortem analysis. For the purpose of saving binary files
the <strong>environment variable</strong> <code>LUX_EXTRA_LOGS</code> may be used. It
refers to a log directory name unique for each test case. The
directory is however not automatically created. It must be created
by you in the test script if you want to use it. If you have created
the directory, it will turn up as a link in the annotated event log.</p>

<p><strong>[include FileName]</strong> <br />
Includes and runs the specified script at this point. The <code>FileName</code>
is relative to the currently executing script, unless given as an
absolute path. <code>.luxinc</code> is preferred as file extension. Variables in
<code>FileName</code> are expanded during parsing of the script, before execution
of the script.</p>

<p><strong>[macro MacroName ArgName1 ArgName2 ...]</strong> <br />
  ... <br />
<strong>[endmacro]</strong> <br />
Declare a macro. The body of the macro consists of all lines up to the
next <code>[endmacro]</code> line. The scope of the arguments are local within
the macro. The arguments can be accessed via their names as normal
variables, such as <code>$ArgName1</code>. <code>[my Var=Value]</code> can be used to assign
temporary variables only valid within the macro. If a macro switches
to another shell it is good practice to switch back to the calling
shell before the end of the macro. One way of doing this is to get the
name of the active shell from the <strong>environment variable</strong>
<code>LUX_SHELLNAME</code> with <code>[my old=$LUX_SHELLNAME]</code> and later switch back
to the shell with <code>[shell $old]</code>.</p>

<p><strong>[invoke MacroName ArgVal1 ArgVal ...]</strong> <br />
Invoke a macro. The arguments are separated with spaces. Arguments
can be quoted with the double quote (<code>"</code>) character. Double quotes
and backslashes (<code>\</code>) must be escaped with a backslash.</p>

<p><strong>[loop Var Item1 Item2 ...]</strong> <br />
  ... <br />
<strong>[endloop]</strong> <br />
Declare a loop. The body of the loop consists of all lines up to the
next <code>[endloop]</code> line. The commands within the loop are repeated for
each item. For each iteration the loop variable <code>Var</code> is set to the
value of the current <code>Item</code>. The scope of the loop variable is the
same as a macro variable (defined with <code>my</code>).</p>

<p>The <code>Item</code> list may contain variables and these are expanded before
the first iteration.  Items in the expanded list are separated with
spaces. For example <code>[loop color blue red green]</code> or
<code>[loop color blue $more]</code> where <code>more</code> is set to <code>"red green"</code>.</p>

<p>When iterating over a set of consecutive integers, such as
<code>[loop iter 4 5 6 7 8 9]</code>, this can be written as a range expression,
like <code>[loop iter 4..9]</code>. By default the increment is 1. A custom
increment can also be set with the construct <code>from..to..incr</code>, such as
<code>[loop iter 4..9..2]</code>. This would be the same as <code>[loop iter 4 6
8]</code>. <code>[loop iter 9..4..2]</code> would be the same as <code>[loop iter 9 7 5]</code>.</p>

<p>In the logs the iteration counter is represented as a negative line
number. For example "8:-2:10" would mean line 10 in the second loop
iteration where the loop starts at line 8.</p>

<h3>Variables</h3>

<p><strong>[local Var=Value]</strong> <br />
assigns a value to a variable that is local to the current
shell. <code>Value</code> may contain references to variables using <code>$Var</code>,
<code>${Var}</code> or <code>$N</code>, where <code>N</code> is an integer. <code>$N</code> refers to a captured
substring from the most recent <code>expect</code> operation. Subsequent <code>send</code>
operations may refer to this new variable in the same manner as
environment variables. In order to prevent variable substitutions and
keep a <code>$Var</code> string literally it must be escaped as <code>$$Var</code>. For
example this is needed when "true" environment variables needs to be
read. In order to read a variable like <code>$?</code> it must be written as
<code>$$?</code>.</p>

<p><strong>[global Var=Value]</strong> <br />
assigns a value to a global variable. Works like <code>[local]</code>, but the
variable setting is propagated to all shells. Global variables may be
set before even if no shell is active.</p>

<p><strong>[my Var=Value]</strong> <br />
assigns a value to a variable with a very limited scope. Works like
<code>[global]</code>, but can only be set and used in a macro or loop. The
variable setting is only valid within the macro that assigns the
variable.</p>

<h3>Multi-line values in variables</h3>

<pre><code>[local Var=Multi\nLine\nValue]
</code></pre>

<p>can be written as</p>

<pre><code>[local Var=
    """
    Multi
    Line
    Value
    """]
</code></pre>

<h3>Variables in variable names</h3>

<p>Variable names are also expanded and may themselves contain variables.</p>

<pre><code>[global v=a]
[local var_${v}=val]
</code></pre>

<p>Once set, the variables may be accessed using the same style</p>

<pre><code>!echo "${var_${v}}"
</code></pre>

<p>or as the ordinary variables they in fact are implemented as</p>

<pre><code>!echo "$var_a"
!echo "${var_a}"
</code></pre>

<p>Here is a small test program demonstrating the usage.</p>

<p>Snippet from the enclosed <code>.../lux/test/var_in_var.lux</code> file:</p>

<blockquote>
<pre><code>[doc Variables in variable names]

[shell var]
  [loop n a b c]
      !echo "var=val"
      ?^var=(.*)$
      [local var_${n}=$1]
      ?SH-PROMPT:
  [endloop]
  !echo "$var_a"
  ?^val$
  ?SH-PROMPT:
  !echo "${var_b}"
  ?^val$
  ?SH-PROMPT:
  [local v=c]
  !echo "${var_${v}}"
  ?^val$
  ?SH-PROMPT:
</code></pre>
</blockquote>

<p>Further, an associative arrays variable name style can be used. It is
merely some syntactic sugar and it can be combined variables in
variable names.</p>

<p>Variables using that style looks like this</p>

<pre><code>[local arr[b]=val]
!echo "${arr[b]}"
</code></pre>

<p>or if combined with variables in variable names</p>

<pre><code>[local v=a]
[local arr[${v}]=val]
!echo "${arr[${v}]}"
</code></pre>

<p>Here is a small test program demonstrating the usage.</p>

<p>Snippet from the enclosed <code>.../lux/test/assoc_array.lux</code> file:</p>

<blockquote>
<pre><code>[doc Associative arrays variable name style]

[shell arr]
  [loop n a b c]
      !echo "arr=val"
      ?^arr=(.*)$
      [local arr[${n}]=$1]
      ?SH-PROMPT:
  [endloop]
  !echo "${arr[a]}"
  ?^val$
  ?SH-PROMPT:
  !echo "${arr[b]}"
  ?^val$
  ?SH-PROMPT:
  !echo "${arr[c]}"
  ?^val$
  ?SH-PROMPT:
  [local v=a]
  !echo "${arr[${v}]}"
  ?^val$
  ?SH-PROMPT:
</code></pre>
</blockquote>

<h3>Built-in variables</h3>

<pre><code>_BS_        - backspace       (ASCII 8)
_TAB_       - horizontal tab  (ASCII 9)
_LF_        - line feed       (ASCII 10)
_CR_        - carriage return (ASCII 13)
 _ESC_      - escape          (ASCII 27)
_DEL_       - delete          (ASCII 127)
_CTRL_A_    - control a       (ASCII 1)
...
_CTRL_Z_    - control z       (ASCII 26)
_ASCII_0    - null            (ASCII 0)
...
_ASCII_127_ - delete          (ASCII 127)
N           - where N is an integer refering to a captured substring
</code></pre>

<h3>Built-in environment variables</h3>

<pre><code>LUX_SHELLNAME       - name of active Lux shell
LUX_START_REASON    - reason for starting a shell (normal|fail|success)
LUX_TIMEOUT         - value of match timeout in the active Lux shell
LUX_FAIL_PATTERN    - value of fail pattern in the active Lux shell
LUX_SUCCESS_PATTERN - value of success pattern in the active Lux shell
PS1                 - shell prompt variable set by Lux
</code></pre>

<h3>Miscellaneous statements</h3>

<p><strong>[doc String]</strong> <br />
<strong>[docN String]</strong> <br />
A test case slogan displayed in the summary log. It is also possible
to document parts of a test case by specifying a documentation level
<code>N</code>. In that case the doc statement should look like <code>[docN String]</code>
where <code>N</code> is an integer. <code>doc2</code> would mean that the documentation is on
level 2. Doc strings can be extracted from the scripts and written to
stdout with the<code>--mode=doc</code> and <code>--doc=N</code> command line options. It
gives a quick overview of the test cases and can be seen as a poor
mans test spec.</p>

<p>The first <code>[doc]</code> documentation string in a script is a bit special as
it is regarded as a one line summary of the script. With <code>--doc=0</code>
only the oneline summary lines are displayed.</p>

<p><strong>[doc]</strong> <br />
  ... <br />
<strong>[enddoc]</strong> <br />
Multi-line documentation, typically to be used first in the script.
The first line is regarded as a one line summary (on level 1) and
the remaining lines on next level (2). Third line must be preceded
by empty line.</p>

<blockquote>
<pre><code>[doc]
One line summary

Details
More details
Yet more details
[enddoc]
</code></pre>
</blockquote>

<p>would have been the same as</p>

<blockquote>
<pre><code>[doc One line summary]
[doc2 Details]
[doc2 More details]
[doc2 Yet more details]
</code></pre>
</blockquote>

<p><strong>[timeout]</strong> <br />
<strong>[timeout Seconds]</strong> <br />
The script expects the shell output to match given
<a href="http://www.erlang.org/doc/man/re.html#regexp_syntax" title="Erlang style regular expressions (re)">regular expression</a>s. But the output must be received within a
given time limit. The <code>timeout</code> command sets the timeout for the
current shell to the given number of seconds multiplied with a
configurated factor. By default the multiplier is <code>1000</code>. For example,
by setting the <code>--multiplier</code> parameter to <code>2000</code> all timeouts will be
doubled. The resulting timeout value affects how long time <code>expect</code>
operations will wait before reporting failure. If the time is omitted
like <code>[timeout]</code>, the timeout is reset to the default timeout
specified with the <code>--timeout</code> configuration parameter. The timeout
value <code>infinity</code> means infinity.</p>

<p><strong>[sleep Seconds]</strong> <br />
waits given number of seconds before proceeding in the script. No
<code>multiplier</code> factor is applied. The <code>sleep</code> command should be avoided
if possible. It absolutely not intended to be used for solving race
conditions. Find out some way to synchronize the test case properly
instead.</p>

<p><strong>[progress String]</strong> <br />
Displays <code>String</code> on the <code>stdout</code> stream together with the rest of the
progress info. May contain newlines.</p>

<p><strong>[config Var=Value]</strong> <br />
assigns a value to a <a href="#config_params">configuration parameter</a>. The
assignment takes place during parsing of the script file. The
configuration parameters in <strong>architecture specific files</strong> can be
overridden by <strong>command line options</strong>. For example <code>[config
timeout=2000]</code> can be overridden with <code>--timeout=4000</code>.  Explicit
<code>[config Var=Value]</code> settings in scripts takes however precedence over
settings in architecture specific files and command line options. See
the section <em>Configuration parameters</em> about valid configuration
parameters. Some config parameters can have multiple values, such as
<code>skip</code> and <code>require</code>. See their respective descriptions. See also the
configuration parameter <code>--config_dir</code> about the location of the
architecture specific files.</p>

<p><strong>[pattern_mode]</strong> <br />
<strong>[pattern_mode PatternMode]</strong> <br />
EXPERIMENTAL FEATURE!!! May be changed or removed in a future release.</p>

<p>Changes the behavior of fail and success patterns in a shell. By
default these patterns are matched against everything found in the
output streams (<code>stdout</code>, <code>stderr</code>). This means that they are both
matched against the characters actually are matching (in a <code>?</code>, <code>??</code>
or <code>???</code> commands) as well as the characters that are skipped (not
explicitely matched). This default behavior can also be achieved by
setting <code>PatternMode</code> to <code>all</code> for the shell.</p>

<p>By setting <code>PatternMode</code> to <code>skip</code> instead, only the skipped (not
explicitely matched) are matched against the fail and success
patterns.</p>

<p>See also the configuration parameter <code>--pattern_mode</code>.
<a name="cmd_line_opts"/></p>

<h1>Command line options</h1>

<p>Normal execution mode for Lux is to execute test suites and most of
the command line options affects the execution in different ways.
There are however a few auxiliary options that can be used to make
Lux perform other tasks.</p>

<ul>
<li>--help</li>
<li>--version</li>
<li>--reltool</li>
<li>--xref</li>
<li>--install</li>
<li>--make</li>
<li>--markdown</li>
<li>--annotate</li>
<li>--history</li>
<li>--merge</li>
<li>--mode</li>
<li>--doc</li>
</ul>

<h2>Script execution</h2>

<pre><code>lux [--mode Mode] [ConfigParam]... [File]...
</code></pre>

<p>Exit status is 0 if all test cases are successful and 1 otherwise.</p>

<p>See the section <a href="#config_params">Configuration parameters</a> about
script execution.</p>

<h2>Release management</h2>

<pre><code>lux --help
lux --version
lux --reltool [--root_dir RootDir]
lux --xref
lux --install [InstallDir] [--root_dir RootDir]
lux --make
lux --markdown
</code></pre>

<p><strong>--help</strong> <br />
Displays a brief list of all command line arguments, as well as a URL
to the full documentation.</p>

<p><strong>-h</strong> <br />
A shortcut for <code>--help</code>.</p>

<p><strong>--version</strong> <br />
Prints out the actual Lux version</p>

<p><strong>--reltool</strong> <br />
Starts the graphical tool <a href="http://www.erlang.org/doc/apps/reltool/index.html" title="Erlang release management tool">Reltool</a> which enables inspection of
internal Lux application dependencies. It is disabled in the
standalone installation.</p>

<p><strong>--xref</strong> <br />
Perform cross reference checks of Lux itself in order to find calls to
undefined functions.</p>

<p><strong>--install [InstallDir]</strong> <br />
See <a href="#../INSTALL">installation</a>. Installs the Lux application as a
standalone application in the <code>InstallDir</code> directory. <code>InstallDir</code>
must exist. If <code>InstallDir</code> is omitted only a dry run is performed. A
standalone installation is self-contained and contains a minimal
Erlang runtime system. It is however not neccessary to install Lux as
standalone. If Erlang already is installed on the system, Lux can make
use of that runtime environment. But sometimes it is useful to avoid
that dependency.</p>

<p><strong>--root_dir <code>RootDir</code></strong> <br />
Directs <a href="http://www.erlang.org/doc/apps/reltool/index.html" title="Erlang release management tool">Reltool</a> to use an alternate Erlang root directory instead
of the one currently being used. Affects <code>--install</code> and <code>--reltool</code>.</p>

<p><strong>--make</strong> <br />
Performs a simplified build only relying on a pre-installed Erlang/OTP
system. To be used with care on obscure platforms. See
<a href="#../INSTALL">installation</a>.</p>

<p><strong>--markdown</strong> <br />
Generates documentation for the Lux debugger on <a href="http://www.daringfireball.net/projects/markdown" title="Markdown">Markdown</a> format.
This is used internally by doc/Makefile.</p>

<h2>Log management</h2>

<pre><code>lux --annotate LogFile
lux --merge TargetLogDir [SourceLogFile]...
lux --history TargetLogDir [SourceLogFile]...
</code></pre>

<p><strong>--annotate LogFile</strong> <br />
Transforms textual log files into HTML format and annotates Lux script
code with log events. The generated HTML file will get the same name
as <code>LogFile</code> but with a <code>.html</code> extension added. See also the
<a href="#config_params">configuration parameter</a> <code>--html</code>.</p>

<p><strong>--merge TargetLogDir [SourceLogFile]...</strong> <br />
Merges the logs from multiple partial runs. When a test suite takes a
long time to run, it is possible to split it up in multiple runs with
a few test cases in each. When all test cases in the suite has
completed, their logs can me merged. The result will look like the
test suite was run in one go. The original logs will be kept as is and
the new <code>lux_summary.log.html</code>log will contain links to the original
logs. For example the original <code>lux_config.log</code> files cannot be merged
as the separate test case runs may been run in very different
environments (ex. different Docker containers).</p>

<p>The <code>--merge</code> option is much safer than using <code>--extend_run</code> as it
enables true parallel runs of the test cases. The <code>--extend_run</code> can
only be used when the test cases are run in sequence after each other.</p>

<p><strong>--history TargetLogDir [SourceLogFile]...</strong> <br />
Generates an HTML file which summarizes the history of all test runs.
The history file will be generated on the <code>TargetLogDir</code> directory and
is named <code>lux_history.html</code>. Its behavior can be customized by using
the <code>--suite</code>, <code>--run</code>, <code>--revision</code> and <code>--hostname</code>
<a href="#config_params">configuration parameters</a>.</p>

<p>The history file generation is done by analyzing <code>lux_summary.log</code>
files. A <code>SourceLogFile</code>s may either be an already existing
<code>lux_history.html</code> file or a directory. When <code>SourceLogFile</code> is a
directory all subdirectories not containing a <code>lux.skip</code> file will be
searched for <code>lux_summary.log</code> files.</p>

<p><code>SourceLogFile</code> may also be a <code>lux_history.html</code> file, in that case
the <code>lux_summary.log</code> files are extracted from the history file. This
can be used for the purpose of merging existing history files. The
<code>SourceLogFile</code> may either be a local filename or an URL. If it is an
URL both the history file and the summary log files are fetched over
the network. The resulting history file will then contain URL's,
implying that it may relocated without getting dangling links.</p>

<p>The <code>SourceLogFile</code> may be prefixed with a suite name, like
<code>SuitePrefix::SourceLogFile</code>. I that case the <code>SuitePrefix</code> will
override the <code>--suite</code> parameter setting from the original run. This
may be useful when a suite has been reused and thus run several
times. For example when there are several versions of the system under
test:</p>

<pre><code>lux --history . debug::PathToDebugLogDir release::PathToReleaseLogDir
</code></pre>

<p><strong>--history_cases WhichCases...</strong>
<code>WhichCases</code> controls the selection of failing test cases in a history
run. It defaults to <code>any</code> which means that all test cases that has
failed at least once are included. Setting it to <code>latest</code> means that
only the test cases that failed in the latest run is included.
<a name="config_params"/></p>

<h1>Configuration parameters</h1>

<pre><code>lux [--mode Mode] [ConfigParam]... [File]...
</code></pre>

<p>Exit status is 0 if all test cases are successful and 1 otherwise.</p>

<p>Configuration parameters can be given as command line options
<code>--Var=Val</code>, as <code>[config Var=Value]</code> statements in a script or in a
<strong>architecture specific file</strong>.</p>

<p>An <code>architecture specific file</code> is a file with configuration
statements only valid for a certain architecture/platform/system.
The syntax of such a file is the same a normal Lux script, but only
configuration settings (<code>[config Var=Value]</code>) are extracted. See
also the configuration parameters <code>--config_name</code> and <code>--config_dir</code>.
The file extension is <code>.luxcfg</code>.</p>

<p>When a test suite (one or more test cases) is to be evaluated, the Lux
engine will determine the software/hardware signature of the system to
construct the name of a architecture specific file. If a file with
that name exists, the architecture specific configuration settings
will be extracted and used as base for the entire test suite. These
settings can however be overridden by command line options and
<code>[config var=val]</code> statements in each test case file.</p>

<p>The Lux engine evaluates one or more Lux files. Lux files has normally
<code>.lux</code> as extension. See the configuration parameter <code>--file_pattern</code>.
If a directory is given as input, all <code>.lux</code> files in that directory
and its sub directories are evaluated. The given files (files or
directories) are called test suites and the derived files (actual Lux
scripts) are called test cases. The configuration parameter <code>--skip</code>
can be used to conditionally skip test cases.</p>

<p>The lookup of configuration parameters/settings is performed in the
following order:</p>

<ol>
<li><p>Command line parameters. Eg. <code>--skip_skip</code> to ignore all skip settings.</p></li>
<li><p>Command line parameters from <code>LUX_FLAGS</code> environment variable. To
be used interactively.</p></li>
<li><p>Command line parameters from <code>LUX_SYSTEM_FLAGS</code> environment
variable. To be used by makefiles, scripts etc.</p></li>
<li><p>Test case specific configuration settings.</p></li>
<li><p>Architecture specific configuration settings from a <code>.luxcfg</code> file
on the <code>--config_dir</code> directory. See <code>--config_name</code> about the naming
convention.</p></li>
<li><p>Non-architecture specific configuration settings defined in a file
named <code>luxcfg</code> on the <code>--config_dir</code> directory.</p></li>
<li><p>Site local default configuration settings defined in a file named
<code>luxcfg</code> on the <code>lux/priv</code> directory.</p></li>
<li><p>Environment variables automatically converted to <code>[config var=val]</code>
settings..</p></li>
<li><p>Hardcoded built-in default values.</p></li>
</ol>

<h2>Test case control</h2>

<p><strong>--mode Mode</strong> <br />
Mode can be one of :</p>

<ul>
<li><code>execute</code>  - evaluate the test cases. This is default.</li>
<li><code>validate</code> - parse all script files and configuration files and
           report syntax errors and warnings.</li>
<li><code>dump</code>     - parse and dump the internal form for all script files
           and configuration files and report syntax errors and
           warnings.</li>
<li><code>expand</code>   - parse and expand source files for all script files
           and configuration files.</li>
<li><code>list</code>     - display a list of all (non-skipped) test cases.
           One file per line.</li>
<li><code>list_dir</code> - display a list of all directories with non-skipped
           test cases. One directory per line.</li>
<li><code>doc</code>      - extract all <code>[doc]</code> and <code>[docN]</code> strings and display
           them on a simple format which is as follows.
           First the script file name is printed on an own line
           ending with a colon, followed by all doc strings, each
           one on a separate line. The doc strings are indented
           with a tab char for each doc level. See [docN].</li>
</ul>

<p><strong>--doc Level</strong> <br />
Implies <code>--mode=doc</code>. Restricts how many documentation levels which
should be displayed. <code>--doc=1</code> only shows documentation on level 1,
<code>--doc=2</code> shows documentation both on level 1 and 2.</p>

<p>The first <code>[doc]</code> documentation string in a script is a bit special as
it is regarded as a one line summary of the script. With <code>--doc=0</code>
only the oneline summary lines are displayed.</p>

<p><strong>--rerun Result</strong> <br />
Rerun old test cases. The test case candidates are found by parsing
old log summary files. If any <code>File</code> is explicitly given on command
line these files are interpreted as log directories possibly
containing summary log files. If no <code>File</code> is given the log directory
referred to by the <code>latest_run</code> link is used.</p>

<p>For each found test case its result must have the same outcome or
higher (worse) than <code>Result</code>.<code>Result</code> is an enum whose names and
relative values are as follows:</p>

<pre><code>enable &lt; success &lt; skip &lt; warning &lt; fail &lt; error &lt; disable
</code></pre>

<p>For example <code>--rerun=fail</code> implies that all old test cases whose
outcome is fail or error will be rerun.</p>

<p>Default is <code>disabled</code>, which means that this behavior is disabled.</p>

<p><strong>-r</strong> <br />
A shortcut for <code>--rerun=fail</code>.</p>

<p><strong>--file_pattern</strong> <br />
Specify file pattern for scripts to be executed when a directory is
given. Defaults to <code>.*.lux$</code>.</p>

<p><strong>--var</strong> <br />
Overrides environment variable settings. Each entry must be of the
form <code>var=value</code>.</p>

<p><strong>--config_name ConfigName</strong> <br />
Normally Lux figures out which system software/hardware it runs on,
but it can explicitly be overridden with the <code>ConfigName</code> option. The
<code>ConfigName</code> is used to read system architecture specific configuration
parameters from a file named <code>ConfigName.luxcfg</code>. By default <code>ConfigName</code>
 is obtained from <code>uname -sm</code> where <code>ConfigName</code> is set to <code>Kernel-Machine</code>.
This behavior can be overridden by adding a file named after the name of
the host (<code>hostname.luxcfg</code>) on the <code>ConfigDir</code> directory.</p>

<p><strong>--config_dir ConfigDir</strong> <br />
A directory where architecture specific configuration files may
reside. The format of the architecture specific files a subset of the
script format. Only <code>[config var=value]</code> statements are extracted from
the architecture specific file. The config settings in the
architecture specific file may be overridden by config settings in the
script files. Config settings in script files may be overridden by
command line options. Architecture specific files are by default
located in the subdirectory called <code>priv</code> in the <code>Lux</code> application.</p>

<p>Non-architecture settings can be put in a file named <code>luxcfg</code>. But
those will be overridden by the architecture specific settings.</p>

<p><strong>--hostname Hostname</strong> <br />
The <code>Hostname</code> overrides the hostname obtained from the operating
system. It may be useful when testing config settings of other
machines or faking the hostname in a test environment with multiple
equivalent slaves.</p>

<p><strong>--require Var</strong> <br />
<strong>--require Var=Value</strong> <br />
Require the given variable to be set. The script will fail if
the variable not is set. This option can be used multiple times,
which means that all given Vars are required to be set.
Typically require is used to test on presence of environment
variables. <code>--require</code> is intended to be used as <code>[config require=Var]</code>
or <code>[config require=Var=Value]</code> statements within scripts. The
construction <strong>Var=Value</strong> is little more restrictive as it
requires the variable to be set to a certain value.</p>

<p><strong>--skip Var</strong> <br />
<strong>--skip Var=Value</strong> <br />
Skip execution of the script if the given variable is set. This
option can be used multiple times, which means that it suffices
that one of the given <code>Var</code>s is set in order to skip the test
case. Typically <code>--skip</code> is used to test on presence of environment
variables. <code>--skip</code> is intended to be used as <code>[config skip=Var]</code>
or <code>[config skip=Var=Value]</code> statements within scripts. The
construction <strong>Var=Value</strong> is little more restrictive as it requires
that the variable is set to a certain value.</p>

<p><strong>--skip_unless Var</strong> <br />
<strong>--skip_unless Var=Value</strong> <br />
Skip execution of the script if the given variable NOT is set. This
option can be used multiple times, which means that it suffices
that one of the given <code>Var</code>s NOT is set in order to skip the test
case. Typically <code>--skip_unless</code> is used to test on absence of
environment variables. <code>--skip_unless</code> is intended to be used as
<code>[config skip_unless=Var]</code> or <code>[config skip_unless=Var=Value]</code>
statements within scripts. The construction <strong>Var=Val</strong> is little more
restrictive as it requires that the variable is set to a certain
value.</p>

<p><strong>--unstable Var</strong> <br />
<strong>--unstable Var=Value</strong> <br />
Mark a test case as unstable if the given variable is set. This
implies failures to be reported as warnings.The option can be used
multiple times, which means that it suffices that one of the given
<code>Var</code>s is set in order to mark the test case as unstable. Typically
<code>--unstable</code> is used to test on presence of environment
variables. <code>--unstable</code> is intended to be used as <code>[config
unstable=Var]</code> or <code>[config unstable=Var=Value]</code> statements within
scripts. The construction <strong>Var=Value</strong> is little more restrictive as
it requires that the variable is set to a certain value.</p>

<p><strong>--unstable_unless Var</strong> <br />
<strong>--unstable_unless Var=Value</strong> <br />
Mark a test case as unstable if the given variable NOT is set. This
implies failures to be reported as warnings. The option can be used
multiple times, which means that it suffices that one of the given
<code>Var</code>s NOT is set in order to mark the test case as unstable.
Typically <code>--unstable_unless</code> is used to test on absence of
environment variables. <code>--unstable_unless</code> is intended to be used as
<code>[config unstable_unless=Var]</code> or <code>[config unstable_unless=Var=Value]</code>
statements within scripts. The construction <strong>Var=Val</strong> is little more
restrictive as it requires that the variable is set to a certain
value.</p>

<p><strong>--skip_unstable</strong> <br />
<strong>--skip_unstable=true</strong> <br />
Skip unstable test cases. See <code>--unstable</code> and <code>--unstable_unless</code>.</p>

<p><strong>--skip_skip</strong> <br />
<strong>--skip_skip=true</strong> <br />
Forces Lux to not care about <code>--skip</code> and <code>--skip_unless</code> settings.
Overrides <code>--skip_unstable</code>.</p>

<p><strong>--fail_when_warning</strong> <br />
<strong>--fail_when_warning=true</strong> <br />
Forces Lux to fail if there are any warnings.</p>

<h2>Log control</h2>

<p><strong>--log_dir LogDir</strong> <br />
A directory where log files will be written. Default is <code>./lux_logs</code>.</p>

<p><strong>--html Html</strong> <br />
The <code>Html</code> option controls whether the logs should be converted to
HTML or not. It is an enum denoting the outcome of the tests.
If the actual outcome is the same or higher than <code>Html</code> then the
logs will be converted. The possible outcome and their relative
values are as follows:</p>

<pre><code>validate &lt; enable &lt; success &lt; skip &lt; warning &lt; fail &lt; error &lt; disable
</code></pre>

<p>Default is <code>enable</code>. <code>validate</code> behaves as <code>enable</code> but will also
perform validation of the generated HTML files. The logs can also be
converted to HTML manually later by using the command line option
<code>--annotate</code>.</p>

<p><strong>--tap LogFile</strong> <br />
A file where <a href="http://testanything.org/" title="Test Anything Protocol">TAP</a> events should be written. The file names
<code>stdout</code> and <code>stderr</code> are specially handled. They causes the log events
to be written to standard output respective standard error. Multiple
"files" can be given. A log file named lux.tap will always be generated,
regardless of this option.</p>

<p><strong>-t</strong> <br />
A shortcut for <code>--progress=silent --tap=stdout</code>.</p>

<p><strong>--junit</strong> <br />
Generate a JUnit test report for the test run that can be used for example
by Jenkins to show test result using the JUnit plugin. The generated test
report will be named <code>lux_junit.xml</code>.</p>

<p><strong>--case_prefix CasePrefix</strong> <br />
A prefix string which is prepended to the script names in the user
friendly log files (TAP and HTML). With this the log files can provide
the context for the test case(s), such as subsystem or test suite.</p>

<h2>Timeouts</h2>

<p><strong>--multiplier Multiplier</strong> <br />
In order to be able to run the tests on very slow hardware, there is a
concept of a <code>Multiplier</code>. Each time a timer is initiated (except
sleep) its value is multiplied with the <code>Multiplier</code> value.
<code>Multiplier</code> is an integer and defaults to <code>1000</code>, meaning that
<code>[timeout 10]</code> actually is 10 seconds. The timeout value 10 is
multiplied with the multiplier value 1000. Perhaps it is easier to
think in milliseconds. 10 means 10<em>1000=10000 milliseconds. By
changing the multiplier to a higher value, such as 3000, it will cause
Lux to use longer timeouts. E.g. 10</em>3000=30000 milliseconds.
<code>--multiplier</code> is intended to be set in architecture/host specific
files to provide different settings on different systems.</p>

<p><strong>--timeout Timeout</strong> <br />
The script expects the shell output to match given
<a href="http://www.erlang.org/doc/man/re.html#regexp_syntax" title="Erlang style regular expressions (re)">regular expression</a>s. But the output must be received within a
given time limit. The <code>Timeout</code> specifies how long it will wait before
the script fails. The <code>Timeout</code> defaults to <code>10000</code> milliseconds
(<code>10</code> seconds). This <code>Timeout</code> can be overridden by the statement
<code>[timeout Timeout]</code> in the script itself.</p>

<p><strong>--cleanup_timeout CleanupTimeout</strong> <br />
When the script reaches the <code>[cleanup]</code> marker, the ordinary
<code>Timeout</code> will be set to <code>CleanupTimeout</code>. The <code>CleanupTimeout</code>
defaults to <code>100000</code> milliseconds (<code>100</code> seconds).</p>

<p><strong>--suite_timeout SuiteTimeout</strong> <br />
If the duration of the execution exceeds the <code>SuiteTimeout</code>, it
is aborted. The <code>SuiteTimeout</code> defaults to <code>infinity</code>, but can
be any positive integer value in the unit of milliseconds.</p>

<p><strong>--case_timeout CaseTimeout</strong> <br />
If the the duration of a single test case exceeds the
<code>CaseTimeout</code>, it is aborted. It can be any positive integer
value in the unit of milliseconds or <code>infinity</code>. The default
is <code>300000</code> (5 minutes).</p>

<p><strong>--flush_timeout FlushTimeout</strong> <br />
An experimental timeout setting.
All output from a shell is buffered and matched against
<a href="http://www.erlang.org/doc/man/re.html#regexp_syntax" title="Erlang style regular expressions (re)">regular expression</a>s. It can however explicitly be flushed by
the script. When this is done, the engine first waits a while
before it discards the output. How long it waits is controlled
by <code>FlushTimeout</code>. It defaults to <code>0</code>. If you want to experiment
with it, <code>1000</code> milliseconds (1 second) can be a resonable value.</p>

<p><strong>--poll_timeout PollTimeout</strong> <br />
An experimental timeout setting.
When the Lux engine receives output from a shell it will
wait in <code>PollTimeout</code> milliseconds for more output before it
tries to match it against any <a href="http://www.erlang.org/doc/man/re.html#regexp_syntax" title="Erlang style regular expressions (re)">regular expression</a>s. It defaults
to <code>0</code>. If you want to experiment with it, <code>100</code> milliseconds
(1/10 second) can be a resonable value.</p>

<p><strong>--pattern_mode PatternMode</strong> <br />
EXPERIMENTAL FEATURE!!! May be changed or removed in a future release.</p>

<p>Changes the default behavior of fail and success patterns. This can
be overridden separately for each shell with the [pattern_mode
PatternMode] command. Valid settings are <code>all</code> and <code>skip</code>. The
default is <code>all</code>.</p>

<p>See the command <code>\[pattern\_mode PatternMode\]</code> for details.</p>

<p><strong>--risky_threshold RiskyThreshold</strong> <br />
An experimental timeout setting.
By default Lux warns for risky timers, i.e. timers that are close
to timeout. This may cause intermittent failures in future runs. By
default it is set to <code>0.85</code> which means that 85% of the timer was
used.</p>

<p><strong>--sloppy_threshold RiskyThreshold</strong> <br />
An experimental timeout setting.
By default Lux warns for sloppy timers, i.e. timers where only a very
small part of the timer is used. This may cause secondary problems,
e.g. test run abortions by Jenkins or similar systems. Such timers are
also rather inconvenient to debug as they may take hours to time out.
By default it is set to <code>0.000000001</code> which means that only 1 ppb
(parts per billion) of the timer was used, or less.</p>

<h2>History control</h2>

<p><strong>--suite Suite</strong>  </p>

<p>The Suite is used for bookkeeping a name which later is used for
printing out the history of test runs. See the
 <a href="#cmd_line_opts">command line option</a> <code>--history</code>.</p>

<p><strong>--run RunId</strong> <br />
The <code>RunId</code> is used for bookkeeping a name which later is used for
printing out the history of test runs. See the
 <a href="#cmd_line_opts">command line option</a> <code>--history</code>.</p>

<p><strong>--extend_run</strong> <br />
<strong>--extend_run=true</strong> <br />
Combines two runs into one. The summary log of an earlier run is
extended with the outcome of the new run. <code>--log_dir</code> can be given
explicitly. If not, the symbolic <code>latest_run</code> link is used to find
a suitable log directory. Note that the runs cannot be run in parallel
as they will write into the same files. It is much better to use the
<code>--merge</code> option to perform the merge of log files when all runs are
done.</p>

<p><strong>--revision Revision</strong> <br />
The <code>Revision</code> is used for bookkeeping a repository revision
(changeset) which later is used for printing out the history of test
runs. See the <a href="#cmd_line_opts">command line option</a> <code>--history</code>.</p>

<p><strong>--hostname Hostname</strong> <br />
The <code>Hostname</code>overrides the hostnames extracted from the log files.
It may for example be useful in a test environment where the test
runs are distributed over multiple equivalent slaves. See the
<a href="#cmd_line_opts">command line option</a> <code>--history</code>.</p>

<p><strong>--html validate</strong>
Performs validation of the generated HTML files.</p>

<p><a name="debugging"/></p>

<h2>Debugging and tracing</h2>

<p><strong>--progress ProgressLevel</strong> <br />
<code>ProgressLevel</code> can be one of <code>silent</code>, <code>summary</code>, <code>brief</code>, <code>doc</code>,
<code>compact</code> and <code>verbose</code>. It defaults to <code>brief</code> which means that
single characters are printed to stdout. <code>doc</code> is like <code>brief</code> but in
this mode doc strings are also printed on stdout. <code>compact</code> means that
an event trace is printed on stdout. It is the same event trace that
is written to the <code>event log</code>. verbose contains the same info as
compact but is more readable (the newlines are expanded). <code>summary</code>
means that no progress is printed. <code>silent</code> means that nothing is
printed. The <code>brief</code> characters have the following meanings:</p>

<pre><code>   . - a new row in the script is being interpreted
   : - output is being received from a shell
   c - the normal cleanup marker
   C - the cleanup marker during premature termination
   z - is printed out each second while sleeping
   W - is printed out when a dynamic warning is issued
   ( - beginning of a macro, loop or an include file
   ) - end of a macro, loop or an include file
   ? - waiting for shell output. Preceded with lineno.
</code></pre>

<p><code>[progress String]</code> can also be used to display progress info.</p>

<p>The <code>ProgressLevel</code> can also interactively be changed via the debugger.</p>

<p><strong>-c</strong> <br />
A shortcut for <code>--progress=compact</code>.</p>

<p><strong>-v</strong> <br />
A shortcut for <code>--progress=verbose</code>.</p>

<p><strong>-t</strong> <br />
A shortcut for <code>--progress=silent --tap=stdout</code>.</p>

<p><strong>--debug</strong> <br />
The debugger is always available (even without this flag) and waiting
for input on the <code>stdin</code> stream. With the <code>--debug</code> flag the debugger
is attached to the script before the first line is executed and
waiting for input. The command <code>attach</code> (<code>a</code> for short) attaches the
debugger to the script and pauses its execution. The command
<code>progress</code> (<code>p</code> for short) toggles the verbosity level between <code>brief</code>
and <code>verbose</code>. Use the debugger command <code>help</code> to get more info about
the available commands. See also the section <a href="#debugging">debugging and tracing</a>.</p>

<p><strong>-d</strong> <br />
A shortcut for <code>--debug</code>.</p>

<p><strong>--debug_file SavedFile</strong> <br />
Loads the commands in the <code>SavedFile</code> before the first line in the
script is executed. See the debugger command <code>save</code> and <code>load</code> for
more info. The format of the <code>SavedFile</code> is very simple and may be
manually edited. For example <code>break</code> and <code>continue</code> may be convenient
commands to add to such a file.</p>

<h2>Miscellaneous</h2>

<p><strong>--newshell</strong> <br />
In <code>--newshell</code> mode shells are created with the <code>[newshell Name]</code> command
and making another shell active is done with <code>[shell Name]</code>. That is the
<code>[shell Name]</code> command cannot be used to create new shells in newhell mode.</p>

<p><strong>--shell_cmd Cmd</strong> <br />
<strong>--shell_args Arg</strong> <br />
These parameters controls which program that will be started when a
script starts a shell. By default <strong><code>/bin/sh -i</code></strong> is started as
<code>--shell_cmd</code> and <code>--shell_args</code> defaults to <code>/bin/sh</code> and <code>-i</code>
respectively. <code>--shell_args</code> is a bit special in how this parameter is
treated by Lux. For example, assume you want to give <code>-i --rcfile
/another/rcfile</code> as arguments to the shell. Then you need to give
<code>--shell_args=-i</code>, <code>--shell_args=--rcfile</code> and
<code>--shell_args=/another/rcfile</code> as separate parameters. Each
<code>--shell_args</code> is literally treated as one argument. It implies
<code>--rcfile</code> to be one argument and <code>/another/rcfile</code> another. Further,
the repetition of the <code>-i</code> argument is needed as the shell argument
list is reset to scratch once an explicit <code>--shell_args</code> is given.</p>

<p><strong>--shell_prompt_cmd PromptCmd</strong> <br />
<strong>--shell_prompt_regexp PromptRegExp</strong> <br />
When Lux starts a shell the prompt is set to <strong><code>SH-PROMPT:</code></strong> by
default. In Bourne shell, which is the default shell, the variable
<code>PS1</code> is used to set the prompt. This is obtained by using the command
<code>export PS1=SH-PROMPT:</code> followed by an explicit match of the prompt
using the regexp <code>^SH-PROMPT:</code>. This behavior can be overridden by
using <code>--shell_prompt_cmd</code> and <code>--shell_prompt_regexp</code> respectively
when using more exotic shells, such as the Bourne Again shell:</p>

<pre><code> [config shell_cmd=/bin/bash]
 [config shell_prompt_cmd=unset PROMPT_COMMAND; export PS1=SH-PROMPT:]
</code></pre>

<p><strong>--shell_wrapper</strong> <br />
<strong>--shell_wrapper [Executable]</strong> <br />
In order to get the terminal settings to work properly in advanced
interactive cases such as tab completion etc., the shell needs to be
executed in a <strong>pseudo terminal</strong>. This is accomplished by using a
wrapper program setting up the terminal correctly. The arguments to
the wrapper program is the name of the shell program with its
arguments (for example <code>/bin/sh -i</code>, see also see <code>--shell_cmd</code> and
<code>--shell_args</code>). The wrapper is expected to first configure the
terminal and then start the shell.</p>

<p>The built-in executable <code>lux/priv/runpty</code> will be used by default as
shell wrapper (if it has been built properly).</p>

<p>It is also possible to use no shell wrapper at all by omitting the
<code>Executable</code> value (or simply set it to the empty string "").</p>

<pre><code> [config shell_wrapper=]
</code></pre>

<p><strong>--shell_wrapper_mode WrapperMode</strong>
The WrapperMode is used for debugging the shell wrapper program. By
default it is set to <code>silent</code>, but it can also be set to <code>debug</code> or
<code>trace</code>. Debug will log wrapper internal events, such as signal
handling. Trace will also log the socket payload. When enabled in a
test case, a log file will be created per shell on the extra logs
directory.</p>

<pre><code> [config shell_wrapper_mode=trace]
</code></pre>

<p><strong>--post_case</strong> <br />
<strong>--post_case [IncludeFile]</strong> <br />
Enable centrally defined cleanup code to be run after each test case
regardless whether they fail or succeed. The purpose of this is to
make it possible to report and possibly undo unwanted side effects
which the cleanup code in the test case has failed to handle.</p>

<p>There may be more than one <code>post_case</code> parameter. For each one a shell
will be started when the test case has ended and the Lux code in the
<code>IncludeFile</code> will be executed in the context of the test case.</p>

<p><strong>--line_term Chars</strong> <br />
Specify the character sequence added to the end of lines sent to
a shell. It defaults to <code>\n</code>.
<a name="logs"/></p>

<h1>Logs</h1>

<p>Lux will create a new directory for each test run. By default the log
files are generated under <code>./lux_logs/run_yyyy_mm_dd_hh_mm_ss_mmmmmm</code>
where <code>run_yyyy_mm_dd_hh_mm_ss_mmmmmm</code> is a unique directory name
generated from the current time. A symbolic link called
<code>./lux_logs/latest_run</code> will also be created. It refers to the newly
created log directory for the latest run. If the <a href="#config_params">configuration
parameter</a> <code>--log_dir LogDir</code> is set, the given path
will be used instead and no symbolic link will be created.</p>

<p>Each test run will result in the following log files:</p>

<ul>
<li><strong>lux_summary.log</strong> which contains information about the outcome of
each test case and paths to test case logs.</li>
<li><strong>lux_config.log</strong> which contains all configuration settings common
for the test suite.</li>
<li><strong>lux_result.log</strong> which contains a condensed summary of the outcome.</li>
<li><strong>lux.tap</strong> is a <a href="http://testanything.org/" title="Test Anything Protocol">TAP</a> compliant log file.</li>
</ul>

<p>For each test case several logs are written:</p>

<ul>
<li><strong>$CASE.event.log</strong> contains every internal event. Such as which
statements that has been executed, output from shells etc.  This is
the main source for information for detailed information about the
outcome of a test case.</li>
<li><strong>$CASE.config.log</strong> contains configuration settings for the test case.</li>
<li><strong>$CASE.$SHELL.stdin.log</strong> contains the raw input to the
shells. There is one such log per shell.</li>
<li><strong>$CASE.$SHELL.stdout.log</strong> contains the raw output from the
shells. There is one such log per shell.</li>
<li><strong>$CASE.orig</strong> is a copy of the test script.</li>
</ul>

<p>The summary log, result log, event logs and config logs are by default
processed and converted to HTML in order to make them easier to
read. This can be controlled with the <code>--html</code> <a href="#config_params">configuration
parameter</a>. The html logs are called
<code>lux_summary.log.html</code> and <code>$CASE.event.log.html</code> respectively.</p>

<p>The outcome of multiple test runs can be assembled in a <strong>history
log</strong>. This log is very useful when Lux is used in a daily build
environment and some test cases suddenly starts to fail. By using the
time line in the history log it can be possible to determine which
checkin to the repository that introduced the first failure. See the
<code>--history</code> <a href="#cmd_line_opts">command line option</a>. Its behavior can be
customized by using the <code>--suite</code>, <code>--run</code> and <code>--revision</code>
<a href="#config_params">configuration parameters</a>.
<a name="debug_cmds"/></p>

<h1>Debugger for Lux scripts</h1>

<p>When <code>lux</code> is started with the <code>--debug</code> option, the debugger
will attach to the script before its execution has started. An
optional file with saved commands may be processed at this stage.
The debugger can also be attached to the script in the middle of
the execution by entering the command "attach" (or an abbreviation
of the command) and pressing the enter key.</p>

<p>Several parameters has a lineno as parameter see <code>help lineno</code>.</p>

<p>Blank command lines implies that the previous command is repeated.
If no command has been entered yet, the command <code>help</code> is assumed.</p>

<p>Commands may be abbreviated. Use the help command (for example
<code>help help</code> (or <code>h h</code> for short) to get more detailed descriptions
of the commands.</p>

<h2>Available commands: </h2>

<ul>
<li>attach   - attach to script and pause its execution</li>
<li>break    - set, delete and list breakpoints</li>
<li>continue - continue script execution</li>
<li>help     - display description of a command</li>
<li>list     - list script source</li>
<li>load     - load file with debug commands</li>
<li>next     - execute next command</li>
<li>progress - set verbosity level of progress</li>
<li>quit     - quit a single test case or the entire test suite</li>
<li>save     - save debug state to file</li>
<li>skip     - skip execution of one or more commands</li>
<li>shell    - connect to a shell</li>
<li>tail     - display log files</li>
<li>TRACE    - start or stop internal tracing</li>
</ul>

<h2>Available parameters:</h2>

<ul>
<li>lineno - lineno in source file</li>
</ul>

<h2>lineno parameter</h2>

<p>Several commands has a lineno as parameter. It is a string which
is divided in several components. The components are separated
with a colon and are used to refer to line numbers in include
files, macros and loops. The first component is a bit special.
It may be a file name or a line number. The file name may be
abbreviated.</p>

<p>Assume that there is a file called main, which includes a file
called outer at line 4 and the file outer includes a file called
inner at line 12.</p>

<p>Here are a few examples of how lineno can be used:</p>

<ul>
<li>3       - line 3 in current file</li>
<li>main    - line 1 in file main</li>
<li>m:3     - line 3 in file main</li>
<li>:3      - line 3 in file main</li>
<li>inner   - line 1 in file inner</li>
<li>outer   - line 1 in file outer</li>
<li>o:12    - line 12 in file outer</li>
<li>4:12:6  - line 6 in file inner if it is included
        on line 12 in outer and outer is included
        on line 4 in main.</li>
</ul>

<h2>TRACE [action] [mode]</h2>

<p>Start or stop internal tracing
Default is to display the trace mode (none|case|suite|event).</p>

<p><strong>Parameters:</strong>  </p>

<ul>
<li>action - Trace action; enum(START|STOP)  </li>
<li>mode   - Trace mode; enum(CASE|SUITE|EVENT)  </li>
</ul>

<h2>attach</h2>

<p>Attach to script and pause its execution</p>

<p><strong>Parameters:</strong>  </p>

<ul>
<li>no parameters</li>
</ul>

<h2>break [lineno] [duration]</h2>

<p>Set, delete and list breakpoints</p>

<p>When a breakpoint is set it may either be normal (default)
or temporary. The difference between them is that normal
breakpoints remains after the break has been triggered,
while temporary breakpoints are automatically deleted when
they have been triggered once. delete is used to immediately
remove the breakpoint.</p>

<p>Without parameters, all breakpoints are listed.</p>

<p><strong>Parameters:</strong>  </p>

<ul>
<li>lineno   - lineno in source file; lineno  </li>
<li>duration - controls the duration of the breakpoint; enum(normal|temporary|delete|skip)  </li>
</ul>

<h2>continue [lineno]</h2>

<p>Continue script execution</p>

<p><strong>Parameters:</strong>  </p>

<ul>
<li>lineno - run to temporary breakpoint at lineno; lineno  </li>
</ul>

<h2>help [command]</h2>

<p>Display description of a command</p>

<p><strong>Parameters:</strong>  </p>

<ul>
<li>command - debugger command; string  </li>
</ul>

<h2>list [n_lines] [lineno]</h2>

<p>List script source</p>

<p>If no "lineno" is given, the listing will start from the
current line or from the latest given "lineno" if no other
commands have been given in between.</p>

<p><strong>Parameters:</strong>  </p>

<ul>
<li>n_lines - number of lines; 1 >= integer =&lt; infinity  </li>
<li>lineno  - start listing at lineno; lineno  </li>
</ul>

<h2>load [file]</h2>

<p>Load file with debug commands</p>

<p><strong>Parameters:</strong>  </p>

<ul>
<li>file - file name. Default is "./lux.debug".; string  </li>
</ul>

<h2>next</h2>

<p>Execute next command
A multi-line command counts as one command.</p>

<p><strong>Parameters:</strong>  </p>

<ul>
<li>no parameters</li>
</ul>

<h2>progress [level]</h2>

<p>Set verbosity level of progress</p>

<p><strong>Parameters:</strong>  </p>

<ul>
<li>level - verbosity level. Toggle between brief and verbose by default.; enum(silent|summary|brief|doc|compact|verbose|etrace|ctrace)  </li>
</ul>

<h2>quit [scope]</h2>

<p>Quit a single test case or the entire test suite
in a controlled manner. Runs cleanup if applicable.</p>

<p><strong>Parameters:</strong>  </p>

<ul>
<li>scope - scope of exit; enum(case|suite)  </li>
</ul>

<h2>save [file]</h2>

<p>Save debug state to file</p>

<p><strong>Parameters:</strong>  </p>

<ul>
<li>file - file name. Default is "lux.debug".; string  </li>
</ul>

<h2>shell [name] [mode]</h2>

<p>Connect to a shell</p>

<p>With no argument, the names of the shells will be listed.
In the listing the active shell is preceeded by an arrow
and zombie shells with an star. Repeating the command
will disconnect the shell. Repeat again to connect...</p>

<p>Once a shell is connected, its stdout will be tapped and
subsequent output will be printed out, beginning with the
buffered (non-processed) data.</p>

<p>Data can also be sent to the stdin of the shell. In
foreground mode all entered text is sent as is to the
shell. The foreground mode is exited with a single """"
line. In background mode (default), the debugger responds
to normal debugger commands as well as a few special
commands which only is available in background mode:</p>

<h2>Sub commands for "shell":</h2>

<ul>
<li>! - sends text with a trailing newline</li>
<li>~ - sends text without a trailing newline</li>
<li>= - displays current output buffer</li>
<li>? - empties the output buffer</li>
</ul>

<p><strong>Parameters:</strong>  </p>

<ul>
<li>name - name of shell; string  </li>
<li>mode - mode of operation; enum(background|foreground)  </li>
</ul>

<h2>skip [lineno]</h2>

<p>Skip execution of one or more commands
Skip until given lineno is reached.</p>

<p><strong>Parameters:</strong>  </p>

<ul>
<li>lineno - lineno in source file; lineno  </li>
</ul>

<h2>tail [index] [format] [n_lines]</h2>

<p>Display log files</p>

<p>With no argument, the names of the log files will be listed.
Each one is preceeded by its index number and optionally a
star. The star means that the log has been updated since the
previous status check. Use the index to display a particular
log. Such as "t 5" for the event log. Press enter to
display more lines. n_lines can be used to override that
behavior andonly display a fixed number of lines regardless
of the command is repeated or not.</p>

<p><strong>Parameters:</strong>  </p>

<ul>
<li>index   - log number; 1 >= integer =&lt; infinity  </li>
<li>format  - display format; enum(compact|verbose)  </li>
<li>n_lines - fixed number of lines; 1 >= integer =&lt; infinity <br />
<a name="examples"/></li>
</ul>

<h1>Examples</h1>

<h2>A successful test case</h2>

<p>Here is an example of a test script. It starts couple of concurrent
shells, sends text to them with the <code>!</code> command and matches expected
output with <code>?</code>. The send and match operations are always performed in
context of the active shell.</p>

<p>Match operations search for a given pattern in the output stream. Once
a match is found, the preceding characters are skipped. There are a
few flavors of match operations. They may be single-line or
multi-line.  Evaluate regular expressions or verbatim. Variables can
be expanded or not. When using regular expressions variables can be
bound to parts of the output and used later in the test case.</p>

<p>There are different variable scopes. They may be accessible from all
shells (global), only accessible within the shell where they were set
(local), only accessible with in their lexical scope (my) such as
within a macro, loop etc.</p>

<p>When a test case has side effects, such as creating files, start
programs etc., the test case should have a <code>[cleanup]</code> section where
the side effect is reversed. Otherwise subsequent test cases may
fail. The cleanup section is always executed, regardless of the script
succeeds or fails.</p>

<p>Snippet from the enclosed <code>.../lux/examples/intro.lux</code> file:</p>

<blockquote>
<pre><code>[doc Test of single and multi line regular expressions]

# Assign a global variable which is accessible in all shells
[global file=removeme.txt]

# Start a shell
[shell single]
  # Send text to the active shell
  !echo foo
  # Match output from the active shell
  # The terminal echoes all input and here we match on the echoed input
  ?echo foo

# Start yet another shell (and make it the active one)
[shell multi]
  # Create a file where bar and baz happens to be indented
  # Variables are
  !echo "foo"      &gt; $file
  !echo "    bar" &gt;&gt; $file
  !echo "  baz"   &gt;&gt; $file
  !echo "fum"     &gt;&gt; $file

  # Single line matches
  !cat $file
  ?foo
  ?bar
  # Don't bother of matching baz. All output between bar and fum is skipped.
  ?fum
  # Match the predefined shell prompt
  ?SH-PROMPT:

  # Multi line match. The first double quote char defines the first
  # column of the regexp. The indentation of bar and baz is significant.
  !cat $file
  """?
  foo
      bar
    baz
  fum
  SH-PROMPT:
  """

# Switch back to the first shell
[shell single]
  # Match the actual output from the echo command
  ?^foo

# Cleanup side effects. The cleanup section is always executed,
# regardless of the script succeeds or fails
[cleanup]
  !rm -f $file
  ?SH-PROMPT:
  # Match command exit status. Observe the double dollar sign which
  # escapes the dollar sign, implying "echo ==$$?==" to be sent to
  # the shell.
  !echo ==$$?==
  ?^==0==
</code></pre>
</blockquote>

<h2>A failing test case</h2>

<p>Test cases are executed until they succeed or fail. The script is
aborted at the first failure. Which may occur when the expected output
not has matched within the given timeout or when the failure pattern has
matched. A failure pattern is local to the given shell and will cause
abort when it matches.</p>

<p>The (match) timeout may explicitly be set within the script, in a
configuration file or given as a command line parameter. If the tests
are run both on fast machines and slow machines, it may be hard to set
an optimal timeout length. Then it may be appropriate to multiply the
timeout with different factors depending on the machine capabilities.
The config parameter called multiplier is used for this purpose. As
all configuration parameters, it can be given a host specific setting
or a architecture specific setting. Or even overridden on command
line.</p>

<p>As a test case may fail early or late in its execution the cleanup
code must cope with this. For example if a test case which normally
creates a file or starts a program fails before that point it must be
written in a manner so it does not fail during the cleanup.</p>

<p>Lux collects data in various logs. Such as an event log, one log per
shell for stdin and stdout etc. But sometimes this is not enough. For
example logs from the SUT (System Under Test). Such logs should be
stored under $LUX<em>EXTRA</em>LOGS. The $LUX<em>EXTRA</em>LOGS is a built-in
variable containing a suitable directory path. The SUT can either be
configured to store its logs there or the logs may be copied during
cleanup. In order to only copy the logs at failure the cleanup code
may test on the variable $LUX<em>START</em>REASON which also is built-in. It
is set to "fail" if the (cleanup) shell was started after a failure
has encountered. These extra logs are located under lux log directory
and it can (also) be reached via an HTML link from the annotated event
log.</p>

<p>Snippet from the enclosed <code>.../lux/examples/fail.lux</code> file:</p>

<blockquote>
<pre><code>[doc Demonstrate a failure]

[global fail_pattern=[Ee][Rr][Rr][Oo][Rr]]
[global eprompt=\d+&gt;\s]

[doc2 Provoke a failure to get something interesting in the logs]

[shell calculator]
  -$fail_pattern|SH-PROMPT:
  !erl

  # Multi-line expect
  """?
  Eshell.*
  $eprompt
  """

  # Multi-line send
  """!
  2+3.
  6+7.
  """

  # Ignore output between 5 and 13
  ?5
  ?13

  # Shorten the match timeout as we deliberately will demo a fail
  [timeout 2]
  !5+13.
  # Next line will fail
  ?19

[cleanup]
  # Save logs at fail
  """!
  if [ "$LUX_START_REASON" = "fail" ]; then
    mkdir -p $LUX_EXTRA_LOGS/erl;
    cp -r ./logs/* $LUX_EXTRA_LOGS/erl;
  fi; true
  """
  ?SH-PROMPT:
</code></pre>
</blockquote>

<h2>Warnings and avoiding failures</h2>

<p>At startup lux imports all environment variables as global lux
variables. making them accessible vith <code>$var</code> syntax. Variables may
also be set in architecture or host specific configuration files. This
may be useful when certain test cases only can be run on some hosts
due to missing libraries, lack of memory etc.</p>

<p>Test cases may be skipped by testing of a certain variable is set at
all <code>[config skip=VAR]</code> or is set to a certain value <code>[config
skip=VAR=val]</code>.  The inverse is also possible, by using <code>[config
skip_unless=VAR]</code> and <code>[config skip_unless=VAR=val]</code> respectively.</p>

<p>During development when some test cases are unstable it is possible to
classify those as unstable. This implies that they are run but instead
of reporting a failed test case as failure it is reported as a
warning.</p>

<p>Here are few examples of a few config settings which may be useful in
a heterogeneous lab environment:</p>

<p>One config file <code>.../lux_config/Linux-i686.luxcfg</code></p>

<blockquote>
<pre><code>[config var=MAKE=make]
[config var=USE_VALGRIND=true]
[config multiplier=1000]
</code></pre>
</blockquote>

<p>and another <code>.../lux_config/SunOS-sun4u.luxcfg</code></p>

<blockquote>
<pre><code>[config var=MAKE=gmake]
[config var=TEST_SUNOS=true]
[config var=SKIP_JAVA=true]
[config var=USE_VALGRIND=false]
[config multiplier=3000]
</code></pre>
</blockquote>

<p>Snippet from the enclosed <code>.../lux/examples/require_fail.lux</code> file:</p>

<blockquote>
<pre><code>[doc Demonstrate an error]

[config require=YADA_MAKE]

[shell setup]
  !${YADA_MAKE} start

[cleanup]
  !${YADA_MAKE} stop
</code></pre>
</blockquote>

<p>Snippet from the enclosed <code>.../lux/examples/warning.lux</code> file:</p>

<blockquote>
<pre><code>[doc Demonstrate a warning]

[global foo=bar]
</code></pre>
</blockquote>

<p>Snippet from the enclosed <code>.../lux/examples/skip.lux</code> file:</p>

<blockquote>
<pre><code>[doc Demonstrate a skipped test]

[doc2 Show examples of config settings]

[config skip=SKIP_JAVA]
[config skip_unless=TEST_SUNOS]

[shell foo]
  ?bar
</code></pre>
</blockquote>

<p>Snippet from the enclosed <code>.../lux/examples/unstable_warn.lux</code> file:</p>

<blockquote>
<pre><code>[doc Demonstrate an unstable test]

[config unstable_unless=TEST_DEVELOP]

[shell foo]
  [timeout 1]
  ?bar
</code></pre>
</blockquote>

<p>Here follow the output from the enclosed example test suite under
<code>.../lux/examples</code>.</p>

<p>Evaluate <code>lux examples</code></p>

<blockquote>
<pre><code>.../lux&gt; lux examples
summary log       : /Users/hmattsso/dev/lux/lux_logs/run_2023_10_25_13_33_18_81235/lux_summary.log
test case         : examples/calc.lux
progress          : ..:...:.:...:..:.:.:....:.:..:.:..:..(....:..:.:.:.:.61????61)19
result            : FAIL at line 19:61 in shell calc
expected*
\d+&gt;\s.*\.
(.*)
\d+&gt;\s
actual match_timeout

[A[J1&gt; 
.. 2+1.
3
2&gt; 
diff
- \d+&gt;\s.*\.
- (.*)
- \d+&gt;\s
+ 
+ [A[J1&gt; 
+ .. 2+1.
+ 3
+ 2&gt; 

test case         : examples/fail.lux
progress          : ..:..:..:.:...:.:..:.:.:..:..:.:.:....:..:..32C..:..:..:..:..:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.
result            : FAIL at line 32 in shell calculator
expected*
19
actual match_timeout

3&gt; 5+13.
18
4&gt; 
diff
- 19
+ 
+ 3&gt; 5+13.
+ 18
+ 4&gt; 

test case         : examples/intro.lux
progress          : ..:..:..:..:.:..:....:..:..:.:..:.:..:..:.:..:.:..:.:..:.:.....:..:.:.:....c.:.....:..:..:..:.:..:.:..:.:..:.
result            : SUCCESS
test case         : examples/loop.lux
progress          : ..:...:.:.((.:..:.:.)(.:.:..:.)(.:.:..:.))((..:.:.:.)(.:.:..:.)(.:..:.:.)(.:.:..:.)(.:.:..:.))((.:.:..:.)(..:.:.:.)(.:..:.:.)(.:..:.)(.:.:..:.)(.:.:..:.)(.:.:..:.)(.:..:.))...:..:.:..:..:.:..:.:..:.:..:.:...:...:.:.:.((.i=1..:..:.:.:..z)(z..i=2...:.:.:.:..z)(z..i=3..:..:.:..z)(:...i=4..:.:..:.):.).c........:..:..:.:..:..:.:.
result            : SUCCESS
test case         : examples/loop_fail.lux
progress          : ..:..:..:.((.i=1..:..:.:..z)(z..i=2..:...z)(z..i=3...:.:.:..z))+5
result            : FAIL at line 5 in shell break
expected*

actual Loop ended without match of break pattern "THIS WILL NEVER MATCH"

diff


test case         : examples/require_fail.lux
result            : FAIL as required variable YADA_MAKE is not set
test case         : examples/skip.lux
result            : SKIP as variable TEST_SUNOS is not set
test case         : examples/unstable_warn.lux
progress          : ..:...:.:.:....7
warning           : 8: FAIL but UNSTABLE as variable TEST_DEVELOP is not set
result            : WARNING at line 7 in shell foo
expected*
bar
actual match_timeout

diff
- bar
+ 

test case         : examples/warning.lux
progress          : W
warning           : 3: Trailing whitespaces
result            : WARNING
successful        : 2
skipped           : 1
examples/skip.lux:6
warnings          : 2
examples/unstable_warn.lux:8 - FAIL but UNSTABLE as variable TEST_DEVELOP is not set
examples/warning.lux:3 - Trailing whitespaces
failed            : 4
examples/calc.lux:19:61 - match_timeout
examples/fail.lux:32 - match_timeout
examples/loop_fail.lux:5 - Loop ended without match of break pattern "THIS WILL NEVER MATCH"
examples/require_fail.lux:3 - FAIL as required variable YADA_MAKE is not set
summary           : FAIL
file:///Users/hmattsso/dev/lux/lux_logs/run_2023_10_25_13_33_18_81235/lux_summary.log.html
.../lux&gt; echo $?
1
</code></pre>
</blockquote>

<p><a name="hardening"/></p>

<h1>Hardening test cases</h1>

<p>Here follows some advices to make the test cases more reliable. In
general we want to avoid race conditions and intermittent failures. We
do also want our test cases to not introduce problems for other test
cases.</p>

<p><strong>?</strong>  </p>

<p>Remove all usage of empty <code>?</code> commands. This command empties the
output streams (<code>stdout</code>, <code>stderr</code>). Already received output is
discarded. This is a <strong>very error prone</strong> command which easily may
cause race conditions and intermittent fails. The command is kept for
backwards compatibility but should really not be used. It is a very
good candidate for deprecation.</p>

<p>Do also look out for unintended emtying of the output streams. Ensure
that there are no <code>Empty multi-line expect command</code> warnings. E.g.</p>

<pre><code>    """?

    """
</code></pre>

<p>is such an occurrence.</p>

<p><strong>[sleep XXX]</strong>  </p>

<p>Remove all usage of sleep commands. This is a <strong>very error prone</strong>
command which easily may cause race conditions and intermittent
fails. In rare situations it may be valid to sleep a while to ensure
proper timing. But it should <strong>never</strong> be used as a safety action when
it is hard to find proper synchronization points. Such usage is error
prone.</p>

<p><strong>Prompt synchronization</strong>  </p>

<p>Always match on prompts, e.g. <code>?SH-PROMPT:</code>. Especially when multiple
commands are executed it is important to have well defined
synchronization points to avoid sending input when the system under
test not is ready for it yet. It is very easy to end up with
intermittent problems when not matching on all prompts. Further it can
be very confusing when performing post mortem analysis of failing
scripts, possibly comparing successful runs with failing dito.</p>

<p>It may even be the case that different environments might have so
different properties that a success/failure on one platform can be a
failure/success on another and that missing prompt synchronisation is
the cause for this.</p>

<p><strong>Protect the stdin</strong>  </p>

<p>Some commands sent to the system under test may read from <code>stdin</code>. To
suppress this behaviour the <code>/dev/null</code> can be piped to their <code>stdin</code>
to take care of that. The Java build system <code>ant</code> is known for this
unwanted behaviour. It can be extremely frustrating to find out that
this is the problem as it sometimes may work if you are lucky with the
timing. Often this problem occurs in conjunction with poor
synchronization points. Proper matching of prompts (mentioned above)
can avoid this problem altogether, making the tests more robust
avoiding intermittent race condition related problems.</p>

<p>An example. Assume the following script where we use <code>make</code> to compile
Java code using the <code>ant</code> tool</p>

<pre><code>    !make
    !ls
    ?SH-PROMPT:
</code></pre>

<p>The infamous <code>ant</code> tool will consume characters written to its
<code>stdin</code>. If the characters <code>ls</code> are sent to the shell port after <code>ant</code>
is done it will be the shell that reads and executes <code>ls</code>. This is
what we intended. But if Lux sends <code>ls</code> while <code>ant</code> still is executing
it will be <code>ant</code> that reads those characters and no <code>ls</code> will be sent
to the shell. So depending on the timing, the script may do what is
intended, or not. In order to fix this race condition we can re-write
the script as</p>

<pre><code>    !make &lt; /dev/stdin
    !ls
    ?SH-PROMPT:
</code></pre>

<p>ensuring <code>ant</code> to not read anything from <code>stdin</code> at all. An even more
robust solution is to also match the prompt after <code>make</code></p>

<pre><code>    !make &lt; /dev/stdin
    ?SH-PROMPT:
    !ls
    ?SH-PROMPT:
</code></pre>

<p><strong>Fix all timer related warnings</strong>  </p>

<p>Once all timers have been adjusted to get rid of the timer related
warnings it is time to harden the script even further by setting some
configuration parameters to more challanging values. For example</p>

<pre><code>    --risky_threshold=0.60
    --sloppy_threshold=0.000001
</code></pre>

<p><strong>Change the timing of the scripts</strong>  </p>

<p>Use the <code>--flush_timeout</code> to cause test cases with active
fail_patterns to fail when the <code>[cleanup]</code> section is entered. The
effect is similar to adding a <code>[sleep XXX]</code> just before <code>[cleanup]</code>.
Default is <code>--flush_timeout=0</code>. Start with <code>--flush_timeout=100</code> to
ultimately use <code>--flush_timeout=1000</code>. The <code>--flush_timeout</code> is used
to control how long Lux should wait for more output before
(automatically) resetting the fail pattern when entering zombie
mode. Resetting the fail pattern implies a final try to match the
pattern before it is actually reset. Besides finding out poorly
written scripts it may be useful with a long(er) setting to see more
output during post mortem analysis.</p>

<p>The <code>--poll_timeout</code> controls how long Lux should wait for more output
from a shell. Shells produces output in many small chunks. When the
first chunk is received Lux looks for consecutive chunks merging them
together into a bigger chunk before trying to match the new
data. Using a <code>--poll_timeout</code> bigger than 0 (milliseconds) increases
the receive window causing more data to be collected before trying to
match the regexps. This typically can cause scripts with poor
synchronization points to fail.</p>

<p>Default is <code>--poll_timeout=0</code>. It reasonable to try with
<code>--poll_timeout=10</code> and when the scripts are adjusted to cope with
this setting it may be time to use the more challanging value<code>
--poll_timeout=100</code>.</p>

<p>The special value <code>--poll_timeout=-1</code> causes Lux to not wait for
consecutive chunks at all before trying to match the regexps. After an
unsuccessful match Lux is looking for next small chunk of data. This
setting may catch open ended regexps, i.e. regexps ending with a
variable sized pattern (such as <code>.*</code>). An open ended regexp will in
some runs match more data than in others. This can cause strange
intermittent problems.</p>

<p><strong>[cleanup]</strong>  </p>

<p>It is good hygiene to ensure that each test case does not affect the
outcome of other test cases. Respect your collegues by not making your
test cases to cause theirs to fail.</p>

<p>Use the <code>[cleanup]</code> section to cleanup side-effects that may affect
consecutive test cases.</p>

<p>It is safe to assume that the cleanup code always is run. If there is
a need to abort a run do never use control-c to abort. Enter the
command <code>q</code> or <code>quit</code> in the Lux debugger instead to abort the test
case in a controlled manner by executing the cleanup code. <code>q s</code> or
<code>quit suite</code> may be used to safely abort all test cases int the suite.</p>

<p>In order to be able to perform post mortem analysis some side effects
needs to be kept. But ensure that these do not affect subsequent test
cases. A god praxis is to copy logs, and what not, to the directory
named in the environment variable <code>LUX_EXTRA_LOGS</code>. Each test case gets
its unique directory path in the variable. Your script needs to create
the directory if it is needed. The extra logs are kept among the other
Lux logs.</p>

<p>Do also run test cases multiple times without any <code>make clean</code> in
between. This is to ensure that possible resets in the beginning of
the test case works as intended. It can be very confusing to run
other peoples test cases when they only work once.</p>

<p><strong>[newshell Name]</strong> <br />
Using the <code>[newshell Name]</code> command instead of <code>[shell Name]</code> when
creating a new shell can avoid the common pitfall of misspelling a
shell name. <code>[shell MisspelledName]</code> will just go ahead and create
yet another shell even if it was unintended. Using <strong>--newshell</strong> will
require all shells to be created with <code>newshell</code> and <code>shell</code> can only
be used to switch to an already existing shell.</p>

<p>It may not be realistic to change all your existing scripts to use
<code>newshell</code>, but you can at least use it for all new scripts.</p>

<p><strong>Compensate for timing differences of heterogeneous hardware</strong>  </p>

<p>The <code>--multiplier</code> setting is intended to be used to handle the
problem with heterogeneous hardware where some machines are faster than
others. The <code>[timeout XXX]</code> used in the scripts can work well on one
machine but cause the script to fail on another. Instead of boosting
the timeout for all machines the <code>--multiplier</code> can be used. By
default it is set to 1000 meaning that <code>[timeout 42]</code> actually is 42
seconds. Perhaps it is easier to think in milliseconds. 42 means
42*1000=42000 milliseconds. By changing the multiplier to a higher
value, such as 3000, will cause Lux to use longer timeouts. E.g.
42*3000=126000 milliseconds.</p>

<p>This should be done in the architecture specific <code>.luxcfg</code> config
files or in a host specific <code>.luxcfg</code> config file. By doing this there
will be adapted timers per architecture or per host.</p>

<p><strong>[config unstable=XXX]</strong> <br />
<strong>[config unstable_unless=XXX]</strong>  </p>

<p>Respect your collegues by withdrawing your intermittently failing test
cases from the test results of the group. Use these constructs to mark
those test cases as unstable. They will then be run but only cause a
warning if they fail.</p>

<pre><code>    [config unstable=XXX]
    [config unstable_unless=XXX]
</code></pre>

<p><a name="warnings"/></p>

<h1>Warnings</h1>

<p>Lux emits various warnings. Some are built-in and others are
configurable. Some are emitted during validation and others during
execution.</p>

<p>Here follows a few configuration pararamenters that may be useful in
this context:</p>

<p><strong>--fail_when_warning</strong> <br />
is used to make a run fail if there are any warnings.</p>

<p><strong>--mode=validate</strong> <br />
is used to validate the scripts and display some warnings. The scripts
are not executed.</p>

<p><strong>--skip_skip</strong> <br />
causes all skip statements to be ignored. The skip statements are
mainly used to handle architecture specific quirks and are not really
applicable during validation. So if you want validation of all scripts
regardless of their skip settings you may need to use <code>--skip_skip</code>.</p>

<p><strong>--mode=dump</strong> <br />
displays the internal form of the main script and its include files as
well as warnings.</p>

<p><strong>--mode=expand</strong> <br />
expands the source code of the script and its include files, this
setting may help you to locate where macros and stuff are defined.</p>

<h2>Warnings emitted during validation</h2>

<p><strong>Warning: Empty multi-line XXX command</strong>  </p>

<p><code>XXX</code> denotes a type of command, such as <code>send</code>, <code>expect</code>, <code>fail</code> etc.</p>

<p>Writing an empty multi-line command like</p>

<pre><code>    """!
    """
</code></pre>

<p>or</p>

<pre><code>    """!

    """
</code></pre>

<p>can be very confusing as it may not be obvious to everyone that both
those constructions are equivalent to</p>

<pre><code>    !
</code></pre>

<p>which means that only one single newline is sent. Use the short form
<code>!</code> of the command for clarity. An even worse use of empty multi-line
commands is</p>

<pre><code>    """?
    """
</code></pre>

<p>and</p>

<pre><code>    """?

    """
</code></pre>

<p>which is equivalent to</p>

<pre><code>    ?
</code></pre>

<p><strong>But that is a totally different command!</strong> What looks like an empty
expect command is in fact a command which empties the output streams
(<code>stdout</code>, <code>stderr</code>). Already received output is discarded. If this is
the intended behaviour it should definitely be rewritten to use the
short form <code>?</code> of the command for clarity. Avoiding this (mis)feature
is however the best solution. At a first look it seems more useful
than it is. It often causes unexpected race patterns.</p>

<p><strong>Warning: Empty send command</strong>  </p>

<p>As you probably have concluded, both</p>

<pre><code>    """~
    """
</code></pre>

<p>and</p>

<pre><code>    """~

    """
</code></pre>

<p>are equivalent to</p>

<pre><code>    ~
</code></pre>

<p>which effectively is a no-op, which most likely is unintended. The
latter is reported as this particular warning, while the first two are
reported as a <code>*Warning: Empty multi-line send command</code> warning.</p>

<p><strong>Warning: Macro name XXX contains whitespace</strong>  </p>

<p>The whitespaces should be removed from the macro name in order to
avoid confusion with other macros.</p>

<p><strong>Warning: Trailing whitespaces</strong>  </p>

<p>The command ends with whitespaces which may be very confusing as the
whitespaces are hard to see.</p>

<p><strong>Warning: Missing summary doc (disabled for now)</strong>  </p>

<p>All scripts should document their purpose. This is done with the
<code>[doc]</code> statement. This warning is displayed when there is no <code>doc</code>
statement in the beginning of the script. The statement can be a
single line <code>doc</code> statement or a multi-line <code>[doc]</code> ... <code>[enddoc]</code>
block.</p>

<p>This particular warning is disabled for now.</p>

<p><strong>Warning: Empty doc text</strong>  </p>

<p>This warns for a <code>[doc ]</code> statement only containing whitespaces. Write
a proper documentation or remove the statement.</p>

<p><strong>Warning: Missing summary line</strong>  </p>

<p>This is a warning only emitted for multi-line documentation with
<code>[doc]</code> ... <code>[enddoc]</code> blocks.</p>

<p>The first line in the documentation block is intended to be a short
catchy summary line. Add that line or use the short form of the
command.</p>

<p><strong>Warning: Empty line expected after summary line</strong>  </p>

<p>This is a warning only emitted for multi-line documentation with
<code>[doc]</code> ... <code>[enddoc]</code> blocks.</p>

<p>The second line in the documentation block is intended to be an empty
line separating the first catchy summary line from the more detailed
documentation.  Add that line and more detailed documentation or use
the short form of the command.</p>

<p><strong>Warning: More documentation lines expected after empty line</strong>  </p>

<p>This is a warning only emitted for multi-line documentation with
<code>[doc]</code> ... <code>[enddoc]</code> blocks.</p>

<p>Following the second line in the documentation block meaty detailed
documentation is expected. Write some or use the short form of the
command.</p>

<h2>Warnings emitted during execution</h2>

<p>All warnings detected during validation may also be emitted during
execution.</p>

<p><strong>Warning: Infinite timer</strong>  </p>

<p>There is no timeout set at all. The <code>suite_timeout</code>, the
<code>case_timeout</code> and the match timeout are all set to <code>infinity</code>. This
means that the script may run forever. Set an explicit timeout to get
a decent behaviour.</p>

<p><strong>Warning: case_timeout > suite_timeout</strong>  </p>

<p>The <code>case_timeout</code> is greater than the <code>suite_timeout</code>. Adjust the
timeouts.</p>

<p><strong>Warning: Match timeout > test case_timeout</strong>  </p>

<p>The match timeout is greater than the <code>case_timeout</code>. Adjust the
timeouts.</p>

<p><strong>Warning: Match timeout > test suite_timeout</strong>  </p>

<p>The match timeout is greater than the <code>suite_timeout</code>. Adjust the
timeouts.</p>

<p><strong>Warning: Risky timer XXX % of max</strong>  </p>

<p>The timer is near to time out. It used <code>XXX</code> percent of the max
value. Increase the timeout to avoid intermittent failures.</p>

<p>By default the threshold is 85%, but it can be configured to something
else than</p>

<pre><code>    --risky_threshold=0.85
</code></pre>

<p><strong>Warning: Sloppy timer &lt; XXX ppb of max</strong>  </p>

<p>The timer is insanely large. Only a <code>XXX</code> fraction of the timer was
used. The timer is very likely to be over-dimensioned. If the duration
of something for example fluctuates between one and one million there
are something that is strange. Very large timers may cause unwanted
side effects. Such as Jenkins killing the job. Or that a failure takes
unreasonable long time to wait for when running manually etc.</p>

<p>By default the threshold is 1 part-per billion (ppb), but it can be
configured to something else than</p>

<pre><code>    --sloppy_threshold=0.000000001
</code></pre>

<p><strong>Warning: Shell XXX exited prematurely with status=YYY and posix=ZZZ</strong>  </p>

<p>The shell exited in an uncontrolled manner. Normally there is no need
to exit shells explicitly. But sometimes it may be neccessary. If the
shell is intended to exit, its expected exit status should be matched
to ensure that it exited properly. To ensure a successful exit,
i.e. with the status set to <code>0</code>, <code>[endshell 0]</code> should be used to
check that the shell exited properly. If any exit status is accepted
<code>[endshell .*]</code> can be used. The regexp can also be used to explicit
match one ore more non-successful exit statuses.</p>

<p><strong>Warning: FAIL at XXX in shell YYY</strong>  </p>

<p>The script fails at line <code>XXX</code>, but it <strong>also</strong> fails during
<code>[cleanup]</code>. The fail in the cleanup is the one that is visible in the
result of the script. The previous fail at <code>XXX</code> is somewhat hidden
and only visible as this warning. (Besides in the nitty gritty details
of the event log.) The reason for giving the cleanup fail precedence
over a "normal" fail is that bugs in the cleanup can cause severe
problems in the test environment with subsequent scripts failing.
There is good hygiene to ensure the cleanup code to be rock solid.</p>

<p><strong>Warning: Fail but UNSTABLE as variable XXX is set</strong>  </p>

<p>This test case fails. But it does also have a <code>[config unstable=XXX]</code>
statement causing the fail to be transformed into a warning. This
construct is intended to be used for buggy scripts which fails
intermittently. It can also be used during development where you want
the script to be run even though it is not fully functional
yet.</p>

<p><strong>Warning: Fail but UNSTABLE as variable XXX is not set</strong>  </p>

<p>This test case fails. But it does also have a <code>[config unstable_unless=XXX]</code>
statement causing the fail to be transformed into a warning. This
construct is intended to be used for buggy scripts which fails
intermittently. It can also be used during development where you want
the script to be run even though it is not fully functional
yet.</p>

<p><strong>Warning: Variable name XXX contains whitespace</strong>  </p>

<p>The whitespaces should be removed from the variable name in order to
avoid confusion with other variables.</p>

<p><strong>Warning: Shell name XXX contains whitespace</strong>  </p>

<p>The whitespaces should be removed from the shell name in order to
avoid confusion with other shells.
<a name="../INSTALL"/></p>

<h1>Installation</h1>

<p>On MacOS, Lux can be installed with</p>

<blockquote>
<pre><code>brew tap hawk/homebrew-hawk
brew install lux
</code></pre>
</blockquote>

<p>It will install Erlang and whatever else Lux needs.</p>

<p>Debian/Ubuntu</p>

<blockquote>
<pre><code>sudo apt-get install erlang erlang-dev
wget https://github.com/hawk/lux/releases/download/lux-2.1/lux_2.1-1.deb
sudo dpkg -i lux_2.1-1.deb
</code></pre>
</blockquote>

<p>Arch Linux</p>

<blockquote>
<pre><code>mkdir tmp
cd tmp
wget https://github.com/hawk/lux/releases/download/lux-2.1/PKGBUILD
makepkg -si
</code></pre>
</blockquote>

<p>TLDR;</p>

<h2>Prerequisites</h2>

<p>The following software is required:</p>

<ul>
<li><p>On BSD based systems, GNU Make is required.</p></li>
<li><p>The tool <strong>Lux</strong> is implemented with <strong><a href="http://www.erlang.org/" title="Erlang programming language">Erlang/OTP</a></strong> and its
runtime system must be installed in order to build the tool. Install
<code>Erlang/OTP</code> from <a href="http://www.erlang.org/" title="Erlang programming language">source</a> or use <a href="https://www.erlang-solutions.com/downloads/download-erlang-otp" title="Pre-built packages at Erlang Solutions">pre-built packages</a>:</p></li>
</ul>

<blockquote>
<pre><code>brew install erlang
</code></pre>
</blockquote>

<ul>
<li>or</li>
</ul>

<blockquote>
<pre><code>sudo apt-get install erlang erlang-dev
</code></pre>
</blockquote>

<ul>
<li><p>By installing the <code>erlang</code> package most of the Erlang apps needed by
Lux will be installed automatically. But on some systems there are
additional Erlang packages which may be needed when using more
exotic features, such as debugging and developing Lux itself:</p>

<ul>
<li><code>--internal_debug</code> requires <code>debugger</code>+<code>wx</code></li>
<li><code>--suite_trace</code>    requires <code>runtime_tools</code></li>
<li><code>--event_trace</code>    requires <code>runtime_tools</code>+<code>et</code>+<code>wx</code></li>
<li><code>--reltool</code>        requires <code>reltool</code></li>
</ul></li>
<li><p>Building Lux using <code>--make</code> requires <code>tools</code>. (Avoid this kind of build.)</p></li>
<li><p>Installation of Lux as standalone (using <code>--install</code>) requires <code>reltool</code>.</p></li>
<li><p>Testing of Lux itself requires <code>tools</code>.</p></li>
<li><p><code>--history</code> require may <code>inets</code>. But only when the logs are referred
to by using URL's. Using local log files does not require <code>inets</code>.</p></li>
<li><p>The documentation is pre-built. Re-generation of the documentation
requires <strong><a href="http://www.daringfireball.net/projects/markdown" title="Markdown">Markdown</a></strong>.</p></li>
</ul>

<h2>Instructions</h2>

<p>Lux can be <code>downloaded</code> from GitHub with</p>

<blockquote>
<pre><code>git clone git@github.com:hawk/lux.git
cd lux
</code></pre>
</blockquote>

<p>Lux is built with</p>

<blockquote>
<pre><code>autoconf
./configure
make
</code></pre>
</blockquote>

<p>When this is done you have a system which can run Lux with</p>

<blockquote>
<pre><code>bin/lux &lt;SOME PARAMS&gt;
</code></pre>
</blockquote>

<p>But you may also install Lux somewhere by using</p>

<blockquote>
<pre><code>make install
</code></pre>
</blockquote>

<p>By default (that is when ./configure has been invoked without
parameters), Lux will be installed under /usr/local. It is effectively
the same as invoking</p>

<blockquote>
<pre><code>./configure --prefix=/usr/local --exec_prefix=/usr/local --bindir=/usr/local/bin --sysconfdir=/usr/local/etc
</code></pre>
</blockquote>

<p><code>make install</code> does also accept various parameters which overrides the
ones given to <code>./configure</code>. Such as</p>

<blockquote>
<pre><code>make prefix=/usr/local exec_prefix=/usr/local bindir=/usr/local/bin sysconfdir=/usr/local/etc install
</code></pre>
</blockquote>

<p>and those parameters may be combined with</p>

<blockquote>
<pre><code>make DESTDIR=/my/staging/area install
</code></pre>
</blockquote>

<h2>Standalone installation</h2>

<p>When building Lux an Erlang/OTP system must be available.</p>

<p>By default that Erlang/OTP system is also used when running Lux.</p>

<p>But it is possible to perform an <code>standalone installation</code> of Lux
where the Lux installation is bundled with Erlang/OTP. This means that
you may in fact uninstall the Erlang/OTP system used for building Lux
and still be able to run Lux as it is self-contained with its own
Erlang/OTP runtime system. A standalone installation is performed with</p>

<blockquote>
<pre><code>mkdir -p &lt;TARGETDIR&gt;
bin/lux --install &lt;TARGETDIR&gt;
</code></pre>
</blockquote>

<p>The installed standalone system may be re-located if needed.</p>

<h2>Obscure platforms</h2>

<p>On "obscure platforms" which have <code>Erlang/OTP</code> but lacks <code>autotools</code>,
<code>make</code> etc. it may still possible to build Lux with</p>

<blockquote>
<pre><code>bin/lux --make
</code></pre>
</blockquote>

<h2>Re-build the documentation</h2>

<p>Simply do</p>

<blockquote>
<pre><code>cd doc
make
open ../lux.html
</code></pre>

<p><a name="../AUTHORS"/></p>
</blockquote>

<h2>Original author</h2>

<ul>
<li>Håkan Mattsson</li>
</ul>

<h2>Contributors</h2>

<ul>
<li>Jan Lindblad (implemented a predecessor to Lux, called Qmscript, as a Python plugin to QMTest)</li>
<li>Sebastian Strollo (runpty)</li>
<li>Martin Björklund (Emacs mode)</li>
<li>Johan Bevemyr (LCS diff)
<a name="references"/></li>
</ul>

<h1>References</h1>

<ol>
<li><a href="https://github.com/hawk/lux" title="LUcid eXpect scripting">Lux - LUcid eXpect scripting</a>  </li>
<li><a href="http://www.nist.gov/el/msid/expect.cfm" title="Expect homepage">Expect homepage</a>  </li>
<li><a href="http://www.erlang.org/" title="Erlang programming language">Erlang programming language</a>  </li>
<li><a href="https://www.erlang-solutions.com/downloads/download-erlang-otp" title="Pre-built packages at Erlang Solutions">Pre-built Erlang packages</a>  </li>
<li><a href="http://www.erlang.org/doc/man/re.html#regexp_syntax" title="Erlang style regular expressions (re)">Erlang style regular expressions (re)</a>  </li>
<li><a href="http://www.pcre.org/" title="PCRE - Perl Compatible Regular Expressions">PCRE - Perl Compatible Regular Expressions</a>  </li>
<li><a href="http://www.erlang.org/doc/apps/reltool/index.html" title="Erlang release management tool">Erlang release management tool</a>  </li>
<li><a href="http://www.daringfireball.net/projects/markdown" title="Markdown">Markdown</a>  </li>
<li><a href="http://testanything.org/" title="Test Anything Protocol">TAP - Test Anything Protocol</a>  </li>
</ol>

</body>
</html>