Auto-generated TOC with https://imthenachoman.github.io/nGitHubTOC/.
- ATF installation
- Running simple atf test apps
- Integrate the ATF into your own application
- Automating ATF test execution
- [How to use the ATF in a "simulation-in-the-loop" setup using gazebo](#how-to-use-the-atf-in-a-simulation-in-the-loop-setup-using-gazebohttpgazebosimorg)
- How to use the ATF for benchmarking
- Manual test execution (without rostest)
see ATF installation.
Run all tests
cd ~/catkin_ws
catkin run_tests -j1
or run tests from a specific package (e.g. for atf_test
):
cd ~/catkin_ws
catkin run_tests atf_test -j1
or
roscd atf_test
catkin run_tests --this -j1
check test summary with
cd ~/catkin_ws
catkin_test_results
If all goes well, you should see the recorded bag files in /tmp/atf_test
:
ls /tmp/atf_test/data
rosbag info /tmp/atf_test/data/ts0_c0_r0_e0_s0_0.bag
rqt_bag /tmp/atf_test/data/ts0_c0_r0_e0_s0_0.bag
and the results in /tmp/atf_test/results_txt
:
cat /tmp/atf_test/results_txt/atf_result.txt
cat /tmp/atf_test/results_txt/ts0_c0_r0_e0_s0_0.txt
You an use the ATF plotter to visualize the results:
rosrun atf_plotter plot.py --help
Example usage:
rosrun atf_plotter plot.py plot-benchmark /tmp/atf_test/results_txt/atf_result_aggregated.bag
A python app with ATF looks as simple as
import atf_core
class Application:
def __init__(self):
# initialize ATF
self.atf = atf_core.ATF()
def execute(self):
# you can call start/(pause/purge)/stop for each testblock during the execution of your app
self.atf.start("testblock_all")
self.atf.start("testblock_1")
# Do something
self.atf.stop("testblock_1")
self.atf.start("testblock_2")
# Do something else
self.atf.stop("testblock_2")
self.atf.stop("testblock_all")
# finally we'll have to call shutdown() to tell the ATF to stop all recordings and wrap up
self.atf.shutdown()
if __name__ == '__main__':
rospy.init_node('test_app')
app = Application()
app.execute()
Alongside with the modifications in the app code, you need to create some configuration files. Please have a look at the atf_test
package within this repository or at the atf test apps repository for configuration references. Typically you will have the following files:
atf/
├── envs # directory for environment specific settings, each environment is configured in one <env_name>.yaml file
│ ├── env1.yaml
│ └── env2.yaml
├── robots # directory for robot specific settings, each robot is configured in one <robot_name>.yaml file
│ ├── robot1.yaml
│ └── robot2.yaml
├── testblocksets # directory for various testblocksets (collection of metrics and groundtruth data)
│ ├── testblockset1.yaml
│ └── testblockset2.yaml
├── tests # directory for test configurations, each test is configured in one <test_name>.yaml file
│ ├── test1.yaml
│ └── test2.yaml
└── test_generation_config.yaml # specifies how to generate test files and defines testsuites (combination of configurations for envs, robots, testblocksets and tests.
The last thing to modify is your package.xml
and CMakeLists.txt
. Add a test dependency to atf_core
to your package.xml
<test_depend>atf_core</test_depend>
and the following line to your CMakeLists.txt
which triggers the test generation and execution:
if(CATKIN_ENABLE_TESTING)
find_package(atf_core REQUIRED)
atf_test(atf/test_generation_config.yaml)
endif()
After a catkin build --force-cmake
you will find all the generated test files in the build
directory of your workspace, e.g. build/atf_test/test_generated
.
You can run the tests automatically or run the tests manually and finally check the test results.
not yet available
If you have your test app written and configuration setup as shown in the example above, there are new catkin targets which trigger the execution of tests in a package:
catkin atf_<YOUR_PACKAGE>_cleaning
: cleans all test artefacts (bag, json and yaml files)catkin atf_<YOUR_PACKAGE>_recording
: triggers thecleaning
target and all recording testscatkin atf_<YOUR_PACKAGE>_analysing
: triggers therecording
target and all analysing testscatkin atf_<YOUR_PACKAGE>_uploading
: triggers theanalysing
target and the uploading testcatkin atf_<YOUR_PACKAGE>
: triggers all tests in your packagecatkin run_tests
: triggers all tests in your catkin workspace
You can use the above targets to setup your continuous integration scripts.
An example using industrial_ci to integrate ATF into Travis CI can be found in this repository or the atf test apps repository. Have a look at the .travis.yml
file.
How to use the ATF in a "simulation-in-the-loop" setup using gazebo
If you'd like to run tests using gazebo, you just setup a launch file which starts all nodes and include that to your application.launch
. As catkin normally uses multiple threads for executing the tests, we'll need to limit that to only one concurrent job as running multiple gazebo instances at a time causes troubles. Thus run your tests with
catkin run_tests -j1
For travis scripts using industrial_ci you can set the PARALLEL_TESTS
environment variable:
env:
global:
- PARALLEL_TESTS=false
TBD
The tests can be generated by calling
catkin build --force-cmake atf_test
Record a single test
rosrun atf_core record_tests.py atf_test -t ts0_c0_r0_e0_s0_0
Record a subset of tests
rosrun atf_core record_tests.py atf_test -t ts0_c0_r0_e0_s0 --> record all iterations of test ts0_c0_r0_e0_s0_*
rosrun atf_core record_tests.py atf_test -t ts0 --> record all tests with ts0: ts0_c*_r*_e*_s*_*
rosrun atf_core record_tests.py atf_test -t c0*r0* --> record all tests with c0 and r0: ts*_c0_r0_s*_*
Record all tests
rosrun atf_core record_tests.py atf_test
Full list of arguments
rosrun atf_core record_tests.py -h
While tests are executed:
- you can watch the progress with
rosrun smach_viewer smach_viewer.py
- you can use RVIZ for visualization
rosrun rviz rviz
rosrun atf_core analyser.py atf_test
or for full result print
rosrun atf_core analyser.py atf_test -v
Full list of arguments
rosrun atf_core analyser.py -h
check results
cat /tmp/atf_test/results_txt/atf_result.txt
cat /tmp/atf_test/results_txt/ts0_c0_r0_e0_s0_0.txt