Introduction
---------------
The load automation framework is designed to allow flexibility and stability to traditional load testing strategies. This utility will allow you to dynamically configure your load testing environment. The framework is made up of 3 parts.

Driver Utility: Dynamically sets up your load environment, configuration and tests.
Load Testing API: ruby api to develop your load tests with
Load Application: the application that runs the load and generates the statistics, currently Tsung

Setup
---------------
To effectively run the load testing framework you'll need at least 2 machines, though more are supported.

Your driver machine is the machine you will be running the Driver Utility on. This machine can also be one of your client machines (described below).
Your client machine is the machine where your load is generated. You can add multiple machines to your client pool to distribute load generation.
Your server or System Under Test (SUT) is the machine running the web application you wish to test. This should not be a client as well, otherwise your statistics will be significantly flawed.

Driver
---------------
Unix/OSX machine
Requires Erlang v5.6.1+
Requires Tsung v1.3.1+ (see dependencies)
Requires Ruby v1.8.6+
rdoc required for documentation
Requires Perl v5.8.8+
Template.pm module required to generate html statistics
Checkout code: Testing Framework Builds
Environment variables:
DRB_PORT=9000

Client
---------------
Unix/OSX machine
Requires Erlang v5.6.1+
Requires Tsung v1.4.1+ (see dependencies)
User running Tsung needs ssh access to all clients with no passphrase
If running into errors starting remote client see Tsung FAQ for possible solutions

Server
---------------
Requires web server running application you wish to test. Port should be accessible by external machines.
Directory Structure
	Kuali-Sakai-Performance-Test-Automation-Framework/
		config/ (all configuration files for the framework)
			config.yaml
			import/ (data files for dynamic attributes)
				users.csv (list of username/password pairs)
				users.format (format for username/user_password)
			tests (the default location for .xml files created when executing driver.rb)
		doc/ (documentation generated by rdoc)
		lib/ (contains all API libraries)
			[Products] (contains API libraries for supported products)
		log/ (default location for log files and xml generated by drivery utility)
		suites/ (suite files the load framework has the option to run)
			[Products]
		tests/ (contains all tests that the suite files can call)
			[Products]
		driver.rb (Driver Utility)

Config
---------------
Inside the config directory you will find the following files:

- config.yaml

The config.yaml file consists of 3 distinct sections:

- The Products we currently support section contains the products that are currently supported. Lines starting with # are ignored. Example:
# Format: "product reference": "directory name in framework"
ks: 'ks'
oae: 'oae'

- The Directory listing for test data in use in the tests section is currently only partially used. It contains information that is extracted from the file for use in the tests. Lines starting with # are ignored. 

- The Agent List section of the config.yaml file contains all the browser agents available for use in your load configuration. The driver utility will read this file. You can add whatever agents you like in this file. Lines starting with # are ignored. Example:

# IE 7.0 - XP
Mozilla/5.0 (Windows; MSIE 7.0; Windows NT 6.0; en-US)
# IE 6.0 - Vista
Mozilla/4.0 (Windows; MSIE 6.0; Windows NT 6.0)
# FireFox 3.5.7 - Vista
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US)
# FireFox 3.5.6 - OSX 10.6 (Snow Leopard)
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6

Doc
---------------
After you have checked out the code/tag you can generate this documentation by running 'rdoc'. This folder and all the documentation for the entire framework will be generated for you. You can open index.html inside of the doc folder to navigate to all documentation.

Log
---------------
This is the default location for all log and xml files generated by the Driver Utility. You can specify your own output locations through the Driver Utility options.

Suites
---------------
This folder contains all the suite files available for the framework to run. You can add as many suite files as you want here, likewise you can give them any name that makes sense for you. The format of the suite files is:

test name,probability

Each suite can have multiple tests, but the probability for all tests must total 100. This gives you flexibility in how you want your sessions to be executed. Example:

authentication.rb,50
create_proposal.rb,25
approve_proposal.rb,25

The test name must refer to the exact filename of a test residing in the tests directory.

driver.rb
---------------
This is the Driver Utility, more on this below.

Driver Utility
---------------
Usage
Usage: driver.rb [OPTIONS]
    -h, --help                       Display help screen
    -c, --config FILE                path to xml config file for everything after <tsung> and before <sessions> : NOT CURRENTLY SUPPORTED
    -p, --product PRODUCT            supported products: ["ks", "oae"]
    -s, --suite SUITE                suite file
    -o, --output FILE                path to xml output
        --contexts PRIMARY_CONTEXT,SECONDARY_CONTEXT
                                     contexts for both primary and secondary servers
    -l, --log FILE                   path to log output from this framework
    -x, --execute                    start the load after generating the XML
    -v, --verbose                    dumps http traffic from tsung
        --tsung_log_level LEVEL      sets the log level for tsung. Available levels: emergency, critical, error, warning, notice (default), info, debug
    -d, --debug                      enable debug logging
    -t, --thinktime                  enable thinktime sleeps

Interactive Shell
---------------
When you initiate the Driver Utility you will enter the interactive shell to setup your configuration. There are specific examples in the Kuali-Sakai-Performance-Test-Automation-Framework/ directory.

The following is a generic example showing the interactive session:

./driver.rb
Your log file is here: /../Kuali-Sakai-Performance-Test-Automation-Framework/log/1318879134.log
Let's setup your test run config...

# Select the product that you are testing

What product are we testing? [select from product listing]

What suite are we testing?

#Enter the name of the suite you want to test
#the name of the suite that you choose must exist in the suites/[product] directory

What suite are we testing? test_suite

What do you want to name your output xml file?

#Name your output xml file

What do you want to name your output xml file? demo.xml
Your xml file will be here: /../Kuali-Sakai-Performance-Test-Automation-Framework/config/tests/demo.xml

#Specify your clients. Must be comma separated

What clients are you driving your tests from? [hostname1,hostname2,...] localhost

#Specify the server host you want to test

What primary servers do you want to test? [hostname1:port,hostname2:port,...] server.com:80

#Specify any secondary web application host

Secondary servers? [reference_name1:hostname1:port,reference_name2:hostname2:port,...] reference_for_this_server_that_is_used_in_the_api:secondary_host.com:80

#Specify the primary server context

What is the primary server context? context_for_primary_server

#Specify the secondary server context if there is a secondary server

What is the secondary server context? context_for_secondary_server

Let's setup your test scenario or phases, you can define multiple phases
Phase 1: How many minutes?

#A load test is made up of one or more test phases. The total duration of your load test will be the total duration of all of your phases. Specify how many minutes you want your first (and possibly only) test phase to be. You will have the option of adding more phases shortly.

Phase 1: How many minutes? 5
At what interval should users be created at this phase? (x seconds per user) 

#For this phase specify at what rate user sessions should be created. Any float value will do. To generate 1 user a second you'd specify 1, to generate 5 users every second you'd specify 0.2

At what interval should users be created at this phase? (x seconds per user) 0.2
Do you want to add another phase? [y/n]

#Specify whether you want to add another phase or not. If y you'll be prompted with same steps as in phase 1.

Do you want to add another phase? [y/n] n
Let's setup what user agents you'd like to test with
Here are the agents you can specify...
1: Mozilla/5.0 (Windows; MSIE 7.0; Windows NT 6.0; en-US)
2: Mozilla/4.0 (Windows; MSIE 6.0; Windows NT 6.0)
3: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US)
4: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6
Specify the number of the agent you'd like to use:

#Agent list is read in from the Agents List section of the config.yaml file in the config directory. Choose which agent you'd like to use. You'll be able to specify multiple agents and #probability of their usage.

Specify the number of the agent you'd like to use: 1
Specify the probability that this agent will be used: [1-100] 

#Specify the probability this agent will be used. Any value less than 100 will prompt you to add another agent. This will repeat until your total probability equals 100.

Specify the probability that this agent will be used: [1-100] 100
Do you want to insert user thinktime? [y/n]

#Thinktime causes the test to insert "sleeps" that simulate a real life user session.

Do you want to insert user thinktime? [y/n] y
Do you want to execute tsung now? [y/n] 

#Specify whether you want to run tsung now or not.

Do you want to execute tsung now? [y/n] y
Do you want to dump http traffic from tsung? [y/n] 

#Specify whether or not you want to dump http traffic. Only select yes if this is a debug run. This will greatly reduce performance.

Do you want to dump http traffic from tsung? [y/n] n

Test: login.rb
Probability: 100
login.rb: Logging in as admin/admin
login.rb: Logging out

Starting tsung with command [tsung -f /Users/test/work/Kuali-Sakai-Performance-Test-Automation-Framework/config/tests/demo.xml start]
Starting Tsung
"Log directory is: /Users/test/.tsung/log/20111017-19:55"

Statistics
---------------

Tsung Logs
---------------
In the log directory that was printed to stdout you will see the following files:

% ls
kctest.xml			tsung.log
match.log			[email protected]
kctest.xml: is the xml file used for the test. This is the same file as generated by the Driver Utility
match.log: contains logging info for match expressions inside of your test. If none are used then nothing will be here.
tsung.log: contains all the http statistics for the load run
tsung_controller[KULSTG:...].log: Contains logging info from the client controllers.

Generating Tsung HTML report
---------------
Tsung comes with a perl utility for generating html reports with graphing, the utility is called tsung_stats.pl and should be in the lib directory where you've installed Tsung. For example if you have Tsung installed in /opt/local/bin/tsung, then the utility will be in /opt/local/lib/tsung/bin/tsung_stats.pl.

To run just execute cd into your log directory and execute 'tsung_stats.pl'

% tsung_stats.pl 
creating subdirectory data 
creating subdirectory images 
creating subdirectory gnuplot_scripts 
warn, last interval (1) not equal to the first, use the first one (10)
No data for Match
No data for Event
No data for Async
No data for Errors

The script will generate 2 html files, both link to each other internally so just open one of them.

report.html
graph.html

You can get a detailed description of all stats here.

Tsung HTML Report Screenshots
---------------

System Resource Monitoring
---------------
Please refer to the System Resource Monitoring project located at the following location in github: https://github.com/kcampos/Open-Synchronized-System-Resource-Monitoring.git
---------------

FAQ
---------------
1. If you see the following error when invoking driver.rb, then add a "log" directory in the Kuali-Sakai-Performance-Test-Automation-Framework directory:
Kuali-Sakai-Performance-Test-Automation-Framework/lib/tsung-api.rb:277:in `initialize': No such file or directory - /Users/test/work/Kuali-Sakai-Performance-Test-Automation-Framework/log/1318877356.log (Errno::ENOENT)

License
---------------
Copyright 2012 rSmart Licensed under the
Educational Community License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may
obtain a copy of the License at
	
	http://www.osedu.org/licenses/ECL-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an "AS IS"
BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
or implied. See the License for the specific language governing
permissions and limitations under the License.