Skip to content

Commit

Permalink
Improve documentation and fixed typos
Browse files Browse the repository at this point in the history
  • Loading branch information
@en-milie
en-milie committed Apr 22, 2021
1 parent c88f703 commit 9c0f7aa
Showing 1 changed file with 41 additions and 36 deletions.
77 changes: 41 additions & 36 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,11 @@ You can use the following Maven command to build the project:

This will output a `cats.jar` file in the current directory. The file is an executable JAR that will run in Linux environments. Just run `chmod +x cats.jar` to make the file executable.

Note: You will need to configure Maven with a [Github PAT](https://docs.github.com/en/free-pro-team@latest/packages/guides/configuring-apache-maven-for-use-with-github-packages) with `read-packages` scope to get some dependencies for the build.
**Note:** You will need to configure Maven with a [Github PAT](https://docs.github.com/en/free-pro-team@latest/packages/guides/configuring-apache-maven-for-use-with-github-packages) with `read-packages` scope to get some dependencies for the build.

## Notes on Unit Tests

You may see some `ERROR` log messages while running the Unit Tests. Those are expected behaviour for testing the negative scenarios of the `Fuzzers`.

# Available commands
To list all available commands, run CATS with no arguments:
Expand All @@ -161,10 +165,6 @@ A minimal run must provide the Swagger/OpenAPI contract, and the URL address of

But there are multiple other arguments you can supply. More details in the [available arguments](#available-arguments) section.

## Notes on Unit Tests

You may see some `ERROR` log messages while running the Unit Tests. Those are expected behaviour for testing the negative scenarios of the `Fuzzers`.

## Notes on skipped Tests
You may notice a significant number of tests marked as `skipped`. CATS will try to apply all `Fuzzers` to all fields, but this is not always possible.
For example the `BooleanFieldsFuzzer` cannot be applied to `String` fields. This is why that test attempt will me marked as skipped.
Expand Down Expand Up @@ -225,11 +225,11 @@ Using some of these options a typical invocation of CATS might look like this:
To get a list of fuzzers just run `./cats.jar list fuzzers`. A list of all available fuzzers will be returned, along with a short description for each.

There are multiple categories of `Fuzzers` available:
- Field `Fuzzers`: which target request body fields or path parameters
- Header `Fuzzers`: which target HTTP headers
- HTTP `Fuzzers`: which target just the interaction with the service (without fuzzing fields or headers)
- ContractInfo `Fuzzers`: which checks the contract for API good practices
- Special `Fuzzers`: a special category which need further configuration and are focused on more complex activities like functional flow or security testing
- `Field Fuzzers` which target request body fields or path parameters
- `Header Fuzzers` which target HTTP headers
- `HTTP Fuzzers` which target just the interaction with the service (without fuzzing fields or headers)
- `ContractInfo Fuzzers` which checks the contract for API good practices
- `Special Fuzzers` a special category which need further configuration and are focused on more complex activities like functional flow or security testing

## Field Fuzzers
`CATS` has currently 28 registered Field `Fuzzers`:
Expand Down Expand Up @@ -478,8 +478,10 @@ You can run only these `Fuzzers` by supplying the `--checkContract` argument.
## Special Fuzzers
### CustomFuzzer
#### Writing Custom Tests
In some cases, the tests generated by CATS will not be sufficient for your situation. Using the `CustomFuzzer` you can supply custom values for specific fields. The cool thing is that you can target a single field, and the rest of the information will be populated by `CATs` using valid data, just like a 'happy' flow request.
It's important to note that `reference data` won't get replaced when using the `CustomFuzzer`. So if there are reference data fields, you must also supply those in the `CustomFuzzer`.
In some cases, the tests generated by CATS will not be sufficient for your situation.
Using the `CustomFuzzer` you can leverage the full power if CATS on not writing any code, while also supply custom values for specific fields.
The cool thing is that you can target a single field, and the rest of the information will be populated by `CATS` using valid data, just like a 'happy' flow request.
**It's important to note that `reference data` won't get replaced when using the `CustomFuzzer`. So if there are reference data fields, you must also supply those in the `CustomFuzzer`.**
The `CustomFuzzer` will only trigger if a valid `customFuzzer.yml` file is supplied. The file has the following syntax:

```yaml
Expand All @@ -498,7 +500,7 @@ The `CustomFuzzer` will only trigger if a valid `customFuzzer.yml` file is suppl
httpMethod: HTTP_NETHOD
```
Some things to note about the `customFuzzer.yml` file:
This is a description of the elements within the `customFuzzer.yml` file:
- you can supply a `description` of the test case. This will be set as the `Scenario` description. If you don't supply a `description` the `testNumber` will be used instead.
- you can have multiple tests under the same path: `test1`, `test2`, etc.
- `expectedResponseCode` is mandatory, otherwise the `Fuzzer` will ignore this test. The `expectedResponseCode` tells CATS what to expect from the service when sending this test.
Expand All @@ -521,6 +523,7 @@ As CATs mostly relies on generated data with small help from some reference data
We need a way to get some identifier from the POST call and send it to the GET call. This is now possible using the `CustomFuzzer`.
The `customFuzzerFile` can have an `output` entry where you can state a variable name, and its fully qualified name from the response in order to set its value.
You can then refer the variable using `${variable_name}` from another test case in order to use its value.

Here is an example:
```yaml
/pet:
Expand All @@ -539,6 +542,7 @@ Here is an example:
```

Suppose the `test_1` execution outputs:

```json
{
"pet":
Expand All @@ -555,7 +559,8 @@ When executing `test_2` the `id` parameter will be replaced with the `petId` var
- variables are visible across all custom tests; please be careful with the naming as they will get overridden

#### Verifying responses
The `CustomFuzzer` can verify more than just the `expectedResponseCode`. This is achieved using the `verify` element. This is an extended version of the above `customFuzzer.yml` file.
The `CustomFuzzer` can verify more than just the `expectedResponseCode`. This is achieved using the `verify` element. This is an extended version of the above `customFuzzer.yml` file.

```yaml
/pet:
test_1:
Expand Down Expand Up @@ -636,11 +641,11 @@ The file uses [Json path](https://github.com/json-path/JsonPath) syntax for all

This is what the `SecurityFuzzer` will do after parsing the above `securityFuzzerFile`:
- it will add the fixed value "My Pet" to all the request for the field `name`
- for each field specified in the `targetFields` i.e. `pet#id` and `pet#description` will create requests for each line from the `xss.txt` file and supply those values in each field
- if you consider the `xss.txt` sample file included in the `CATs` repo, this means that it will send 21 requests targeting `pet#id` and 21 requests targeting `pet#description` i.e. a total of 42 tests
- for each field specified in the `targetFields` i.e. `pet#id` and `pet#description` it will create requests for each line from the `xss.txt` file and supply those values in each field
- if you consider the `xss.txt` sample file included in the `CATs` repo, this means that it will send 21 requests targeting `pet#id` and 21 requests targeting `pet#description` i.e. a total of **42 tests**
- for each of these 42 tests, the `SecurityFuzzer` will expect a `200` response code. If another response code is returned, then `CATs` will report the test as `error`.

As an idea on how to create security tests, you can split the [nasty strings](https://github.com/minimaxir/big-list-of-naughty-strings) into multiple files of interest for your particular context.
As an idea on how to create security tests, you can split the [nasty strings](https://github.com/minimaxir/big-list-of-naughty-strings) into multiple files of interest in your particular context.
You can have a `sql_injection.txt`, a `xss.txt`, a `command_injection.txt` and so on. For each of these files, you can create a test entry in the `securityFuzzerFile` where you include the fields you think are meaningful for these types of tests.
(It was a deliberate choice (for now) to not include all fields by default.) The `expectedResponseCode` should be tweaked according to your particular context.
Your service might sanitize data before validation, so might be perfectly valid to expect a `200` or might validate the fields directly, so might be perfectly valid to expect a `400`.
Expand Down Expand Up @@ -759,10 +764,10 @@ You can also have the ability to send the same reference data for ALL paths (jus
all:
address#zip: 123
```
This will try to replace `address#zip` in **all** requests (if the field present).
This will try to replace `address#zip` in **all** requests (if the field is present).

## Removing fields
There are (rare) cases when some fields my not make sense together. Something like: if you send `firstName` and `lastName`, you are not allowed to also send `name`.
There are (rare) cases when some fields may not make sense together. Something like: if you send `firstName` and `lastName`, you are not allowed to also send `name`.
As OpenAPI does not have the capability to send request fields which are dependent on each other, you can use the `refData` file to instruct CATS to remove fields before sending a request to the service.
You can achieve this by using the `cats_remove_field` as a value for the fields you want to remove. For the above case the `refData` field will look as follows:

Expand All @@ -786,30 +791,30 @@ all:

This will add the `Accept` header to all calls and the `jwt` header to the specified paths.

# URL Params
You can use `--urlParams` to send values for placeholders inside the contract paths. For example, if your contract paths look like: `/service/{version}/pets`, you can run cats as:
`./cats.jar --contract=api.yml --server=http://localhost:8080 --urlParams=version:v1.0`

so that each fuzzed path will replace `version` with `v1.0`.

# Edge Spaces Strategy
There isn't a consensus on how you should handle situations when you send leading and trailing spaces or leading and trailing valid values within fields. One strategy for the service will be to trim these values and consider them valid, while some other services will just consider them to be invalid.
You can control how CATS should expect such cases to be handled by the service using the `--edgeSpacesStrategy` argument. You can set this to `error` or `success` depending on how you expect the service to behave:
- `error` means than the service will consider the values to be invalid, even if the value itself is valid, but has leading or trailing spaces.
- `success` means that the service will trim the value and validate it afterwards.
There isn't a consensus on how you should handle situations when you trail or prefix valid values with spaces.
One strategy will be to have the service trimming spaces before doing the validation, while some other services will just validate them as they are.
You can control how CATS should expect such cases to be handled by the service using the `--edgeSpacesStrategy` argument.
You can set this to `trimAndValidate` or `validateAndTrim` depending on how you expect the service to behave:
- `trimAndValidate` means that the service will first trim the spaces and after that run the validation
- `validateAndTrim` means that the service runs the validation first without any trimming of spaces

This is a **global setting** i.e. configured when CATS starts and all `Fuzzer` expects a consistent behaviour from all the service endpoints.

# URL Parameters
There are cases when certain parts of the request URL are parameterized. For example a case like: `/{version}/pets`. `{version}` is supposed to have the same value for all requests. This is why you can supply actual values to replace such parameters using the `urlParams` argument.
You can supply a `;` separated list of `name:value` pairs to replace the `name` parameters with their corresponding `value`. For example supplying `--urlParams=version:v1.0` will replace the `version` parameter from our example above with the value "v1.0".
There are cases when certain parts of the request URL are parameterized. For example a case like: `/{version}/pets`. `{version}` is supposed to have the same value for all requests.
This is why you can supply actual values to replace such parameters using the `--urlParams` argument.
You can supply a `;` separated list of `name:value` pairs to replace the `name` parameters with their corresponding `value`.
For example supplying `--urlParams=version:v1.0` will replace the `version` parameter from the above example with the value `v1.0`.

# Dealing with AnyOf, AllOf and OneOf
CATS also supports schemas with `oneOf`, `allOf` and `anyOf` composition. CATS wil consider all possible combinations when creating the fuzzed payloads.

# Dynamic values in configuration files
The following configuration files: `securityFuzzerFile, customFuzzerFile, refData` support setting dynamic values for the inner fields.
For now **the support only exists** for `java.time.*`, but more types of elements might come in the near future.
For now **the support only exists** for `java.time.*` and `org.apache.commons.lang3.*`, but more types of elements will come in the near future.

Let's suppose you have a date/date-time field and you want to set it to 10 days from now. You can do this by setting this as a value `T(java.time.OffsetDateTime).now().plusDays(10)`.
Let's suppose you have a date/date-time field, and you want to set it to 10 days from now. You can do this by setting this as a value `T(java.time.OffsetDateTime).now().plusDays(10)`.
This will return an ISO compliant time in UTC format.

A `customFuzzerFile` using this can look like:
Expand All @@ -835,7 +840,7 @@ You can also check the responses using a similar syntax and also accounting for
The syntax of dynamically setting dates is compliant with the [Spring Expression Language](https://docs.spring.io/spring-framework/docs/3.0.x/reference/expressions.html) specs.

# Running behind proxy
If you need to run CATS behind a proxy, you can supply the following arguments: `proxyHost` and `proxyPort`.
If you need to run CATS behind a proxy, you can supply the following arguments: `--proxyHost` and `--proxyPort`.
A typical run with proxy settings on `localhost:8080` will look as follows:

```bash
Expand All @@ -854,7 +859,7 @@ By default, CATS trusts all server certificates and doesn't perform hostname ver
For two-way SSL you can specify a JKS file (Java Keystore) that holds the client's private key using the following arguments:
- `--sslKeystore` Location of the JKS keystore holding certificates used when authenticating calls using one-way or two-way SSL
- `--sslKeystorePwd` The password of the `sslKeystore`
- `--sslKeyPwd` The password of the private key from the `sslKeystore`
- `--sslKeyPwd` The password of the private key within the `sslKeystore`

For details on how to load the certificate and private key into a Java Keystore you can use this guide: [https://mrkandreev.name/blog/java-two-way-ssl/](https://mrkandreev.name/blog/java-two-way-ssl/).

Expand Down Expand Up @@ -891,4 +896,4 @@ All custom files that can be used by CATS (`customFuzzerFile`, `headers`, `refDa
You can find some selector examples here: [JsonPath](https://github.com/json-path/JsonPath).

# Contributing
Please refer to [CONTRIBUTING.md](CONTRIBUTING.md)
Please refer to [CONTRIBUTING.md](CONTRIBUTING.md).

0 comments on commit 9c0f7aa

Please sign in to comment.