Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create API-Testing-Guidelines.md #117

Merged
merged 6 commits into from
Jan 24, 2024
Merged

Create API-Testing-Guidelines.md #117

Changes from 3 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
aa3f2fd
Create API-Testing-Guidelines.md
shilpa-padgaonkar Jan 8, 2024
57ef1a2
Update API-Testing-Guidelines.md
shilpa-padgaonkar Jan 9, 2024
2cbf0da
Update API-Testing-Guidelines.md
shilpa-padgaonkar Jan 12, 2024
cbfb886
Update documentation/API-Testing-Guidelines.md
shilpa-padgaonkar Jan 19, 2024
b026270
Update API-Testing-Guidelines.md
shilpa-padgaonkar Jan 22, 2024
0e98b46
Update documentation/API-Testing-Guidelines.md
shilpa-padgaonkar Jan 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 41 additions & 0 deletions documentation/API-Testing-Guidelines.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# API Testing Guidelines

## Table of Contents
1. [Introduction](#introduction)
2. [Test Cases contributions for APIs](#contribution)
3. [Location of feature file](#location)
4. [Best practices and recommendations](#recommendations)
7. [References](#references)

## Introduction <a name="introduction"></a>
This document captures guidelines for the API testing in CAMARA project. These guidelines are applicable to every API to be worked out under the CAMARA initiative.

## Test Cases contribution for APIs <a name="contribution"></a>
Based on the decision taken in the commonalities working group and as documented in [API-Readiness-Checklist](https://github.com/camaraproject/Commonalities/blob/main/documentation/API-Readiness-Checklist.md), a Gherkin feature file will be added to the main subproject repo and this will fullfil the minimum criteria of readiness w.r.t API test cases and documentation. If subprojects also intend to add test implementations, an aligned single implementation that is agreed amongst all provider implementers could be added to the main subproject repo. If no alignment is possible, each provider implementer will add the test implementation to their own repos.
shilpa-padgaonkar marked this conversation as resolved.
Show resolved Hide resolved

## Location of feature file <a name="location"></a>
The feature file will reside under
```
Subproject_Repository/code/Test_definitions/file_name.feature
for e.g. https://github.com/camaraproject/QualityOnDemand/blob/main/code/Test_definitions/QoD_API_Test.feature
```

## Best practices and recommendations <a name="recommendations"></a>

* One feature file per API is advisable so that all scenarios can be covered corresponding to the API & the corresponding resource.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should not impose one single .feature file per API, but one feature file per operation or API endpoint/resource. In general, it is not advisable to test several unrelated "API features" within the same .feature file. For closely related operations we may add scenarios in the same file testing crossed behaviour between several operations.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shilpa-padgaonkar following today Commonalities meeting I proposed to update this sentence but I'm not able to update the file. My proposal for line 25:
Granularity of the feature file must be decided at the project level but it is recommended to:

  • group in one file all scenarios for one given API capability (that can cover several endpoints).
  • provide several files when one CAMARA API (yaml) covers several independent functions that can be provided independently

Perhaps @jlurien could provide a better wording. I think we're aligned.

* Third Person pronoun usage in feature file is advisable as using the third person, conveys information in a more official manner.
* It is recommended to only have one When/Then in a feature file as per cucumber official documentation's recommendations. However, in case of complex scenarios, several when/then can be allowed.
* The recommended format for scenario identifier is shown below
```
'@'<(mandatory)name of the resource>_<(mandatory)number XX>_<(optional)short detail in lower case and using underscore “_” as the separator>)
for e.g. (@check_simswap_01_verify_swap_true_default_maxage).
```

* Recommendation is to have API literal request value (with example) but in case of exceptions, it is fine to use ready payload as well.


## References <a name="references"></a>

* [One feature file per API](https://www.testquality.com/blog/tpost/v79acjttj1-cucumber-and-gherkin-language-best-pract)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where does this reference recommend having one file per API? APIs are not explicitly mentioned, but considering several files for specific parts is advised:

How should the files be separated?
Because separating the features might result in a large number of files, you must consider how to separate the features into various files.

A popular solution is to have a file with features that collect everything relevant to a specific part of the program and even organize it into directories.

In the issue #94, another reference was mentioned. There, it is advised to:

Putting multiple features in a single feature file is not recommended. It is recommended to have a separate feature file for each feature. However, it is acceptable to include multiple scenarios in a single feature file.

So we should identify what are the "features" in the context of an API, and put a feature per file. To me a feature in this context is an API operation or resource. For simple APIs with single operations, it is quite straight-forward to decide that one file is enough, e.g. location-verification API, but for more complex APIs, such as QoD, we may need several files to test unrelated features, e.g. discovery of QoD profiles, creating QoD sessions, etc

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jlurien The reference link for one feature per API is https://copyprogramming.com/howto/multiple-feature-inside-single-feature-file#multiple-feature-inside-single-feature-file @shilpa-padgaonkar

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mdomale. In my understanding, the guidelines there does not advice for one feature per API but one "feature" per *.feature file. So that is what I suggest, to identify which are the "features" to test in the context of an API, and put each feature in each own *.feature file. Thus, an API may have several *.feature files for testing different aspects of the API. In simple APIs with just one endpoint that will not be the case.

* [Scenario Identifier](https://support.smartbear.com/cucumberstudio/docs/tests/best-practices.html#scenario-content-set-up-writing-standards)
* [One when/then](https://cucumber.io/docs/gherkin/reference/)