Python Acceptance Test Frameworks Comparison
Contents
| Property | Behave | Pytest-BDD | Robot Framework | Hamgam (proprietary) |
|---|---|---|---|---|
| GitHub stars | 3k | 1.2k | 8.5k | 0 |
| Used by | 10.1k | 3k | 8.8k | 30 |
| Contributers | 73 | 50 | 177 | 7 |
| Forks | 682 | 208 | 2.2k | 0 |
| Commit/Week | 1 | 1 | 10 | 1 |
| Open issues | 60 | 110 | 275 | 0 |
| Open PRs | 28 | 33 | 32 | 0 |
| Gherkin/GWT More BDD and agile friendly. Also, can utilize Xray steps library. | ✓ | Almost ✓ A subset of the Gherkin (for example, doesn't support tags for different scenario outline examples). | ✗ But has some translators, however, the result is not human-readable. Also, you can mimic GWT using custom keywords. | ✗ Currently, you can only write scenarios in a GWT manner. |
| JIRA Xray automation support The exported test cases from Xray are directly usable, and the execution outputs can be imported directly. | ✓ By exporting Cucumber test feature files and importing JSON results. | ✓ By exporting Cucumber test feature files and importing JUnit XML report. | ✓ By exporting Generic test files and importing the JUnit XML results or using the REST APIs. | ✗ |
| Custom languages (rather than Gherkin/GWT) | ✗ | ✗ | ✓ | ✓ (but only in Python) |
| Graphical report | ✓ | Partly ✓ Can use Pytest graphical reports (such as pytest-html, pytest-html-reporter, or any other JUnit based reports). | ✓ Also supports multiple alternative reporters. | ✓ |
| Writing custom reports | ✓ Using Formatters and Reporters. | ✓ Using Pytest hooks | ✓ | ✓ |
| Custom data driver | ✗ Can be implemented in a tricky way, but has an open pull-request for CSVs. | ✗ Can be implemented in a very tricky way. | ✓ Using a robotframework-datadriver plugin. | ✓ |
| Dry run | Syntax: ✓ Reporters: Partly ✓ Type check: ✗ Auto-wiring: ✗ Pre/post-conditions: ✗ | Syntax: ✗ Reporters: ✗ Type check: ✗ Auto-wiring: ✗ Pre/post-conditions: ✗ | Syntax: ✓ Reporters: ✗ Type check: ✗ Auto-wiring: ✗ Pre/post-conditions: ✗ | Syntax: ✓ Reporters: ✓ Type check: ✓ Auto-wiring: ✓ Pre/post-conditions: ✓ |
| Declarative steps/keywords/actions | ✗ | ✗ | ✓ | ✓ |
| Filter tests | File: ✓ Scenario: ✓ Parameters: ✗ | File: ✓ Scenario: ✓ Parameters: ✓ | File: ✓ Scenario: ✓ Parameters: ✗ But each record can have a name and be filtered. | File: ✓ Scenario: ✓ Parameters: ✓ |
| T-wise or pair-wise testing execution | ✗ But can be implemented using hooks. | ✗ But can be implemented using hooks. | ✗ But can be implemented in the data-driver layer. | ✓ |
| Auto-wiring | ✗ Can be implemented manually. | ✗ Can be implemented manually. | ✗ Can be implemented manually. | ✓ |
| Pre/post-conditions | ✗ Can be implemented manually. | ✗ Can be implemented manually. | ✗ Can be implemented manually. | ✓ |
| Hooks | ✓ | ✓ | ✓ | ✓ |
| Parallel execution | ✗ But has some wrappers and has an open pull-request. | ✓ Using Pytest parallel execution plugins. | ✓ Using plugins. | ✓ But only in a single process. |
| Model-based testing | ✗ | ✗ | ✗ | ✓ |
| Unit-test friendly Can have acceptance unit-tests and reuse codes. | ✗ | ✓ | ✗ | ✗ |
| Marks/tags | ✓ | ✓ But for the whole scenario not the examples. | ✓ | ✓ But for the whole scenario not the examples. |
| IDE friendly | Highlight: ✓ Auto-complete: ✗ Execute: ✓ | Highlight: ✓ Auto-complete: ✗ Execute: ✓ | Highlight: ✓ Auto-complete: ✓ Execute: ✓ | Highlight: ✓ Auto-complete: ✓ Execute: ✗ |
| Programming statements (for, if, while, ...) | ✗ | ✗ | ✓ | ✓ |
| Background (setup/teardown) | ✓ | ✓ | ✓ | ✗ |
| Reusable fixtures | ✓ | ✓ Using Pytest fixtures mechanism. | ✓ | ✓ |
| Localization | ✓ | ✗ | ✓ | ✗ |
| Learning curve | Good | Good | Good | Normal (for scenarios) |
| Rich plugins and related libraries | ✗ | Partly ✓ Can use Pytest plugins, but doesn't have GWT steps. | ✓ Selenium, HTTP requests, Appium, REST, SSH | ✗ |
| In-browser execution | ✗ | ✗ | ✓ Using WASM Python. | ✗ |
| Multiple programming languages for steps implementation | ✓ Most programming languages have some implementations for Gherkin. | ✓ Most programming languages have some implementations for Gherkin. | ✓ Using Remote Library concept. | ✗ It only supports Python. |
