Contents

Python Acceptance Test Frameworks Comparison

Contents
PropertyBehavePytest-BDDRobot FrameworkHamgam (proprietary)
GitHub stars3k1.2k8.5k0
Used by10.1k3k8.8k30
Contributers73501777
Forks6822082.2k0
Commit/Week11101
Open issues601102750
Open PRs2833320
Gherkin/GWT
More BDD and agile friendly. Also, can utilize Xray steps library.
✓Almost ✓
A subset of the Gherkin (for example, doesn't support tags for different scenario outline examples).
✗
But has some translators, however, the result is not human-readable.
Also, you can mimic GWT using custom keywords.
✗
Currently, you can only write scenarios in a GWT manner.
JIRA Xray automation support
The exported test cases from Xray are directly usable, and the execution outputs can be imported directly.
✓
By exporting Cucumber test feature files and importing JSON results.
✓
By exporting Cucumber test feature files and importing JUnit XML report.
✓
By exporting Generic test files and importing the JUnit XML results or using the REST APIs.
✗
Custom languages (rather than Gherkin/GWT)✗✗✓✓ (but only in Python)
Graphical report✓Partly ✓
Can use Pytest graphical reports (such as pytest-html, pytest-html-reporter, or any other JUnit based reports).
✓
Also supports multiple alternative reporters.
✓
Writing custom reports✓
Using Formatters and Reporters.
✓
Using Pytest hooks
✓✓
Custom data driver✗
Can be implemented in a tricky way, but has an open pull-request for CSVs.
✗
Can be implemented in a very tricky way.
✓
Using a robotframework-datadriver plugin.
✓
Dry runSyntax: ✓
Reporters: Partly ✓
Type check: ✗
Auto-wiring: ✗
Pre/post-conditions: ✗
Syntax: ✗
Reporters: ✗
Type check: ✗
Auto-wiring: ✗
Pre/post-conditions: ✗
Syntax: ✓
Reporters: ✗
Type check: ✗
Auto-wiring: ✗
Pre/post-conditions: ✗
Syntax: ✓
Reporters: ✓
Type check: ✓
Auto-wiring: ✓
Pre/post-conditions: ✓
Declarative steps/keywords/actions✗✗✓✓
Filter testsFile: ✓
Scenario: ✓
Parameters: ✗
File: ✓
Scenario: ✓
Parameters: ✓
File: ✓
Scenario: ✓
Parameters: ✗
But each record can have a name and be filtered.
File: ✓
Scenario: ✓
Parameters: ✓
T-wise or pair-wise testing execution✗
But can be implemented using hooks.
✗
But can be implemented using hooks.
✗
But can be implemented in the data-driver layer.
✓
Auto-wiring✗
Can be implemented manually.
✗
Can be implemented manually.
✗
Can be implemented manually.
✓
Pre/post-conditions✗
Can be implemented manually.
✗
Can be implemented manually.
✗
Can be implemented manually.
✓
Hooks✓✓✓✓
Parallel execution✗
But has some wrappers and has an open pull-request.
✓
Using Pytest parallel execution plugins.
✓
Using plugins.
✓
But only in a single process.
Model-based testing✗✗✗✓
Unit-test friendly
Can have acceptance unit-tests and reuse codes.
✗✓✗✗
Marks/tags✓✓
But for the whole scenario not the examples.
✓✓
But for the whole scenario not the examples.
IDE friendlyHighlight: ✓
Auto-complete: ✗
Execute: ✓
Highlight: ✓
Auto-complete: ✗
Execute: ✓
Highlight: ✓
Auto-complete: ✓
Execute: ✓
Highlight: ✓
Auto-complete: ✓
Execute: ✗
Programming statements (for, if, while, ...)✗✗✓✓
Background (setup/teardown)✓✓✓✗
Reusable fixtures✓✓
Using Pytest fixtures mechanism.
✓✓
Localization✓✗✓✗
Learning curveGoodGoodGoodNormal (for scenarios)
Rich plugins and related libraries✗Partly ✓
Can use Pytest plugins, but doesn't have GWT steps.
✓
Selenium, HTTP requests, Appium, REST, SSH
✗
In-browser execution✗✗✓
Using WASM Python.
✗
Multiple programming languages for steps implementation✓
Most programming languages have some implementations for Gherkin.
✓
Most programming languages have some implementations for Gherkin.
✓
Using Remote Library concept.
✗
It only supports Python.