Exploring OpenStack Rally and Tempest Tools for Seamless Cloud Testing

OpenStack has become a go-to solution for organizations seeking to deploy and manage their own private cloud infrastructure. As cloud environments grow in complexity, the need for robust testing tools becomes increasingly vital to ensure stability, reliability and performance. OpenStack Rally and Tempest are two prominent tools designed specifically for testing and benchmarking OpenStack deployments. In this blog post, we’ll delve into the features, benefits, and use cases of these tools, shedding light on their significance in maintaining a healthy cloud ecosystem.

OpenStack Rally: Overview and Features

OpenStack Rally is a popular benchmarking and testing tool created to streamline the process of testing OpenStack deployments. Rally is designed to provide a comprehensive testing framework, helping developers and operators identify performance bottlenecks, assess scalability, and improve the overall quality of OpenStack installations. Some key features of OpenStack Rally include:

Modular Architecture: Rally is modular, allowing users to define test cases based on specific use cases, components, and scenarios. This modularity ensures that relevant tests can be executed efficiently without unnecessary overhead.

Wide Range of Scenarios: Rally provides a variety of pre-defined scenarios that replicate real-world use cases, such as launching instances, creating networks, and performing load tests. Users can also customize scenarios to suit their specific deployment requirements.

Scalability Testing: Rally supports scaling tests, allowing users to simulate a high number of concurrent users and workloads to gauge system performance under stress.

Plugin Support: Rally supports a plugin architecture that facilitates the integration of additional testing tools and frameworks, enabling users to extend the tool’s capabilities.

Comprehensive Reporting: Rally generates detailed HTML reports that provide insights into test results, performance metrics, and comparisons across different test runs, aiding in the identification of improvements and regressions.

==============================================================================================================================================
RALLY Installation
==============================================================================================================================================
root@OpenStack-Pilot-SC:/opt/openstack-ansible/playbooks# ansible-playbook os-rally-install.yml -vv
root@infra1-utility-container-b10ec761:/openstack/venvs/rally-23.2.2/bin# ./rally deployment create --fromenv --name=secunderabad

+--------------------------------------+---------------------+--------------+------------------+--------+
| uuid                             | created_at      | name     | status       | active |
+--------------------------------------+---------------------+--------------+------------------+--------+
| 302d3d3f-0926-4bc6-abce-ad6207ab1cfc | 2022-07-07T10:57:59 | secunderabad | deploy->finished |    |
+--------------------------------------+---------------------+--------------+------------------+--------+
Using deployment: 302d3d3f-0926-4bc6-abce-ad6207ab1cfc
~/.rally/openrc was updated

HINTS:
* To use standard OpenStack clients, set up your env by running:
    source ~/.rally/openrc
  OpenStack clients are now configured, e.g run:
    openstack image list
root@infra1-utility-container-b10ec761:/openstack/venvs/rally-23.2.2/bin#

==============================================================================================================================================
RALLY Plugins
==============================================================================================================================================
root@infra1-utility-container-b10ec761:/openstack/venvs/rally-23.2.2/bin# ./rally plugin list
+----------------------+----------------------------------------------------------------+-----------+--------------------------------------------------------------+
| Plugin base      | Name                                                       | Platform  | Title                                                                |
+----------------------+----------------------------------------------------------------+-----------+--------------------------------------------------------------+
| Chart            | EmbeddedChart                                              | default   | Chart for embedding custom html as a complete chart.                 |
| Chart            | EmbeddedExternalChart                                      | default   | Chart for embedding external html page as a complete chart.          |
| Chart            | Lines                                                      | default   | Display results as a generic chart with lines.                       |
| Chart            | OSProfiler                                                 | default   | Chart for embedding OSProfiler data.                                 |
| Chart            | Pie                                                        | default   | Display results as pie, calculate average values for additive data.  |
| Chart            | StackedArea                                                | default   | Display results as a stacked area.                                   |
| Chart            | Star sTable                                                | default   | Calculate statistics for additive data and display it as a table.    |
| Chart            | Table                                                      | default   | Display complete output as table, can not be used for additive data. |
| Chart            | TextArea                                                   | default   | Arbitrary text                                                       |
| Context          | allow_ssh                                                  | openstack | Sets up security groups for all users to access VM via SSH.          |
| Context          | api_versions                                               | openstack | Context for specifying OpenStack clients versions and service types. |
| Context          | audit_templates                                            | openstack | Creates Watcher audit templates for tenants.                         |
| Context          | ca_certs                                                   | openstack | Creates ca certs.                                                    |
| Context          | cluster_templates                                          | openstack | Creates Magnum cluster template.                                     |
| Context          | clusters                                                   | openstack | Creates specified amount of Magnum clusters.                         |
| Context          | dummy_context                                              | default   | Dummy context.                                                       |
| Context          | existing_network                                           | openstack | This context supports using existing networks in Rally.              |
| Context          | flavors                                                    | openstack | Context creates a list of flavors.                                   |
| Context          | heat_dataplane                                             | openstack | Context class for creating a stack by given template.                |
| Context          | image_command_customizer                                   | openstack | Context class for generating images customized by a command execution|
| Context          | images                                                     | openstack | Uploads specified Glance images to every tenant.                     |
| Context          | keypair                                                    | openstack | Create Nova KeyPair for each user.                                   |
.....
....
...
..
.
| Validator         | required_neutron_extensions                               | openstack | Validator checks if the specified Neutron extension is available      |
| Validator         | required_param_or_context                                 | default   | Validator checks if the required image is specified.                  |
| Validator         | required_params                                           | default   | Scenario required parameter validator.                                |
| Validator         | required_platform                                         | default   | Validates specification of specified platform for the workload.       |
| Validator         | required_platform                                         | openstack | Validates credentials for OpenStack platform.                         |
| Validator         | required_services                                         | openstack | Validator checks if specified OpenStack services are available.       |
| Validator         | restricted_parameters                                     | default   | Validates that parameters are not set.                                |
| Validator         | valid_command                                             | openstack | Check that parameter is a proper command-specifying dictionary.       |
| Validator         | validate_heat_template                                    | openstack | Validates heat template.                                              |
| Validator         | volume_type_exists                                        | openstack | Returns validator for volume types.                                   |
| Validator         | workbook_contains_workflow                                | openstack | Validate that workflow exist in workbook when workflow is passed      |
+----------------------+----------------------------------------------------------------+-----------+---------------------------------------------------------------+
root@infra1-utility-container-b10ec761:/openstack/venvs/rally-23.2.2/bin#

==============================================================================================================================================
RALLY TASK Execution
==============================================================================================================================================
root@infra1-utility-container-b10ec761:/openstack/venvs/rally-23.2.2/bin# ./rally task start samples/tasks/scenarios/nova/boot-and-delete.json
--------------------------------------------------------------------------------
Preparing input task
--------------------------------------------------------------------------------
Task is:
{
"NovaServers.boot_and_delete_server": [
     {
         "args": {
             "flavor": {
                 "name": "m1.tiny"
             },
             "image": {
                 "name": "^cirros-0.5.1"
             }
             "force_delete": false
         },
         "runner": {
             "type": "constant",
             "times": 10,
             "concurrency": 2
         },
         "context": {}
     }
]
}

Task syntax is correct :)
Running Rally version 3.3.0
--------------------------------------------------------------------------------
Task  157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7: started
--------------------------------------------------------------------------------
Running Task... This can take a while...
To track task status use:
    rally task status
    or
    rally task detailed

Using task: 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7
2022-07-12 23:05:36.910 15260 INFO rally.task.engine [-] Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 | Starting:  Task validation.
2022-07-12 23:05:36.917 15260 INFO rally.task.engine [-] Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 | Starting:  Task validation of syntax.
2022-07-12 23:05:36.925 15260 INFO rally.task.engine [-] Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 | Completed: Task validation of syntax.
2022-07-12 23:05:36.926 15260 INFO rally.task.engine [-] Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 | Starting:  Task validation of required platforms.
2022-07-12 23:05:36.939 15260 INFO rally.task.engine [-] Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 | Completed: Task validation of required platforms.
2022-07-12 23:05:36.939 15260 INFO rally.task.engine [-] Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 | Starting:  Task validation of semantics.
2022-07-12 23:05:36.940 15260 INFO rally.task.engine [-] Check health of the environment '302d3d3f-0926-4bc6-abce-ad6207ab1cfc'.
2022-07-12 23:05:38.021 15260 INFO rally.task.engine [-] Platform existing@openstack (available: True): OK!
2022-07-12 23:05:40.703 15260 INFO rally.task.context [-] Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 | Context users@openstack setup()  finished in 2.66 sec
2022-07-12 23:05:43.392 15260 INFO rally.task.context [-] Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 | Context users@openstack cleanup() started
2022-07-12 23:05:46.439 15260 INFO rally.task.context [-] Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 | Context users@openstack cleanup() finished in 3.05 sec
2022-07-12 23:05:46.439 15260 INFO rally.task.engine [-] Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 | Completed: Task validation of semantic.
2022-07-12 23:05:46.439 15260 INFO rally.task.engine [-] Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 | Completed: Task validation.
2022-07-12 23:05:46.440 15260 INFO rally.api [-] Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 input file is valid.
2022-07-12 23:05:46.440 15260 INFO rally.api [-] Run Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 against Deployment 302d3d3f-0926-4bc6-abce-ad6207ab1cfc
2022-07-12 23:05:46.440 15260 INFO rally.task.engine [-] Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 | Starting:  Running task.
2022-07-12 23:05:46.466 15260 INFO rally.task.engine [-] Running workload:
  position = 0
  config = {
   "version": 2,
   "title": "A cropped version of a bigger task.",
   "description": "Auto-generated task from a single workload (uuid=96ff3093-72d9-4f47-97b1-d68ef8c8b13b)",
   "subtasks": [
   {
      "title": "NovaServers.boot_and_delete_server",
      "description": "Boot and delete a server.",
      "scenario": {
         "NovaServers.boot_and_delete_server": {
            "flavor": {
               "name": "m1.tiny"
            },
            "image": {
               "name": "^cirros-0.5.1"
            },
            "force_delete": false
         }
      },
      "contexts": {},
      "runner": {
         "constant": {
            "times": 10,
            "concurrency": 2
         }
      },
      "hooks": [],
      "sla": {
         "failure_rate": {
            "max": 0
         }
      }
   }
   ]
}

2022-07-12 23:05:49.162 15260 INFO rally.task.context [-] Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 | Context users@openstack setup()  finished in 2.68 sec
2022-07-12 23:05:49.162 15260 INFO rally.task.context [-] Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 | Context cleanup@openstack setup()  finished in 0.00 msec
2022-07-12 23:05:50.752 15309 INFO rally.task.runner [-] Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 | ITER: 1 START
2022-07-12 23:05:50.756 15311 INFO rally.task.runner [-] Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 | ITER: 2 START
2022-07-12 23:06:02.686 15311 INFO rally.task.runner [-] Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 | ITER: 2 END: OK
2022-07-12 23:06:02.688 15311 INFO rally.task.runner [-] Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 | ITER: 3 START
2022-07-12 23:06:03.232 15309 INFO rally.task.runner [-] Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 | ITER: 1 END: OK
2022-07-12 23:06:03.233 15309 INFO rally.task.runner [-] Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 | ITER: 4 START
2022-07-12 23:06:13.443 15309 INFO rally.task.runner [-] Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 | ITER: 4 END: OK
2022-07-12 23:06:13.444 15309 INFO rally.task.runner [-] Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 | ITER: 5 START
2022-07-12 23:06:14.158 15311 INFO rally.task.runner [-] Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 | ITER: 3 END: OK
2022-07-12 23:06:14.160 15311 INFO rally.task.runner [-] Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 | ITER: 6 START
2022-07-12 23:06:22.866 15309 INFO rally.task.runner [-] Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 | ITER: 5 END: OK
2022-07-12 23:06:22.867 15309 INFO rally.task.runner [-] Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 | ITER: 7 START
2022-07-12 23:06:26.448 15311 INFO rally.task.runner [-] Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 | ITER: 6 END: OK
2022-07-12 23:06:26.450 15311 INFO rally.task.runner [-] Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 | ITER: 8 START
2022-07-12 23:06:31.781 15309 INFO rally.task.runner [-] Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 | ITER: 7 END: OK
2022-07-12 23:06:31.782 15309 INFO rally.task.runner [-] Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 | ITER: 9 START
2022-07-12 23:06:35.335 15311 INFO rally.task.runner [-] Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 | ITER: 8 END: OK
2022-07-12 23:06:35.336 15311 INFO rally.task.runner [-] Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 | ITER: 10 START
2022-07-12 23:06:41.725 15309 INFO rally.task.runner [-] Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 | ITER: 9 END: OK
2022-07-12 23:06:44.970 15311 INFO rally.task.runner [-] Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 | ITER: 10 END: OK
2022-07-12 23:06:44.992 15260 INFO rally.task.context [-] Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 | Context cleanup@openstack cleanup() started
2022-07-12 23:06:46.373 15260 INFO rally.task.context [-] Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 | Context cleanup@openstack cleanup() finished in 1.38 sec
2022-07-12 23:06:46.373 15260 INFO rally.task.context [-] Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 | Context users@openstack cleanup() started
2022-07-12 23:06:48.894 15260 INFO rally.task.context [-] Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 | Context users@openstack cleanup() finished in 2.52 sec
2022-07-12 23:06:50.691 15260 INFO rally.task.engine [-] Load duration is: 53.215972
2022-07-12 23:06:50.691 15260 INFO rally.task.engine [-] Full runner duration is: 54.257012
2022-07-12 23:06:50.691 15260 INFO rally.task.engine [-] Full duration is: 62.414564
2022-07-12 23:06:50.734 15260 INFO rally.task.engine [-] Finished workload 1/1 of subtask 1  (completed 1 of 1 in general).
2022-07-12 23:06:50.764 15260 INFO rally.task.engine [-] Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 | Completed: Running task.

--------------------------------------------------------------------------------
Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7: finished
--------------------------------------------------------------------------------

test scenario NovaServers.boot_and_delete_server
args position 0
args values:
{
  "args": {
"flavor": {
   "name": "m1.tiny"
},
"image": {
   "name": "^cirros-0.5.1"
},
"force_delete": false
  },
  "runner": {
"times": 10,
"concurrency": 2
  },
  "contexts": {},
  "sla": {
"failure_rate": {
   "max": 0
}
  },
  "hooks": []
}

--------------------------------------------------------------------------------
Task 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 has 0 error(s)
--------------------------------------------------------------------------------
+-----------------------------------------------------------------------------------------------------------------------+
|                                             Response Times (sec)                                                      |
+--------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| Action              | Min (sec) | Median (sec) | 90%ile (sec) | 95%ile (sec) | Max (sec) | Avg (sec) | Success | Count|
+---------------------+-----------+--------------+--------------+--------------+-----------+-----------+--------+-------+
| nova.boot_server    | 6.45      | 7.292        | 9.549        | 9.622        | 9.694     | 7.738     | 100.0%  | 10   |
| nova.delete_server  | 2.425     | 2.718        | 3.061        | 3.261        | 3.46      | 2.778     | 100.0%  | 10   |
| total               | 8.884     | 10.075       | 12.306       | 12.392       | 12.478    | 10.516    | 100.0%  | 10   |
|  -> duration        | 7.884     | 9.075        | 11.306       | 11.392       | 11.478    | 9.516     | 100.0%  | 10   |
|  -> idle_duration   | 1.0       | 1.0          | 1.0          | 1.0          | 1.0       | 1.0       | 100.0%  | 10   |
+--------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
Load duration: 53.216
Full duration: 62.4146

HINTS:
* To plot HTML graphics with this data, run:
    rally task report 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 --out output.htm
* To generate a JUnit report, run:
    rally task export 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 --type junit-xml --to output.xml
* To get raw JSON output of task results, run:
    rally task report 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 --json --out output.json
root@infra1-utility-container-b10ec761:/openstack/venvs/rally-23.2.2/bin#

==============================================================================================================================================
RALLY Task Execution O/P in HTML format
==============================================================================================================================================
root@infra1-utility-container-b10ec761:/openstack/venvs/rally-23.2.2/bin# ./rally task report 157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7 --out output.html
2022-07-12 23:09:01.240 15425 INFO rally.api [-] Building 'html' report for the following task(s): '157a3b50-67cd-4f8a-a4ae-a2bbf6d84bb7'.
2022-07-12 23:09:01.274 15425 INFO rally.api [-] The report has been successfully built.
root@infra1-utility-container-b10ec761:/openstack/venvs/rally-23.2.2/bin#

RALLY Sample O/P html report

OpenStack Tempest: Overview and Features

OpenStack Tempest is another essential tool in the OpenStack ecosystem, primarily focused on API and integration testing. Tempest ensures that OpenStack services adhere to the expected API behavior and interact seamlessly with one another. Some key features of OpenStack Tempest include:

API Testing: Tempest tests the compatibility and consistency of OpenStack services by making API calls and validating responses against defined expectations. This helps identify discrepancies and inconsistencies across different OpenStack service versions.

Integration Testing: Beyond API testing, Tempest also performs integration tests that assess the interoperability of various OpenStack services. This ensures that components interact correctly and that system-level functionality is maintained.

Configurable Testing: Tempest allows users to configure test scenarios and cases to suit their deployment environments. This flexibility enables organizations to focus on the specific areas of OpenStack that are most relevant to their use cases.

Scenario Testing: Tempest offers scenario tests that simulate real-world usage patterns. These tests involve complex interactions between different services and provide valuable insights into the overall system performance.

Gate Testing: In the OpenStack development process, Tempest is commonly used in the gate testing phase, where proposed changes to the OpenStack codebase are tested to ensure they don’t introduce regressions or break existing functionality.

==============================================================================================================================================
TEMPEST Installation
==============================================================================================================================================
root@infra1-utility-container-b10ec761:/openstack/venvs/rally-23.2.2/bin# ./rally verify create-verifier --type tempest --name tempest-veifier
2022-07-19 11:35:57.762 176001 INFO rally.api [-] Creating verifier 'tempest-verifier'.
2022-07-19 11:35:57.838 176001 INFO rally.verification.manager [-] Cloning verifier repo from https://opendev.org/openstack/tempest.
2022-07-19 11:36:10.624 176001 INFO rally.verification.manager [-] Creating virtual environment. It may take a few minutes.
2022-07-19 11:36:37.009 176001 INFO rally.api [-] Verifier 'tempest-verifier' (UUID=02e37674-f3c2-418c-814b-e584f26bf1ff) has been successfully created!
Using the verifier 'tempest-verifier' (UUID=02e37674-f3c2-418c-814b-e584f26bf1ff) as the default verifier for the future CLI operations.
root@infra1-utility-container-b10ec761:/openstack/venvs/rally-23.2.2/bin#
root@infra1-utility-container-b10ec761:/openstack/venvs/rally-23.2.2/bin# ./rally deployment list
+--------------------------------------+---------------------+--------------+------------------+--------+
| uuid                             | created_at      | name     | status       | active |
+--------------------------------------+---------------------+--------------+------------------+--------+
| 4c8f2ca9-be28-4c7a-90d2-ef9200db6254 | 2022-06-15T09:58:10 | existing | deploy->finished |    |
| 302d3d3f-0926-4bc6-abce-ad6207ab1cfc | 2022-07-07T10:57:59 | secunderabad | deploy->finished | *  |
+--------------------------------------+---------------------+--------------+------------------+--------+
root@infra1-utility-container-b10ec761:/openstack/venvs/rally-23.2.2/bin#
root@infra1-utility-container-b10ec761:/openstack/venvs/rally-23.2.2/bin# ./rally verify configure-verifier
2022-07-19 11:40:17.981 176095 INFO rally.api [-] Configuring verifier 'tempest-verifier' (UUID=02e37674-f3c2-418c-814b-e584f26bf1ff) for deployment 'secunderabad' (UUID=302d3d3f-0926-4bc6-abce-ad6207ab1cfc).
2022-07-19 11:40:18.804 176095 INFO rally.api [-] Verifier 'tempest-verifier' (UUID=02e37674-f3c2-418c-814b-e584f26bf1ff) has been successfully configured for deployment 'secunderabad' (UUID=302d3d3f-0926-4bc6-abce-ad6207ab1cfc)!
root@infra1-utility-container-b10ec761:/openstack/venvs/rally-23.2.2/bin#

==============
Run Tempest
===============
The command `rally verify start` launches auto-verification. This command expects only one argument: the test set name.
If this argument is not specified, smoke tests (the default) will be executed. Output of this command is similar to the Tempest output.

Valid test set names include: full, smoke, baremetal, compute, data_processing, identity, image, network, object_storage, orchestration,
telemetry, and volume.

root@infra1-utility-container-b10ec761:/openstack/venvs/rally-23.2.2/bin# ./rally verify start
2022-07-19 11:44:24.964 176100 INFO rally.api [-] Starting verification (UUID=13206051-44a3-4825-963f-9a27d704a129) for deployment 'secunderabad' (UUID=302d3d3f-0926-4bc6-abce-ad6207ab1cfc) by verifier 'tempest-verifier' (UUID=02e37674-f3c2-418c-814b-e584f26bf1ff).
2022-07-19 11:44:27.409 176100 INFO rally.task.context [-] Verification 02e37674-f3c2-418c-814b-e584f26bf1ff | Context tempest@default setup()  finished in 2.00 sec
2022-07-19 11:44:27.409 176100 INFO rally.task.context [-] Verification 02e37674-f3c2-418c-814b-e584f26bf1ff | Context testr@default setup()  finished in 0.03 msec
2022-07-19 11:44:32.147 176100 INFO tempest-verifier [-] {21} tempest.api.compute.admin.test_quotas.QuotasAdminTestV257 ... skip: b'The microversion range[2.57 - latest] of this test is out of the configuration range[None - None].'
2022-07-19 11:44:32.186 176100 INFO tempest-verifier [-] {22} tempest.api.compute.admin.test_auto_allocate_network.AutoAllocateNetworkTest ... skip: b'The microversion range[2.37 - latest] of this test is out of the configuration range[None - None].'
2022-07-19 11:44:32.189 176100 INFO tempest-verifier [-] {22} tempest.api.compute.admin.test_flavors_microversions.FlavorsV255TestJSON ... skip: b'The microversion range[2.55 - latest] of this test is out of the configuration range[None - None].'
2022-07-19 11:44:32.258 176100 INFO tempest-verifier [-] {12} tempest.api.compute.admin.test_hypervisor.HypervisorAdminV228Test ... skip: b'The microversion range[2.28 - latest] of this test is out of the configuration range[None - None].'
2022-07-19 11:44:32.286 176100 INFO tempest-verifier [-] {12} tempest.api.compute.admin.test_keypairs_v210.KeyPairsV210TestJSON ... skip: b'The microversion range[2.10 - latest] of this test is out of the configuration range[None - None].'
2022-07-19 11:44:32.287 176100 INFO tempest-verifier [-] {12} tempest.api.compute.admin.test_live_migration.LiveMigrationRemoteConsolesV26Test ... skip: b'The microversion range[2.6 - latest] of this test is out of the configuration range[None - None].'
2022-07-19 11:44:32.287 176100 INFO tempest-verifier [-] {12} tempest.api.compute.admin.test_quotas.QuotaClassesAdmin257Test ... skip: b'The microversion range[2.57 - latest] of this test is out of the configuration range[None - None].'
2022-07-19 11:44:32.385 176100 INFO tempest-verifier [-] {2} tempest.api.compute.admin.test_fixed_ips_negative.FixedIPsNegativeTestJson ... skip: Fixed IPs Negative Test Json skipped as neutron is available'
2022-07-19 11:44:32.388 176100 INFO tempest-verifier [-] {10} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON ... skip: b'The os-agents API is not supported.'
..
..
..
..
2022-07-19 12:37:59.228 176100 INFO tempest-verifier [-] {0} tempest.scenario.test_network_v6.Tests Getting Address.test_slaac_from_os ... fail [372.447s]
2022-07-19 12:38:02.902 176100 INFO rally.task.context [-] Verification 02e37674-f3c2-418c-814b-e584f26bf1ff | Context testr@default cleanup() started
2022-07-19 12:38:02.902 176100 INFO rally.task.context [-] Verification 02e37674-f3c2-418c-814b-e584f26bf1ff | Context testr@default cleanup() finished in 0.54 msec
2022-07-19 12:38:02.903 176100 INFO rally.task.context [-] Verification 02e37674-f3c2-418c-814b-e584f26bf1ff | Context tempest@default cleanup() started
2022-07-19 12:38:03.762 176100 INFO rally.task.context [-] Verification 02e37674-f3c2-418c-814b-e584f26bf1ff | Context tempest@default cleanup() finished in 0.86 sec
2022-07-19 12:38:03.991 176100 INFO rally.api [-] Verification (UUID=13206051-44a3-4825-963f-9a27d704a129) has been successfully finished for deployment 'secunderabad' (UUID=302d3d3f-0926-4bc6-abce-ad6207ab1cfc)!

======
Totals
======
Ran: 1625 tests in 3207.636 sec.
 - Success: 1046
 - Skipped: 486
 - Expected failures: 0
 - Unexpected success: 0
 - Failures: 93

Using verification (UUID=13206051-44a3-4825-963f-9a27d704a129) as the default verification for the future operations.
root@infra1-utility-container-b10ec761:/openstack/venvs/rally-23.2.2/bin#

root@infra1-utility-container-b10ec761:/openstack/venvs/rally-23.2.2/bin# ./rally verify list
+--------------------------------------+------+------------------+-----------------+---------------------+---------------------+----------+--------+
| UUID                             | Tags | Verifier name | Deployment name | Started at      | Finished at     | Duration | Status |
+--------------------------------------+------+------------------+-----------------+---------------------+---------------------+----------+--------+
| 13206051-44a3-4825-963f-9a27d704a129 | - | tempest-verifier | secunderabad | 2022-07-19T06:14:24 | 2022-07-19T07:08:03 | 0:53:39  | failed |
+--------------------------------------+------+------------------+-----------------+---------------------+---------------------+----------+--------+
root@infra1-utility-container-b10ec761:/openstack/venvs/rally-23.2.2/bin#
root@infra1-utility-container-b10ec761:/openstack/venvs/rally-23.2.2/bin# ./rally verify report --uuid 132060-44a3-4825-963f-9a27d704a129 --type html --to ./rally_tempest_report_SC-AZ.html
2022-07-19 12:44:21.926 177082 INFO rally.api [-] Building 'html' report for the following verification(s): '13206051-44a3-4825-963f-9a27d704a129'
2022-07-19 12:44:21.981 177082 INFO rally.api [-] The report has been successfully built.
Saving the report to './rally_tempest_report_SC-AZ.html' file. It may take some time.
The report has been successfully saved.
root@infra1-utility-container-b10ec761:/openstack/venvs/rally-23.2.2/bin#
root@infra1-utility-container-b10ec761:/openstack/venvs/rally-23.2.2/bin# ls -ltr *html
-rw-r--r-- 1 root root  163698 Jul 15 23:04 output.html
-rw-r--r-- 1 root root   55176 Jul 18 12:16 8.5_Boot_Nova_servers_37.html
-rw-r--r-- 1 root root   69541 Jul 18 12:20 8.4_Create_Cinder_volumes_250.html
-rw-r--r-- 1 root root   58920 Jul 18 12:41 8.3_Create_Neutron_ports_50.html
-rw-r--r-- 1 root root   86857 Jul 18 15:18 1.3_Keystone_authentication_500.html
-rw-r--r-- 1 root root   51157 Jul 18 17:05 1.2_Create_and_delete_image.html
-rw-r--r-- 1 root root 1001510 Jul 19 12:44 rally_tempest_report_SC-AZ.html
root@infra1-utility-container-b10ec761:/openstack/venvs/rally-23.2.2/bin#

TEMPEST Sample O/P html report

OpenStack Rally and Tempest are two powerful tools that play pivotal roles in maintaining the reliability, performance, and quality of OpenStack cloud deployments. Rally excels in benchmarking and performance testing, helping users identify scalability limits and optimize system performance. Tempest, on the other hand, focuses on API and integration testing, ensuring consistent behavior and interoperability among various OpenStack services.

By incorporating these tools into their workflows, organizations can streamline the testing process, identify issues early in the development cycle, and create more stable and robust OpenStack deployments. As the OpenStack ecosystem continues to evolve, Rally and Tempest will remain invaluable assets in the pursuit of delivering reliable and high-performance cloud infrastructure solutions.

ABOUT PINAKASTRA

Pinakastra is an organization committed to making cloud computing infrastructure more accessible to enterprises, academia & research organizations by helping them in IT infrastructure consolidation through the Pinakastra™ cloud platform.

HAPPY LEARNING!

Tags: benchmarking, CloudInfrastructure, cloudtesting, OpenStack, rally, Tempest