Are there also smoke tests failing? I'm asking because this would block our CI until all failing smoke tests are fixed. So I would prefer to fix the smoke tests (In Ardana and Crowbar) before we accepts this.
Just finished running tempest smoke on Crowbar 8 and all tests passed. For ardana the CI does not use the smoke filter, instead it has a customized filter which does not include fwaas tests yet.
This is the list of smoke tests for fwaas:
{1} neutron_fwaas.tests.tempest_plugin.tests.api.test_fwaas_extensions.FWaaSExtensionTestJSON.test_create_show_delete_firewall [22.201023s] ... ok
{1} neutron_fwaas.tests.tempest_plugin.tests.api.test_fwaas_extensions.FWaaSExtensionTestJSON.test_create_update_delete_firewall_policy [2.464890s] ... ok
{1} neutron_fwaas.tests.tempest_plugin.tests.api.test_fwaas_extensions.FWaaSExtensionTestJSON.test_create_update_delete_firewall_rule [3.072841s] ... ok
{1} neutron_fwaas.tests.tempest_plugin.tests.api.test_fwaas_extensions.FWaaSExtensionTestJSON.test_firewall_rule_insertion_position_removal_rule_from_policy [4.529578s] ... ok
{1} neutron_fwaas.tests.tempest_plugin.tests.api.test_fwaas_extensions.FWaaSExtensionTestJSON.test_list_firewall_policies [0.929528s] ... ok
{1} neutron_fwaas.tests.tempest_plugin.tests.api.test_fwaas_extensions.FWaaSExtensionTestJSON.test_list_firewall_rules [1.106186s] ... ok
{1} neutron_fwaas.tests.tempest_plugin.tests.api.test_fwaas_extensions.FWaaSExtensionTestJSON.test_show_firewall_policy [1.157961s] ... ok
{1} neutron_fwaas.tests.tempest_plugin.tests.api.test_fwaas_extensions.FWaaSExtensionTestJSON.test_show_firewall_rule [1.103788s] ... ok
Did you test it? Are the tests passing at least partially?
Flavio did some testing. I will ask him to comment here.
This is part of a effort to get better test coverage on ardana (SCRD-7496) and neutron-fwaas is one of the services missing coverage by tempest.
I did run those tests against ardana and the results are: - Passed: 11 - Skipped: 2 - Expected Fail: 0 - Unexpected Success: 0 - Failed: 16
I still need to go through the failed tests and find out why they are failing, anyways, we need those tests on the package.
Are there also smoke tests failing? I'm asking because this would block our CI until all failing smoke tests are fixed. So I would prefer to fix the smoke tests (In Ardana and Crowbar) before we accepts this.
Just finished running tempest smoke on Crowbar 8 and all tests passed. For ardana the CI does not use the smoke filter, instead it has a customized filter which does not include fwaas tests yet.
This is the list of smoke tests for fwaas: {1} neutron_fwaas.tests.tempest_plugin.tests.api.test_fwaas_extensions.FWaaSExtensionTestJSON.test_create_show_delete_firewall [22.201023s] ... ok {1} neutron_fwaas.tests.tempest_plugin.tests.api.test_fwaas_extensions.FWaaSExtensionTestJSON.test_create_update_delete_firewall_policy [2.464890s] ... ok {1} neutron_fwaas.tests.tempest_plugin.tests.api.test_fwaas_extensions.FWaaSExtensionTestJSON.test_create_update_delete_firewall_rule [3.072841s] ... ok {1} neutron_fwaas.tests.tempest_plugin.tests.api.test_fwaas_extensions.FWaaSExtensionTestJSON.test_firewall_rule_insertion_position_removal_rule_from_policy [4.529578s] ... ok {1} neutron_fwaas.tests.tempest_plugin.tests.api.test_fwaas_extensions.FWaaSExtensionTestJSON.test_list_firewall_policies [0.929528s] ... ok {1} neutron_fwaas.tests.tempest_plugin.tests.api.test_fwaas_extensions.FWaaSExtensionTestJSON.test_list_firewall_rules [1.106186s] ... ok {1} neutron_fwaas.tests.tempest_plugin.tests.api.test_fwaas_extensions.FWaaSExtensionTestJSON.test_show_firewall_policy [1.157961s] ... ok {1} neutron_fwaas.tests.tempest_plugin.tests.api.test_fwaas_extensions.FWaaSExtensionTestJSON.test_show_firewall_rule [1.103788s] ... ok
Thanks a lot. Can you do the same change for Queens:Staging and Rocky:Staging please? Then I think we are good and can accept it.