No failure patterns match your search.
Environment: INT
Failure patterns: 8 · Failures matched: 16 · Total runs: 32 · Runs affected: 13
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...full failure pattern: ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s) | e2e | 7 | 21.88%7 of 32 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s) | Show trend detailsAug 9: 0 · Aug 10: 1 · Aug 11: 2 · Aug 12: 0 · Aug 13: 0 · Aug 14: 1 · Aug 15: 1 · Aug 16: 2 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
<*errors.joinError | 0xc000e21080>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-wxt5q8" in resourcegroup="rg-zstream-upgrade-4-21-2h58qhpsnrpc" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/560bf569-795d-414b-a2b3-a9facd263303
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/560bf569-795d-414b-a2b3-a9facd263303",
"name": "560bf569-795d-414b-a2b3-a9facd263303",
"status": "Failed",
"startTime": "2026-08-16T05:33:51.945312233Z",
"endTime": "2026-08-16T05:57:52.72615778Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s7fmjoq31pobk77og51cbq33r20i6c8 still exists (deletion dispatched at 2026-08-16T05:33:52Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
<*errors.joinError | 0xc000e21080>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-wxt5q8" in resourcegroup="rg-zstream-upgrade-4-21-2h58qhpsnrpc" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/560bf569-795d-414b-a2b3-a9facd263303
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/560bf569-795d-414b-a2b3-a9facd263303",
"name": "560bf569-795d-414b-a2b3-a9facd263303",
"status": "Failed",
"startTime": "2026-08-16T05:33:51.945312233Z",
"endTime": "2026-08-16T05:57:52.72615778Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s7fmjoq31pobk77og51cbq33r20i6c8 still exists (deletion dispatched at 2026-08-16T05:33:52Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
...
occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
<*errors.joinError | 0xc000cfc978>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-66rshlg99zgt" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8ddf8e60-153e-4cf4-8998-18f59012fe70
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8ddf8e60-153e-4cf4-8998-18f59012fe70",
"name": "8ddf8e60-153e-4cf4-8998-18f59012fe70",
"status": "Failed",
"startTime": "2026-08-16T01:41:05.507310234Z",
"endTime": "2026-08-16T02:05:11.724659298Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s7c9m5ga9c22o9l9gp6h377pus8tmsq still exists (deletion dispatched at 2026-08-16T01:41:05Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
<*errors.joinError | 0xc000cfc978>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-66rshlg99zgt" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8ddf8e60-153e-4cf4-8998-18f59012fe70
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8ddf8e60-153e-4cf4-8998-18f59012fe70",
"name": "8ddf8e60-153e-4cf4-8998-18f59012fe70",
"status": "Failed",
"startTime": "2026-08-16T01:41:05.507310234Z",
"endTime": "2026-08-16T02:05:11.724659298Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s7c9m5ga9c22o9l9gp6h377pus8tmsq still exists (deletion dispatched at 2026-08-16T01:41:05Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
...
occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
<*errors.joinError | 0xc0019e20c0>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-d67bt4" in resourcegroup="admin-credential-lifecycle-test-spsfx8vqd5fs" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0c921227-e625-4a28-b3ba-4987345a2518
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0c921227-e625-4a28-b3ba-4987345a2518",
"name": "0c921227-e625-4a28-b3ba-4987345a2518",
"status": "Failed",
"startTime": "2026-08-15T14:57:41.798428506Z",
"endTime": "2026-08-15T15:21:44.602683219Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s72q36hsert5uuh6te15p0fo48t72h2 still exists (deletion dispatched at 2026-08-15T14:57:42Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
<*errors.joinError | 0xc0019e20c0>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-d67bt4" in resourcegroup="admin-credential-lifecycle-test-spsfx8vqd5fs" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0c921227-e625-4a28-b3ba-4987345a2518
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0c921227-e625-4a28-b3ba-4987345a2518",
"name": "0c921227-e625-4a28-b3ba-4987345a2518",
"status": "Failed",
"startTime": "2026-08-15T14:57:41.798428506Z",
"endTime": "2026-08-15T15:21:44.602683219Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s72q36hsert5uuh6te15p0fo48t72h2 still exists (deletion dispatched at 2026-08-15T14:57:42Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
...
occurredContributing tests (6)
Affected runs (7)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...full failure pattern: ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 6.25%2 of 32 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsAug 9: 0 · Aug 10: 0 · Aug 11: 0 · Aug 12: 1 · Aug 13: 0 · Aug 14: 1 · Aug 15: 0 · Aug 16: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
<*errors.joinError | 0xc00135c558>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-jpzcs287cj6r" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/cd93bd00-514b-401b-bf27-49e4ea29be09
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/cd93bd00-514b-401b-bf27-49e4ea29be09",
"name": "cd93bd00-514b-401b-bf27-49e4ea29be09",
"status": "Failed",
"startTime": "2026-08-14T21:39:45.318174028Z",
"endTime": "2026-08-14T22:03:46.852635941Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s6jm4plnlohu8m1l542gjfropjna9bh still exists (deletion dispatched at 2026-08-14T21:39:47Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
<*errors.joinError | 0xc00135c558>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-jpzcs287cj6r" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/cd93bd00-514b-401b-bf27-49e4ea29be09
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/cd93bd00-514b-401b-bf27-49e4ea29be09",
"name": "cd93bd00-514b-401b-bf27-49e4ea29be09",
"status": "Failed",
"startTime": "2026-08-14T21:39:45.318174028Z",
"endTime": "2026-08-14T22:03:46.852635941Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s6jm4plnlohu8m1l542gjfropjna9bh still exists (deletion dispatched at 2026-08-14T21:39:47Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
...
occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
<*errors.joinError | 0xc00187e180>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-qzpz7k97nzs5" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4dbc86e4-ab2e-43e0-b2b9-06ef2b78ae11
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4dbc86e4-ab2e-43e0-b2b9-06ef2b78ae11",
"name": "4dbc86e4-ab2e-43e0-b2b9-06ef2b78ae11",
"status": "Failed",
"startTime": "2026-08-12T15:38:34.396183363Z",
"endTime": "2026-08-12T16:02:41.259426366Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s545hojv14h7uer77e56nqi1sop4hdu still exists (deletion dispatched at 2026-08-12T15:38:34Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
<*errors.joinError | 0xc00187e180>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-qzpz7k97nzs5" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4dbc86e4-ab2e-43e0-b2b9-06ef2b78ae11
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4dbc86e4-ab2e-43e0-b2b9-06ef2b78ae11",
"name": "4dbc86e4-ab2e-43e0-b2b9-06ef2b78ae11",
"status": "Failed",
"startTime": "2026-08-12T15:38:34.396183363Z",
"endTime": "2026-08-12T16:02:41.259426366Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s545hojv14h7uer77e56nqi1sop4hdu still exists (deletion dispatched at 2026-08-12T15:38:34Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
...
occurredContributing tests (2)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...full failure pattern: ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 3.12%1 of 32 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsAug 9: 0 · Aug 10: 0 · Aug 11: 0 · Aug 12: 0 · Aug 13: 0 · Aug 14: 0 · Aug 15: 0 · Aug 16: 2 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
<*errors.joinError | 0xc002c10150>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-vjqjrc66m7hp" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ef664610-d7a7-4211-b960-7c5ef19010a6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ef664610-d7a7-4211-b960-7c5ef19010a6",
"name": "ef664610-d7a7-4211-b960-7c5ef19010a6",
"status": "Failed",
"startTime": "2026-08-16T02:15:26.507725852Z",
"endTime": "2026-08-16T02:39:32.627355563Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s7ceegbsetggi2btn9o1aunt17miklq still exists (deletion dispatched at 2026-08-16T02:15:27Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
<*errors.joinError | 0xc002c10150>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-vjqjrc66m7hp" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ef664610-d7a7-4211-b960-7c5ef19010a6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ef664610-d7a7-4211-b960-7c5ef19010a6",
"name": "ef664610-d7a7-4211-b960-7c5ef19010a6",
"status": "Failed",
"startTime": "2026-08-16T02:15:26.507725852Z",
"endTime": "2026-08-16T02:39:32.627355563Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s7ceegbsetggi2btn9o1aunt17miklq still exists (deletion dispatched at 2026-08-16T02:15:27Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
...
occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
<*errors.joinError | 0xc000fe0258>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="fips-enabled-cluster" in resourcegroup="fips-enabled-8fx5hz8rtgfc" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9f3328cc-bfbb-4690-87d9-7278e3150a65
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9f3328cc-bfbb-4690-87d9-7278e3150a65",
"name": "9f3328cc-bfbb-4690-87d9-7278e3150a65",
"status": "Failed",
"startTime": "2026-08-16T01:09:17.93392394Z",
"endTime": "2026-08-16T01:33:20.823057314Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s7bmgbu2s9urdae5dap0s9uns7ic01k still exists (deletion dispatched at 2026-08-16T01:09:18Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
<*errors.joinError | 0xc000fe0258>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="fips-enabled-cluster" in resourcegroup="fips-enabled-8fx5hz8rtgfc" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9f3328cc-bfbb-4690-87d9-7278e3150a65
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9f3328cc-bfbb-4690-87d9-7278e3150a65",
"name": "9f3328cc-bfbb-4690-87d9-7278e3150a65",
"status": "Failed",
"startTime": "2026-08-16T01:09:17.93392394Z",
"endTime": "2026-08-16T01:33:20.823057314Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s7bmgbu2s9urdae5dap0s9uns7ic01k still exists (deletion dispatched at 2026-08-16T01:09:18Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
...
occurredContributing tests (2)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShiftfull failure pattern: ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s) | e2e | 1 | 3.12%1 of 32 job runs affected | IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s) | Show trend detailsAug 9: 0 · Aug 10: 0 · Aug 11: 0 · Aug 12: 0 · Aug 13: 0 · Aug 14: 0 · Aug 15: 0 · Aug 16: 1 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:122]: failed to create nodepool ephemeral-np with ephemeral OS disk
Unexpected error:
<*fmt.wrapError | 0xc000eb7820>:
failed to create NodePool ephemeral-np: failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-lx86w6jrvwnc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/34bc491d-3cad-42c9-9d5e-e1047be81dd6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/34bc491d-3cad-42c9-9d5e-e1047be81dd6",
"name": "34bc491d-3cad-42c9-9d5e-e1047be81dd6",
"status": "Failed",
"startTime": "2026-08-16T01:42:16.130592799Z",
"endTime": "2026-08-16T02:01:21.628841591Z",
"error": {
"code": "InternalServerError",
"message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:122]: failed to create nodepool ephemeral-np with ephemeral OS disk
Unexpected error:
<*fmt.wrapError | 0xc000eb7820>:
failed to create NodePool ephemeral-np: failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-lx86w6jrvwnc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/34bc491d-3cad-42c9-9d5e-e1047be81dd6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/34bc491d-3cad-42c9-9d5e-e1047be81dd6",
"name": "34bc491d-3cad-42c9-9d5e-e1047be81dd6",
"status": "Failed",
"startTime": "2026-08-16T01:42:16.130592799Z",
"endTime": "2026-08-16T02:01:21.628841591Z",
"error": {
"code": "InternalServerError",
"message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
}
}
--------------------------------------------------------------------------------
...
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
tls: failed to verify certificate: x509: certificate signed by unknown authorityfull failure pattern: tls: failed to verify certificate: x509: certificate signed by unknown authority Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s) | e2e | 1 | 3.12%1 of 32 job runs affected | IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s) | Show trend detailsAug 9: 0 · Aug 10: 0 · Aug 11: 0 · Aug 12: 0 · Aug 13: 0 · Aug 14: 1 · Aug 15: 0 · Aug 16: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:173]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-fjsxfk"
Unexpected error:
<*fmt.wrapError | 0xc002bdc000>:
[strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-d42wd.apps.aro.l9t1e2j1t7s1a4j.vx77.uksouth.aroapp-hcp.azure-test.net": tls: failed to verify certificate: x509: certificate signed by unknown authority
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:173]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-fjsxfk"
Unexpected error:
<*fmt.wrapError | 0xc002bdc000>:
[strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-d42wd.apps.aro.l9t1e2j1t7s1a4j.vx77.uksouth.aroapp-hcp.azure-test.net": tls: failed to verify certificate: x509: certificate signed by unknown authority
...
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
5.0 HCCO ingress LB scope fix (OCPBUGS-98571) still not available; remove this skip or update the deadlinefull failure pattern: 5.0 HCCO ingress LB scope fix (OCPBUGS-98571) still not available; remove this skip or update the deadline Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 3.12%1 of 32 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsAug 9: 0 · Aug 10: 1 · Aug 11: 0 · Aug 12: 0 · Aug 13: 0 · Aug 14: 0 · Aug 15: 0 · Aug 16: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:82]: 5.0 HCCO ingress LB scope fix (OCPBUGS-98571) still not available; remove this skip or update the deadline fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:82]: 5.0 HCCO ingress LB scope fix (OCPBUGS-98571) still not available; remove this skip or update the deadline Contributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...full failure pattern: ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 3.12%1 of 32 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsAug 9: 0 · Aug 10: 1 · Aug 11: 0 · Aug 12: 0 · Aug 13: 0 · Aug 14: 0 · Aug 15: 0 · Aug 16: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
<*errors.joinError | 0xc0013d4168>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-m7c7wgglqlsp" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/91b584e4-bd58-44ca-a6c5-99484aacd7a0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/91b584e4-bd58-44ca-a6c5-99484aacd7a0",
"name": "91b584e4-bd58-44ca-a6c5-99484aacd7a0",
"status": "Failed",
"startTime": "2026-08-10T14:23:03.880421497Z",
"endTime": "2026-08-10T14:47:04.220397711Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s3oq0533bes1sn3aho547hgvdknskkg still exists (deletion dispatched at 2026-08-10T14:23:04Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
<*errors.joinError | 0xc0013d4168>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-m7c7wgglqlsp" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/91b584e4-bd58-44ca-a6c5-99484aacd7a0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/91b584e4-bd58-44ca-a6c5-99484aacd7a0",
"name": "91b584e4-bd58-44ca-a6c5-99484aacd7a0",
"status": "Failed",
"startTime": "2026-08-10T14:23:03.880421497Z",
"endTime": "2026-08-10T14:47:04.220397711Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s3oq0533bes1sn3aho547hgvdknskkg still exists (deletion dispatched at 2026-08-10T14:23:04Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
...
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
add password: HTTP 403, Authorization_RequestDenied: Insufficient privileges to complete the operation.: error status code received from the APIfull failure pattern: add password: HTTP 403, Authorization_RequestDenied: Insufficient privileges to complete the operation.: error status code received from the API Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 3.12%1 of 32 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsAug 9: 0 · Aug 10: 0 · Aug 11: 1 · Aug 12: 0 · Aug 13: 0 · Aug 14: 0 · Aug 15: 0 · Aug 16: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:135]: failed to add password to app registration
Unexpected error:
<*fmt.wrapError | 0xc000526b60>:
add password: HTTP 403, Authorization_RequestDenied: Insufficient privileges to complete the operation.: error status code received from the API
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:135]: failed to add password to app registration
Unexpected error:
<*fmt.wrapError | 0xc000526b60>:
add password: HTTP 403, Authorization_RequestDenied: Insufficient privileges to complete the operation.: error status code received from the API
...
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||