CIHealth Failure Patterns

Environment: DEV

Failure patterns: 54 · Failures matched: 92 · Total runs: 90 · Runs affected: 48

Failure pattern

Failed at

Trend

Also in

ERROR CODE: DeadlineExceeded; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down pod network instances, pod networks, namespaces...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down pod network instances, pod networks, namespaces; [descendantResources] remaining resources: applydesires, serviceProviderClusters; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:321]: Unexpected error:
    <*errors.joinError | 0x2b12a0180120>: 
    resource group "rg-candidate-4-22-9q8kddc4rfc6": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-22-hscqkp" in resourcegroup="rg-candidate-4-22-9q8kddc4rfc6" to finish deleting: GET https://rp.j7963648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/46e9ad9e-9f53-4038-ba36-f40dec0ff2b1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/46e9ad9e-9f53-4038-ba36-f40dec0ff2b1",
      "name": "46e9ad9e-9f53-4038-ba36-f40dec0ff2b1",
      "status": "Failed",
      "startTime": "2026-10-28T17:51:06.474152018Z",
      "endTime": "2026-10-28T18:15:11.708953381Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7963648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/46e9ad9e-9f53-4038-ba36-f40dec0ff2b1
            correlation ID: 4741ead8-d72b-47f6-84e5-df7418434c6d
            client request ID: 9a224eab-3688-4c8c-9f47-6f8238e2c607
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:321]: Unexpected error:
    <*errors.joinError | 0x2b12a0180120>: 
    resource group "rg-candidate-4-22-9q8kddc4rfc6": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-22-hscqkp" in resourcegroup="rg-candidate-4-22-9q8kddc4rfc6" to finish deleting: GET https://rp.j7963648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/46e9ad9e-9f53-4038-ba36-f40dec0ff2b1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/46e9ad9e-9f53-4038-ba36-f40dec0ff2b1",
      "name": "46e9ad9e-9f53-4038-ba36-f40dec0ff2b1",
      "status": "Failed",
      "startTime": "2026-10-28T17:51:06.474152018Z",
      "endTime": "2026-10-28T18:15:11.708953381Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7963648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/46e9ad9e-9f53-4038-ba36-f40dec0ff2b1
            correlation ID: 4741ead8-d72b-47f6-84e5-df7418434c6d
            client request ID: 9a224eab-3688-4c8c-9f47-6f8238e2c607
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:321]: Unexpected error:
    <*errors.joinError | 0x2a43d7c880a8>: 
    resource group "private-kas-2t2bq848xclk": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kas" in resourcegroup="private-kas-2t2bq848xclk" to finish deleting: GET https://rp.j7963648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/827c464e-64e1-4e42-980f-59af842b0bed
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/827c464e-64e1-4e42-980f-59af842b0bed",
      "name": "827c464e-64e1-4e42-980f-59af842b0bed",
      "status": "Failed",
      "startTime": "2026-10-28T17:45:46.805727964Z",
      "endTime": "2026-10-28T18:09:51.562148289Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7963648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/827c464e-64e1-4e42-980f-59af842b0bed
            correlation ID: c4e40049-9c6b-42d0-ae1f-bfdcd35c5399
            client request ID: 5bcb7ef3-7dfb-48a5-a7c7-360bec18ff93
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:321]: Unexpected error:
    <*errors.joinError | 0x2a43d7c880a8>: 
    resource group "private-kas-2t2bq848xclk": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kas" in resourcegroup="private-kas-2t2bq848xclk" to finish deleting: GET https://rp.j7963648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/827c464e-64e1-4e42-980f-59af842b0bed
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/827c464e-64e1-4e42-980f-59af842b0bed",
      "name": "827c464e-64e1-4e42-980f-59af842b0bed",
      "status": "Failed",
      "startTime": "2026-10-28T17:45:46.805727964Z",
      "endTime": "2026-10-28T18:09:51.562148289Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7963648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/827c464e-64e1-4e42-980f-59af842b0bed
            correlation ID: c4e40049-9c6b-42d0-ae1f-bfdcd35c5399
            client request ID: 5bcb7ef3-7dfb-48a5-a7c7-360bec18ff93
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:321]: Unexpected error:
    <*errors.joinError | 0x17fd8fc22138>: 
    resource group "kusto-logs-2tq8m2fllmvs": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-2tq8m2fllmvs" to finish deleting: GET https://rp.j7963648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e027677d-2084-4178-b70c-b4e22dfc37b1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e027677d-2084-4178-b70c-b4e22dfc37b1",
      "name": "e027677d-2084-4178-b70c-b4e22dfc37b1",
      "status": "Failed",
      "startTime": "2026-10-28T17:46:42.22634078Z",
      "endTime": "2026-10-28T18:10:50.94970388Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7963648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e027677d-2084-4178-b70c-b4e22dfc37b1
            correlation ID: fcf84780-ffda-49de-a03f-17dbeb0ef756
            client request ID: 070116fb-514a-47f7-ae2b-d06066429ea7
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:321]: Unexpected error:
    <*errors.joinError | 0x17fd8fc22138>: 
    resource group "kusto-logs-2tq8m2fllmvs": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-2tq8m2fllmvs" to finish deleting: GET https://rp.j7963648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e027677d-2084-4178-b70c-b4e22dfc37b1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e027677d-2084-4178-b70c-b4e22dfc37b1",
      "name": "e027677d-2084-4178-b70c-b4e22dfc37b1",
      "status": "Failed",
      "startTime": "2026-10-28T17:46:42.22634078Z",
      "endTime": "2026-10-28T18:10:50.94970388Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7963648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e027677d-2084-4178-b70c-b4e22dfc37b1
            correlation ID: fcf84780-ffda-49de-a03f-17dbeb0ef756
            client request ID: 070116fb-514a-47f7-ae2b-d06066429ea7
occurred

Contributing tests (3)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-28 16:16	PR #7777	2115477832887963648

ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<u...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '1' s... message truncated; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-8844z2" for cluster "cluster-candidate-4-22-8844z2"
Unexpected error:
    <*fmt.wrapError | 0x34a8b7c1a9e0>: 
    failed to create NodePool np-8844z2: failed waiting for nodepool="np-8844z2" for cluster "cluster-candidate-4-22-8844z2" in resourcegroup="rg-candidate-4-22-p8n8vcw2knj6" to finish creating: GET https://rp.j7668608.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e163bbab-c728-4e64-85d2-39934825ad2c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e163bbab-c728-4e64-85d2-39934825ad2c",
      "name": "e163bbab-c728-4e64-85d2-39934825ad2c",
      "status": "Failed",
      "startTime": "2026-10-28T17:07:17.250607245Z",
      "endTime": "2026-10-28T17:26:25.565644744Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y8h5w3s1s0z4u9d-np-8844z2-q5kw2-zpqqj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-p8n8vcw2knj6-candidate-8844z2--managed/providers/Microsoft.Compute/virtualMachines/y8h5w3s1s0z4u9d-np-8844z2-q5kw2-zpqqj to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine y8h5w3s1s0z4u9d-np-8844z2-q5kw2-qsnbp: Failed: interfaces failed to create or update. err: failed to get existing resource rg-candidate-4-22-p8n8vcw2knj6-candidate-8844z2--managed/y8h5w3s1s0z4u9d-np-8844z2-q5kw2-qsnbp-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-p8n8vcw2knj6-candidate-8844z2--managed/providers/Microsoft.Network/networkInterfaces/y8h5w3s1s0z4u9d-np-8844z2-q5kw2-qsnbp-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'e92b9f76-b040-4cf6-a4dd-5c8bdc759e69' exceeded. Please try again after '1' s... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7668608.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e163bbab-c728-4e64-85d2-39934825ad2c
            correlation ID: 8a8572c9-5da8-4651-8233-b7c79a67a3b3
            client request ID: fc96c7e7-a6f5-442a-9e18-6d3eec0c11ee
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-8844z2" for cluster "cluster-candidate-4-22-8844z2"
Unexpected error:
    <*fmt.wrapError | 0x34a8b7c1a9e0>: 
    failed to create NodePool np-8844z2: failed waiting for nodepool="np-8844z2" for cluster "cluster-candidate-4-22-8844z2" in resourcegroup="rg-candidate-4-22-p8n8vcw2knj6" to finish creating: GET https://rp.j7668608.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e163bbab-c728-4e64-85d2-39934825ad2c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e163bbab-c728-4e64-85d2-39934825ad2c",
      "name": "e163bbab-c728-4e64-85d2-39934825ad2c",
      "status": "Failed",
      "startTime": "2026-10-28T17:07:17.250607245Z",
      "endTime": "2026-10-28T17:26:25.565644744Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y8h5w3s1s0z4u9d-np-8844z2-q5kw2-zpqqj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-p8n8vcw2knj6-candidate-8844z2--managed/providers/Microsoft.Compute/virtualMachines/y8h5w3s1s0z4u9d-np-8844z2-q5kw2-zpqqj to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine y8h5w3s1s0z4u9d-np-8844z2-q5kw2-qsnbp: Failed: interfaces failed to create or update. err: failed to get existing resource rg-candidate-4-22-p8n8vcw2knj6-candidate-8844z2--managed/y8h5w3s1s0z4u9d-np-8844z2-q5kw2-qsnbp-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-p8n8vcw2knj6-candidate-8844z2--managed/providers/Microsoft.Network/networkInterfaces/y8h5w3s1s0z4u9d-np-8844z2-q5kw2-qsnbp-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'e92b9f76-b040-4cf6-a4dd-5c8bdc759e69' exceeded. Please try again after '1' s... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7668608.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e163bbab-c728-4e64-85d2-39934825ad2c
            correlation ID: 8a8572c9-5da8-4651-8233-b7c79a67a3b3
            client request ID: fc96c7e7-a6f5-442a-9e18-6d3eec0c11ee
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-28 15:54	PR #7757	2115472312617668608

ERROR CODE: DeploymentFailed; detail code ServiceUnavailable; detail message Error while processing network security perimeter association: The request timed out. Diagnostic infor...

full failure pattern:

ERROR CODE: DeploymentFailed; detail code ServiceUnavailable; detail message Error while processing network security perimeter association: The request timed out. Diagnostic information: timestamp '20261029T002034Z', subscription id '<null>', tracking id '<uuid>', request correlation id '<null>'.; provider Microsoft.Network

Signal:

provision

none

Full failure examples (1)

time=2026-10-29T00:19:50.303Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=2 description="Step nsp\n  Kind: ARM\n  Template: templates/mgmt-nsp.bicep\n  Parameters: configurations/mgmt-nsp.tmpl.bicepparam"
time=2026-10-29T00:19:50.785Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=2
time=2026-10-29T00:19:51.232Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=2 deployment=3b71804a5ee2727cb58ddf39f360c1023278ead072315414fda717fe3af5856e portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j8187392-mgmt-2%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F3b71804a5ee2727cb58ddf39f360c1023278ead072315414fda717fe3af5856e
time=2026-10-29T00:20:51.534Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=2 err="stamp 2: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8187392-mgmt-2/providers/Microsoft.Resources/deployments/3b71804a5ee2727cb58ddf39f360c1023278ead072315414fda717fe3af5856e/operationStatuses/08584083736944643362\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8187392-mgmt-2/providers/Microsoft.Resources/deployments/profile-binoiepvfvifg\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8187392-mgmt-2/providers/Microsoft.Network/networkSecurityPerimeters/nsp-j8187392-mgmt-2/resourceAssociations/nsp-j8187392-mgmt-2-ra-59b6d068-1622-5f44-a17f-e1a3ef5859d7\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"ServiceUnavailable\\\",\\r\\n                \\\"message\\\": \\\"Error while processing request for association /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8187392-mgmt-2/providers/Microsoft.Network/networkSecurityPerimeters/nsp-j8187392-mgmt-2/resourceAssociations/nsp-j8187392-mgmt-2-ra-59b6d068-1622-5f44-a17f-e1a3ef5859d7, Error message: The request timed out. Diagnostic information: timestamp '20261029T002034Z', subscription id '<null>', tracking id '00000000-0000-0000-0000-000000000000', request correlation id '<null>'.\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/nsp (stamp=2)	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-28 23:23	PR #7662	2115585397328187392

ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-bkpxxt' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...

full failure pattern:

ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-bkpxxt' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'.; provider Microsoft.Network

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-7tbfp9"
Unexpected error:
    <*errors.joinError | 0x28672532e420>: 
    VerifyIngressDNSRecord(aro.r8a3f2x2r3m7g7z.8i1p.j8906368.hcp.osadev.cloud) failed: VerifyIngressDNSRecord(aro.r8a3f2x2r3m7g7z.8i1p.j8906368.hcp.osadev.cloud) timed out after 10m0.005s: DNS A record "*.e2e-dns-bkpxxt" not yet present in zone rg-candidate-4-23-cwv9mngrfv9r-candidate-7tbfp9--managed/aro.r8a3f2x2r3m7g7z.8i1p.j8906368.hcp.osadev.cloud: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-cwv9mngrfv9r-candidate-7tbfp9--managed/providers/Microsoft.Network/dnsZones/aro.r8a3f2x2r3m7g7z.8i1p.j8906368.hcp.osadev.cloud/A/*.e2e-dns-bkpxxt
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: NotFound
    --------------------------------------------------------------------------------
    {
      "code": "NotFound",
      "message": "The resource record '*.e2e-dns-bkpxxt' does not exist in resource group 'rg-candidate-4-23-cwv9mngrfv9r-candidate-7tbfp9--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-cwv9mngrfv9r-candidate-7tbfp9--managed/providers/Microsoft.Network/dnsZones/aro.r8a3f2x2r3m7g7z.8i1p.j8906368.hcp.osadev.cloud/A/*.e2e-dns-bkpxxt
            correlation ID: 00f470eb-29ca-4c55-b41f-faa8336d52a1
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-7tbfp9"
Unexpected error:
    <*errors.joinError | 0x28672532e420>: 
    VerifyIngressDNSRecord(aro.r8a3f2x2r3m7g7z.8i1p.j8906368.hcp.osadev.cloud) failed: VerifyIngressDNSRecord(aro.r8a3f2x2r3m7g7z.8i1p.j8906368.hcp.osadev.cloud) timed out after 10m0.005s: DNS A record "*.e2e-dns-bkpxxt" not yet present in zone rg-candidate-4-23-cwv9mngrfv9r-candidate-7tbfp9--managed/aro.r8a3f2x2r3m7g7z.8i1p.j8906368.hcp.osadev.cloud: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-cwv9mngrfv9r-candidate-7tbfp9--managed/providers/Microsoft.Network/dnsZones/aro.r8a3f2x2r3m7g7z.8i1p.j8906368.hcp.osadev.cloud/A/*.e2e-dns-bkpxxt
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: NotFound
    --------------------------------------------------------------------------------
    {
      "code": "NotFound",
      "message": "The resource record '*.e2e-dns-bkpxxt' does not exist in resource group 'rg-candidate-4-23-cwv9mngrfv9r-candidate-7tbfp9--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-cwv9mngrfv9r-candidate-7tbfp9--managed/providers/Microsoft.Network/dnsZones/aro.r8a3f2x2r3m7g7z.8i1p.j8906368.hcp.osadev.cloud/A/*.e2e-dns-bkpxxt
            correlation ID: 00f470eb-29ca-4c55-b41f-faa8336d52a1
            client request ID: unavailable
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-28 19:56	PR #7757	2115533269758906368

ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-czbqgt' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...

full failure pattern:

ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-czbqgt' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'.; provider Microsoft.Network

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-kzwwft"
Unexpected error:
    <*errors.joinError | 0x2fea822a5d0>: 
    VerifyIngressDNSRecord(aro.m3u7u2n3l0l3p3b.0iw8.j3991936.hcp.osadev.cloud) failed: VerifyIngressDNSRecord(aro.m3u7u2n3l0l3p3b.0iw8.j3991936.hcp.osadev.cloud) timed out after 10m0s: DNS A record "*.e2e-dns-czbqgt" not yet present in zone rg-candidate-4-23-qkhkjtrdhq8n-candidate-kzwwft--managed/aro.m3u7u2n3l0l3p3b.0iw8.j3991936.hcp.osadev.cloud: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-qkhkjtrdhq8n-candidate-kzwwft--managed/providers/Microsoft.Network/dnsZones/aro.m3u7u2n3l0l3p3b.0iw8.j3991936.hcp.osadev.cloud/A/*.e2e-dns-czbqgt
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: NotFound
    --------------------------------------------------------------------------------
    {
      "code": "NotFound",
      "message": "The resource record '*.e2e-dns-czbqgt' does not exist in resource group 'rg-candidate-4-23-qkhkjtrdhq8n-candidate-kzwwft--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-qkhkjtrdhq8n-candidate-kzwwft--managed/providers/Microsoft.Network/dnsZones/aro.m3u7u2n3l0l3p3b.0iw8.j3991936.hcp.osadev.cloud/A/*.e2e-dns-czbqgt
            correlation ID: ef507981-8069-4371-b52a-e18fb184cf37
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-kzwwft"
Unexpected error:
    <*errors.joinError | 0x2fea822a5d0>: 
    VerifyIngressDNSRecord(aro.m3u7u2n3l0l3p3b.0iw8.j3991936.hcp.osadev.cloud) failed: VerifyIngressDNSRecord(aro.m3u7u2n3l0l3p3b.0iw8.j3991936.hcp.osadev.cloud) timed out after 10m0s: DNS A record "*.e2e-dns-czbqgt" not yet present in zone rg-candidate-4-23-qkhkjtrdhq8n-candidate-kzwwft--managed/aro.m3u7u2n3l0l3p3b.0iw8.j3991936.hcp.osadev.cloud: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-qkhkjtrdhq8n-candidate-kzwwft--managed/providers/Microsoft.Network/dnsZones/aro.m3u7u2n3l0l3p3b.0iw8.j3991936.hcp.osadev.cloud/A/*.e2e-dns-czbqgt
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: NotFound
    --------------------------------------------------------------------------------
    {
      "code": "NotFound",
      "message": "The resource record '*.e2e-dns-czbqgt' does not exist in resource group 'rg-candidate-4-23-qkhkjtrdhq8n-candidate-kzwwft--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-qkhkjtrdhq8n-candidate-kzwwft--managed/providers/Microsoft.Network/dnsZones/aro.m3u7u2n3l0l3p3b.0iw8.j3991936.hcp.osadev.cloud/A/*.e2e-dns-czbqgt
            correlation ID: ef507981-8069-4371-b52a-e18fb184cf37
            client request ID: unavailable
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-28 18:23	PR #7785	2115509692493991936

VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...

full failure pattern:

VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: no customer resource ID -> live ReadDesire HostedCluster ->

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-d5n27jjnnw7n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc252dd22260>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-d5n27jjnnw7n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-d5n27jjnnw7n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
        err: <*errors.errorString | 0xc252ebecf50>{
            s: "no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-d5n27jjnnw7n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc252dd22260>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-d5n27jjnnw7n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-d5n27jjnnw7n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
        err: <*errors.errorString | 0xc252ebecf50>{
            s: "no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-28 21:11	PR #7814	2115552146735763456

failed to roll out Helm release: resource Deployment/clusters-service/clusters-service not ready. status: InProgress, message: Available: 2/3 context deadline exceeded

full failure pattern:

failed to roll out Helm release: resource Deployment/clusters-service/clusters-service not ready. status: InProgress, message: Available: 2/3 context deadline exceeded

Signal:

provision

none

Full failure examples (1)

time=2026-10-28T17:01:13.002Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n"
time=2026-10-28T17:01:13.550Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy values="map[azureArmHelperIdentityCertName:armHelperPoolCert-25 azureArmHelperIdentityClientId:ea5e9c6c-1933-464d-8bf6-db343c22b92d azureArmHelperMockFpaPrincipalId:47f69502-0065-4d9a-b19b-d403e183d2f4 azureCloudEnvironmentName:AzurePublicCloud azureCsMiClientId:ae6cf536-acc0-414c-ada8-f268aeeb97e3 azureFirstPartyApplicationClientId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX azureMiMockServicePrincipalCertName:msiMockPoolCert-17 azureMiMockServicePrincipalClientId:e7112f7f-5b45-480e-bc44-abf9f219e6f0 azureMiMockServicePrincipalPrincipalId:39eefc4f-5e24-491d-9f09-bcc0ad573bcf azureOperatorsMI:map[dev:map[cloudControllerManager:map[roleDefinitions:[map[id:a1f96423-95ce-4224-ab27-4e3dc72facd4 name:Azure Red Hat OpenShift Cloud Controller Manager]]] cloudNetworkConfig:map[roleDefinitions:[map[id:be7a6435-15ae-4171-8f30-4a343eff9e8f name:Azure Red Hat OpenShift Network Operator]]] clusterApiAzure:map[roleDefinitions:[map[id:88366f10-ed47-4cc0-9fab-c8a06148393e name:Azure Red Hat OpenShift Hosted Control Planes Cluster API Provider]]] controlPlane:map[roleDefinitions:[map[id:fc0c873f-45e9-4d0d-a7d1-585aab30c6ed name:Azure Red Hat OpenShift Hosted Control Planes Control Plane Operator]]] diskCsiDriver:map[roleDefinitions:[map[id:5b7237c5-45e1-49d6-bc18-a1f62f400748 name:Azure Red Hat OpenShift Disk Storage Operator]]] fileCsiDriver:map[roleDefinitions:[map[id:0d7aedc0-15fd-4a67-a412-efad370c947e name:Azure Red Hat OpenShift File Storage Operator]]] imageRegistry:map[roleDefinitions:[map[id:8b32b316-c2f5-4ddf-b05b-83dacd2d08b5 name:Azure Red Hat OpenShift Image Registry Operator]]] ingress:map[roleDefinitions:[map[id:0336e1d3-7a87-462b-b6db-342b63f7802c name:Azure Red Hat OpenShift Cluster Ingress Operator]]] kms:map[roleDefinitions:[map[id:12338af0-0e69-4776-bea7-57ae8d297424 name:Key Vault Crypto User]]] serviceManagedIdentity:map[roleDefinitions:[map[id:c0ff367d-66d8-445e-917c-583feb0ef0d4 name:Azure Red Hat OpenShift Hosted Control Planes Service Managed Identity]]]] public:map[cloudControllerManager:map[roleDefinitions:[map[id:a1f96423-95ce-4224-ab27-4e3dc72facd4 name:Azure Red Hat OpenShift Cloud Controller Manager]]] cloudNetworkConfig:map[roleDefinitions:[map[id:be7a6435-15ae-4171-8f30-4a343eff9e8f name:Azure Red Hat OpenShift Network Operator]]] clusterApiAzure:map[roleDefinitions:[map[id:88366f10-ed47-4cc0-9fab-c8a06148393e name:Azure Red Hat OpenShift Hosted Control Planes Cluster API Provider]]] controlPlane:map[roleDefinitions:[map[id:fc0c873f-45e9-4d0d-a7d1-585aab30c6ed name:Azure Red Hat OpenShift Hosted Control Planes Control Plane Operator]]] diskCsiDriver:map[roleDefinitions:[map[id:5b7237c5-45e1-49d6-bc18-a1f62f400748 name:Azure Red Hat OpenShift Disk Storage Operator]]] fileCsiDriver:map[roleDefinitions:[map[id:0d7aedc0-15fd-4a67-a412-efad370c947e name:Azure Red Hat OpenShift File Storage Operator]]] imageRegistry:map[roleDefinitions:[map[id:8b32b316-c2f5-4ddf-b05b-83dacd2d08b5 name:Azure Red Hat OpenShift Image Registry Operator]]] ingress:map[roleDefinitions:[map[id:0336e1d3-7a87-462b-b6db-342b63f7802c name:Azure Red Hat OpenShift Cluster Ingress Operator]]] kms:map[roleDefinitions:[map[id:12338af0-0e69-4776-bea7-57ae8d297424 name:Key Vault Crypto User]]] serviceManagedIdentity:map[roleDefinitions:[map[id:c0ff367d-66d8-445e-917c-583feb0ef0d4 name:Azure Red Hat OpenShift Hosted Control Planes Service Managed Identity]]]] roleSetName:dev] backplaneURL: batchProcesses: batchProcessesDryRun:true clientId:foo clientScopes:openid clientSecret:bar clusterErrorReport:false cosmosDatabaseName:arohcpci01-rp-j3195648 cosmosDatabaseUrl:https://arohcpci01-rp-j3195648.documents.azure.com:443/ cpuRequest:630m csDeploymentStrategy:map[rollingUpdate:map[maxSurge:1 maxUnavailable:0] type:RollingUpdate] dataPlaneHAProxyImage:arohcpocpdev.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678 databaseAuthMethod:postgres databaseDisableTls:true databaseHost:ocm-cs-db databaseK8sSecretName:ocm-cs-db databaseName:ocm-cs-db databasePassword:TheBlurstOfTimes databasePort:5432 databaseServiceName:ocm-cs-db databaseUser:ocm databaseVolumeCapacity:512Mi debugMaxDuration:5m debugPort: defaultExpiration:0 deployment:map[zoneCount:3] environment:arohcpci01 forceMigration: fpaCertName:firstPartyCert2 gatewayUrl:http://127.0.0.1:9090 global:map[imageDigest:sha256:b6e44eb348113224aa292d06b6cbf4154e881fe0a8617e245e7c6555f046d495] imageDigest:sha256:b6e44eb348113224aa292d06b6cbf4154e881fe0a8617e245e7c6555f046d495 imageRegistry:arohcpsvcdev.azurecr.io imageRepository:app-sre/aro-hcp-clusters-service insecure:false jwksUrl:http://localhost logLevel:4 machinePoolMigrationWorkerPeriod:1h managedIdentitiesDataPlaneAudienceResource:https://dummy.org maximumExpiration:0 memoryLimit:unlimited memoryRequest:150Mi monitoringApiGroup:monitoring.coreos.com ocpAcrResourceId:/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev ocpAcrResourceUrl:arohcpocpdev.azurecr.io oidcIssuerBaseUrl:https://arohcpci01oidcj3195648.z9.web.core.windows.net/ oidcIssuerBlobServiceUrl:https://arohcpci01oidcj3195648.blob.core.windows.net/ postgresqlVersion:12 provisionShardClusterLimit:100 pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:app-sre/aro-hcp-clusters-service:pull workloadIdentityClientId:a5d5ff83-799c-44fd-a535-8c4386588d02 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] region:canadacentral replicas:3 runtimeMode:aro-hcp serviceAccountName:clusters-service serviceKeyVaultName:aro-hcp-dev-svc-kv tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX tlsCertificatesEnabled:true tlsCertificatesIssuer:Self tokenUrl:http://localhost useAzureDB:false userDefinedDnsBaseDomain:i1.devshift.org]"
time=2026-10-28T17:01:13.550Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy
time=2026-10-28T17:01:13.717Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy namespace=clusters-service
time=2026-10-28T17:01:13.899Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy dryRun=false
time=2026-10-28T17:01:13.899Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy
time=2026-10-28T17:01:13.952Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy
time=2026-10-28T17:11:19.322Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy err="resource Deployment/clusters-service/clusters-service not ready. status: InProgress, message: Available: 2/3\ncontext deadline exceeded"
time=2026-10-28T17:11:19.322Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy err="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource Deployment/clusters-service/clusters-service not ready. status: InProgress, message: Available: 2/3\ncontext deadline exceeded"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.ClusterService/service/deploy	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-28 16:20	PR #7311	2115478705303195648

missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=11333, acm-agent=0

full failure pattern:

missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=11333, acm-agent=0

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 600.005s.
Expected success, but got an error:
    <*errors.errorString | 0xca1fe06c6d0>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=11333, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=11333, acm-agent=0",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 600.005s.
Expected success, but got an error:
    <*errors.errorString | 0xca1fe06c6d0>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=11333, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=11333, acm-agent=0",
    }

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-28 12:14	PR #7662	2115416945330229248

missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=14276, acm-agent=0

full failure pattern:

missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=14276, acm-agent=0

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 600.005s.
Expected success, but got an error:
    <*errors.errorString | 0x867032a97d0>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=14276, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=14276, acm-agent=0",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 600.005s.
Expected success, but got an error:
    <*errors.errorString | 0x867032a97d0>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=14276, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=14276, acm-agent=0",
    }

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-28 03:31	PR #7662	2115285263486816256

missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=7034, acm-agent=0

full failure pattern:

missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=7034, acm-agent=0

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 600.000s.
Expected success, but got an error:
    <*errors.errorString | 0x20bd1e04e7a0>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=7034, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=7034, acm-agent=0",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 600.000s.
Expected success, but got an error:
    <*errors.errorString | 0x20bd1e04e7a0>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=7034, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=7034, acm-agent=0",
    }

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-28 07:07	PR #7662	2115339656903004160

missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=7169, acm-agent=0

full failure pattern:

missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=7169, acm-agent=0

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 600.001s.
Expected success, but got an error:
    <*errors.errorString | 0x2de4f1b0a5d0>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=7169, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=7169, acm-agent=0",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 600.001s.
Expected success, but got an error:
    <*errors.errorString | 0x2de4f1b0a5d0>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=7169, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=7169, acm-agent=0",
    }

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-28 09:35	PR #7662	2115376885591445504

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift

Signal:

e2e

PROD, STG

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:582]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0x1fcb88002b40>: 
    failed to create HCP cluster np-downgrade-llw24f: failed waiting for cluster="np-downgrade-llw24f" in resourcegroup="rg-np-downgrade-llw24f-kqx7lnhpdqpq" to finish creating: GET https://rp.j3784704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dcefa597-82c1-4b2e-9dc9-ff35d2de5251
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dcefa597-82c1-4b2e-9dc9-ff35d2de5251",
      "name": "dcefa597-82c1-4b2e-9dc9-ff35d2de5251",
      "status": "Failed",
      "startTime": "2026-10-28T23:17:28.994096116Z",
      "endTime": "2026-10-28T23:46:34.623847458Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3784704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dcefa597-82c1-4b2e-9dc9-ff35d2de5251
            correlation ID: 162093f7-f0bd-4b1e-b072-d657b497ab59
            client request ID: 0bb02caa-9eaa-4069-8399-efdeced8dac8
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:582]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0x1fcb88002b40>: 
    failed to create HCP cluster np-downgrade-llw24f: failed waiting for cluster="np-downgrade-llw24f" in resourcegroup="rg-np-downgrade-llw24f-kqx7lnhpdqpq" to finish creating: GET https://rp.j3784704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dcefa597-82c1-4b2e-9dc9-ff35d2de5251
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dcefa597-82c1-4b2e-9dc9-ff35d2de5251",
      "name": "dcefa597-82c1-4b2e-9dc9-ff35d2de5251",
      "status": "Failed",
      "startTime": "2026-10-28T23:17:28.994096116Z",
      "endTime": "2026-10-28T23:46:34.623847458Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3784704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dcefa597-82c1-4b2e-9dc9-ff35d2de5251
            correlation ID: 162093f7-f0bd-4b1e-b072-d657b497ab59
            client request ID: 0bb02caa-9eaa-4069-8399-efdeced8dac8
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0x3a2a28c8a260>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-h8dst6v865z6" to finish creating: GET https://rp.j1157504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/455ab4ea-ab03-4a58-91c6-5ad5406405ff
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/455ab4ea-ab03-4a58-91c6-5ad5406405ff",
      "name": "455ab4ea-ab03-4a58-91c6-5ad5406405ff",
      "status": "Failed",
      "startTime": "2026-10-28T20:16:02.73250898Z",
      "endTime": "2026-10-28T20:45:04.251585998Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1157504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/455ab4ea-ab03-4a58-91c6-5ad5406405ff
            correlation ID: 582a61df-2172-42e6-962b-b544533d3da6
            client request ID: a46122fb-f2eb-46f8-9be1-73dd5a442cce
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0x3a2a28c8a260>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-h8dst6v865z6" to finish creating: GET https://rp.j1157504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/455ab4ea-ab03-4a58-91c6-5ad5406405ff
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/455ab4ea-ab03-4a58-91c6-5ad5406405ff",
      "name": "455ab4ea-ab03-4a58-91c6-5ad5406405ff",
      "status": "Failed",
      "startTime": "2026-10-28T20:16:02.73250898Z",
      "endTime": "2026-10-28T20:45:04.251585998Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1157504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/455ab4ea-ab03-4a58-91c6-5ad5406405ff
            correlation ID: 582a61df-2172-42e6-962b-b544533d3da6
            client request ID: a46122fb-f2eb-46f8-9be1-73dd5a442cce
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0x370d5101eb20>: 
    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-tgjxrh8mcrw6" to finish creating: GET https://rp.j8717568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cce1648d-4b31-48bb-9884-f170bb5391eb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cce1648d-4b31-48bb-9884-f170bb5391eb",
      "name": "cce1648d-4b31-48bb-9884-f170bb5391eb",
      "status": "Failed",
      "startTime": "2026-10-28T19:28:55.409225524Z",
      "endTime": "2026-10-28T19:58:03.222551854Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8717568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cce1648d-4b31-48bb-9884-f170bb5391eb
            correlation ID: d6e5235d-8c3c-4404-82e9-f3e3b3165e1e
            client request ID: 4d040942-ab41-4fbc-8d6e-d69a84105ef1
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0x370d5101eb20>: 
    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-tgjxrh8mcrw6" to finish creating: GET https://rp.j8717568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cce1648d-4b31-48bb-9884-f170bb5391eb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cce1648d-4b31-48bb-9884-f170bb5391eb",
      "name": "cce1648d-4b31-48bb-9884-f170bb5391eb",
      "status": "Failed",
      "startTime": "2026-10-28T19:28:55.409225524Z",
      "endTime": "2026-10-28T19:58:03.222551854Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8717568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cce1648d-4b31-48bb-9884-f170bb5391eb
            correlation ID: d6e5235d-8c3c-4404-82e9-f3e3b3165e1e
            client request ID: 4d040942-ab41-4fbc-8d6e-d69a84105ef1
occurred

Contributing tests (6)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest	1

Affected runs (6)

Date (UTC)	Associated PR	Prow job
2026-10-28 21:44	PR #7693	2115560333123784704
2026-10-28 19:15	PR #7666	2115522964811157504
2026-10-28 18:33	PR #7764	2115512320468717568
2026-10-28 18:12	PR #7704	2115506901079822336
2026-10-28 11:45	PR #7782	2115409675166945280
2026-10-28 07:07	PR #7662	2115339656903004160

alert [hcp] KubeDeploymentReplicasMismatch fired

full failure pattern:

alert [hcp] KubeDeploymentReplicasMismatch fired

Signal:

alert

none

Full failure examples (3)

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-28T20:52:20Z
  Ended:       2026-10-28T21:03:23Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1157504-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.172:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1157504/providers/microsoft.monitor/accounts/hcps-j1157504", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-p48cd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-28T20:52:20Z
  Ended:       2026-10-28T21:03:23Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1157504-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.172:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1157504/providers/microsoft.monitor/accounts/hcps-j1157504", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-p48cd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-28T20:03:50Z
  Ended:       2026-10-28T20:15:46Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8717568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.254:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8717568/providers/microsoft.monitor/accounts/hcps-j8717568", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-59dhx", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-28T20:03:50Z
  Ended:       2026-10-28T20:15:46Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8717568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.254:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8717568/providers/microsoft.monitor/accounts/hcps-j8717568", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-59dhx", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-28T13:12:32Z
  Ended:       2026-10-28T13:28:32Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6945280-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.108:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6945280/providers/microsoft.monitor/accounts/hcps-j6945280", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-fct9q", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire	4

Affected runs (4)

Date (UTC)	Associated PR	Prow job
2026-10-28 19:15	PR #7666	2115522964811157504
2026-10-28 18:33	PR #7764	2115512320468717568
2026-10-28 11:45	PR #7782	2115409675166945280
2026-10-28 07:07	PR #7662	2115339656903004160

alert [hcp] KubePodCrashLooping fired

full failure pattern:

alert [hcp] KubePodCrashLooping fired

Signal:

alert

none

Full failure examples (3)

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-28T20:39:20Z
  Ended:       2026-10-28T21:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1157504-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.172:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1157504/providers/microsoft.monitor/accounts/hcps-j1157504", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="777a1913-78dc-4e11-9844-4f50d6675327"
  Description: Pod ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-28T20:40:18Z
  Ended:       2026-10-28T21:06:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1157504-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.172:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1157504/providers/microsoft.monitor/accounts/hcps-j1157504", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="777a1913-78dc-4e11-9844-4f50d6675327"
  Description: Pod ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-28T19:50:44Z
  Ended:       2026-10-28T20:18:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j8717568-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.254:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8717568/providers/microsoft.monitor/accounts/hcps-j8717568", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="aad698ce-6b47-41d1-8ddd-e6e4cb15b5d7"
  Description: Pod ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-28T19:50:44Z
  Ended:       2026-10-28T20:19:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j8717568-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.254:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8717568/providers/microsoft.monitor/accounts/hcps-j8717568", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="aad698ce-6b47-41d1-8ddd-e6e4cb15b5d7"
  Description: Pod ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-28T12:59:31Z
  Ended:       2026-10-28T13:28:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j6945280-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.108:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6945280/providers/microsoft.monitor/accounts/hcps-j6945280", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="5496ebfd-2ab4-43b8-9f9a-e0483581f4bd"
  Description: Pod ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-28T13:00:33Z
  Ended:       2026-10-28T13:27:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j6945280-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.108:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6945280/providers/microsoft.monitor/accounts/hcps-j6945280", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="5496ebfd-2ab4-43b8-9f9a-e0483581f4bd"
  Description: Pod ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubePodCrashLooping does not fire	4

Affected runs (4)

Date (UTC)	Associated PR	Prow job
2026-10-28 19:15	PR #7666	2115522964811157504
2026-10-28 18:33	PR #7764	2115512320468717568
2026-10-28 11:45	PR #7782	2115409675166945280
2026-10-28 07:07	PR #7662	2115339656903004160

alert [hcp] KubeStatefulSetReplicasMismatch fired

full failure pattern:

alert [hcp] KubeStatefulSetReplicasMismatch fired

Signal:

alert

none

Full failure examples (3)

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-28T20:37:24Z
  Ended:       2026-10-28T21:04:22Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j1157504-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.172:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1157504/providers/microsoft.monitor/accounts/hcps-j1157504", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-p48cd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-28T20:37:24Z
  Ended:       2026-10-28T21:01:24Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j1157504-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.172:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1157504/providers/microsoft.monitor/accounts/hcps-j1157504", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-p48cd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-28T19:48:54Z
  Ended:       2026-10-28T20:14:47Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j8717568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.254:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8717568/providers/microsoft.monitor/accounts/hcps-j8717568", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-59dhx", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-28T19:48:54Z
  Ended:       2026-10-28T20:14:47Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j8717568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.254:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8717568/providers/microsoft.monitor/accounts/hcps-j8717568", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-59dhx", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-28T12:57:34Z
  Ended:       2026-10-28T13:23:33Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j6945280-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.108:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6945280/providers/microsoft.monitor/accounts/hcps-j6945280", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-fct9q", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-28T12:57:34Z
  Ended:       2026-10-28T13:23:33Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j6945280-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.108:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6945280/providers/microsoft.monitor/accounts/hcps-j6945280", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-fct9q", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k/etcd has not matched the expected number of replicas for longer than 15 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubeStatefulSetReplicasMismatch does not fire	4

Affected runs (4)

Date (UTC)	Associated PR	Prow job
2026-10-28 19:15	PR #7666	2115522964811157504
2026-10-28 18:33	PR #7764	2115512320468717568
2026-10-28 11:45	PR #7782	2115409675166945280
2026-10-28 07:07	PR #7662	2115339656903004160

alert [hcp] KubePodNotReady fired

full failure pattern:

alert [hcp] KubePodNotReady fired

Signal:

alert

none

Full failure examples (3)

alert fired 44 time(s) (22 unknown, 22 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-29T00:44:39Z
  Ended:       2026-10-29T00:55:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4ck31s4rriq8m3aod51np4tkaadmt-cluster", pod="etcd-1", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2to4ck31s4rriq8m3aod51np4tkaadmt-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-29T00:44:39Z
  Ended:       2026-10-29T00:55:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4chff48f13vs6hovvitkv564933vi-cluster", pod="etcd-2", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2to4chff48f13vs6hovvitkv564933vi-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-29T00:44:40Z
  Ended:       2026-10-29T00:56:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4d8f2i0osrmij511ou0t7rfhmnlsn-cluster", pod="etcd-0", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2to4d8f2i0osrmij511ou0t7rfhmnlsn-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-10-29T00:44:40Z
  Ended:       2026-10-29T00:56:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4cmr4pghcqg0403a5jq11dvfnlikl-cluster", pod="etcd-0", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2to4cmr4pghcqg0403a5jq11dvfnlikl-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-10-29T00:44:40Z
  Ended:       2026-10-29T00:56:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4cm85p376bf4l9af5geeqascs19re-cluster", pod="etcd-1", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2to4cm85p376bf4l9af5geeqascs19re-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-10-29T00:44:40Z
  Ended:       2026-10-29T00:56:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4cldmf4jan6oonnq2tg3nk9r52s81-cluster", pod="etcd-1", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2to4cldmf4jan6oonnq2tg3nk9r52s81-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-10-29T00:44:40Z
  Ended:       2026-10-29T00:55:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4cm3bgo3iqkg6triqkq45fn7rdlr8-cluster", pod="etcd-2", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2to4cm3bgo3iqkg6triqkq45fn7rdlr8-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-10-29T00:45:40Z
  Ended:       2026-10-29T00:56:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4dtdrbk41153d35r2v928ecfh6bmp-cluster", pod="etcd-1", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2to4dtdrbk41153d35r2v928ecfh6bmp-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-10-29T00:45:41Z
  Ended:       2026-10-29T00:56:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4dgi32u5j8fk020vol0qqv1ds1k11-cluster", pod="etcd-1", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2to4dgi32u5j8fk020vol0qqv1ds1k11-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-10-29T00:45:41Z
  Ended:       2026-10-29T00:56:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4clhtd0hi1sub7sbhcb7dhg3pba48-cluster", pod="etcd-0", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2to4clhtd0hi1sub7sbhcb7dhg3pba48-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-10-29T00:45:41Z
  Ended:       2026-10-29T00:56:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4clv3ma8s477hmmkla42d167b93ol-cluster", pod="etcd-0", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2to4clv3ma8s477hmmkla42d167b93ol-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-10-29T00:46:40Z
  Ended:       2026-10-29T00:56:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4davicql96oofkg045duq6nghl7ff-cluster", pod="etcd-0", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2to4davicql96oofkg045duq6nghl7ff-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-10-29T00:46:40Z
  Ended:       2026-10-29T00:55:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4e557h8kaiucvu5gvvdsgqh4o581v-cluster", pod="etcd-2", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2to4e557h8kaiucvu5gvvdsgqh4o581v-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-10-29T00:46:40Z
  Ended:       2026-10-29T00:56:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4ct1nqdjgi22uib0cecj6up04l0m7-cluster", pod="etcd-2", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2to4ct1nqdjgi22uib0cecj6up04l0m7-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-10-29T00:46:41Z
  Ended:       2026-10-29T00:56:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4clpeml5pupvinmlvbpse0ie0hfs9-cluster", pod="etcd-0", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2to4clpeml5pupvinmlvbpse0ie0hfs9-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-10-29T00:46:41Z
  Ended:       2026-10-29T00:56:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4darhtjunfn21raucc3dfc59v09jp-cluster", pod="etcd-2", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2to4darhtjunfn21raucc3dfc59v09jp-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Resolved
  Started:     2026-10-29T00:46:41Z
  Ended:       2026-10-29T00:56:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4ckvnln5c26n1qo12ulvfvcdqeadr-cluster", pod="etcd-1", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2to4ckvnln5c26n1qo12ulvfvcdqeadr-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Resolved
  Started:     2026-10-29T00:46:41Z
  Ended:       2026-10-29T00:56:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4clafsmcnsm8b1l90dllklk71mg71-cluster", pod="etcd-1", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2to4clafsmcnsm8b1l90dllklk71mg71-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Resolved
  Started:     2026-10-29T00:46:41Z
  Ended:       2026-10-29T00:56:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4d88ealur630tj4sst5o6qkjn1dgl-cluster", pod="etcd-0", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2to4d88ealur630tj4sst5o6qkjn1dgl-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Resolved
  Started:     2026-10-29T00:46:41Z
  Ended:       2026-10-29T00:55:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4ctr3n2rtj0ed87918esf64n0iglc-cluster", pod="etcd-1", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2to4ctr3n2rtj0ed87918esf64n0iglc-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes.
Firing 21:
  State:       Resolved
  Started:     2026-10-29T00:46:41Z
  Ended:       2026-10-29T00:56:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4db5tooak0klk2r251u93t1sccjm0-cluster", pod="etcd-1", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2to4db5tooak0klk2r251u93t1sccjm0-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes.
Firing 22:
  State:       Resolved
  Started:     2026-10-29T00:46:41Z
  Ended:       2026-10-29T00:56:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4cl4qqobidbthts4n7r6dv1md7maq-cluster", pod="etcd-1", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2to4cl4qqobidbthts4n7r6dv1md7maq-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-28T23:18:11Z
  Ended:       2026-10-28T23:25:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3784704-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to2tdqitlfqi4phjdduu6fem2hl4t4t-s0w7b3l1w4i5e1k", pod="openshift-apiserver-55d5d58f8b-5ftzv", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2to2tdqitlfqi4phjdduu6fem2hl4t4t-s0w7b3l1w4i5e1k/openshift-apiserver-55d5d58f8b-5ftzv has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-28T20:00:50Z
  Ended:       2026-10-28T20:06:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5001984-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tnvsms9l35e1dde1cr2bnekd5epm5j9-ingress-cert-le", pod="openshift-apiserver-7cb4d98fc-rzx96", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tnvsms9l35e1dde1cr2bnekd5epm5j9-ingress-cert-le/openshift-apiserver-7cb4d98fc-rzx96 has been in a non-ready state for longer than 5 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubePodNotReady does not fire	3

Affected runs (3)

Date (UTC)	Associated PR	Prow job
2026-10-28 23:45	PR #7793	2115590796378902528
2026-10-28 21:44	PR #7693	2115560333123784704
2026-10-28 18:22	PR #7814	2115509651545001984

alert [svc] BackendControllerQueueDepthHigh fired

full failure pattern:

alert [svc] BackendControllerQueueDepthHigh fired

Signal:

alert

none

Full failure examples (2)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-28T21:09:13Z
  Ended:       2026-10-28T21:26:12Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j8906368-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-28T19:42:52Z
  Ended:       2026-10-28T20:02:53Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j3991936-svc", component="backend", name="csstatedump", region="westus3", severity="warning"
  Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-28 19:56	PR #7757	2115533269758906368
2026-10-28 18:23	PR #7785	2115509692493991936

alert [svc] KubePodNotReady fired

full failure pattern:

alert [svc] KubePodNotReady fired

Signal:

alert

none

Full failure examples (1)

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-28T17:07:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3195648-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-64b56768b9-r7v9d", region="canadacentral", severity="warning"
  Description: Pod clusters-service/clusters-service-64b56768b9-r7v9d has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-28T17:07:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3195648-svc", component="kubernetes-infrastructure", namespace="fleet", pod="fleet-controller-7b9596cccb-g9thf", region="canadacentral", severity="warning"
  Description: Pod fleet/fleet-controller-7b9596cccb-g9thf has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-28T17:07:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3195648-svc", component="kubernetes-infrastructure", namespace="aro-hcp-admin-api", pod="admin-api-6d56468ffd-t9snj", region="canadacentral", severity="warning"
  Description: Pod aro-hcp-admin-api/admin-api-6d56468ffd-t9snj has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-28T17:07:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3195648-svc", component="kubernetes-infrastructure", namespace="aro-hcp-admin-api", pod="admin-api-6d56468ffd-rm24l", region="canadacentral", severity="warning"
  Description: Pod aro-hcp-admin-api/admin-api-6d56468ffd-rm24l has been in a non-ready state for longer than 5 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert KubePodNotReady does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-28 16:20	PR #7311	2115478705303195648

ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<u...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.... message truncated; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:139]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
    <*fmt.wrapError | 0xc2b6fc92f40>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-4pl8j4mzdr2l" to finish creating: GET https://rp.j0412544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/67bb2317-9c7e-4d3b-8e3c-0ad00e76a125
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/67bb2317-9c7e-4d3b-8e3c-0ad00e76a125",
      "name": "67bb2317-9c7e-4d3b-8e3c-0ad00e76a125",
      "status": "Failed",
      "startTime": "2026-10-28T18:04:23.346337134Z",
      "endTime": "2026-10-28T18:23:31.470906003Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v2e1t9h4b7t5u0a-np-1-cd7br-b9qvn: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v2e1t9h4b7t5u0a-np-1-cd7br-b9qvn: Failed: interfaces failed to create or update. err: failed to get existing resource autoscaling-cluster-4pl8j4mzdr2l--managed/v2e1t9h4b7t5u0a-np-1-cd7br-b9qvn-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-4pl8j4mzdr2l--managed/providers/Microsoft.Network/networkInterfaces/v2e1t9h4b7t5u0a-np-1-cd7br-b9qvn-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'f0916b6b-4fcf-4b8b-b7f8-5a66898b1190' exceeded. Please try again after '2' seconds after additional tokens are available.... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j0412544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/67bb2317-9c7e-4d3b-8e3c-0ad00e76a125
            correlation ID: 7d414729-3b90-4c4b-9628-b2993ebd631c
            client request ID: 0c6fdc27-946b-4c40-9ac6-c7629afa8ed5
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:139]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
    <*fmt.wrapError | 0xc2b6fc92f40>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-4pl8j4mzdr2l" to finish creating: GET https://rp.j0412544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/67bb2317-9c7e-4d3b-8e3c-0ad00e76a125
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/67bb2317-9c7e-4d3b-8e3c-0ad00e76a125",
      "name": "67bb2317-9c7e-4d3b-8e3c-0ad00e76a125",
      "status": "Failed",
      "startTime": "2026-10-28T18:04:23.346337134Z",
      "endTime": "2026-10-28T18:23:31.470906003Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v2e1t9h4b7t5u0a-np-1-cd7br-b9qvn: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v2e1t9h4b7t5u0a-np-1-cd7br-b9qvn: Failed: interfaces failed to create or update. err: failed to get existing resource autoscaling-cluster-4pl8j4mzdr2l--managed/v2e1t9h4b7t5u0a-np-1-cd7br-b9qvn-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-4pl8j4mzdr2l--managed/providers/Microsoft.Network/networkInterfaces/v2e1t9h4b7t5u0a-np-1-cd7br-b9qvn-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'f0916b6b-4fcf-4b8b-b7f8-5a66898b1190' exceeded. Please try again after '2' seconds after additional tokens are available.... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j0412544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/67bb2317-9c7e-4d3b-8e3c-0ad00e76a125
            correlation ID: 7d414729-3b90-4c4b-9628-b2993ebd631c
            client request ID: 0c6fdc27-946b-4c40-9ac6-c7629afa8ed5
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-28 16:59	PR #7811	2115488597250412544

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want >= <count> (autoscaling min); node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x1c827a968340>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-9k6kks4sqmhh" to finish creating: GET https://rp.j5392000.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e7e6b86c-a0ff-4c35-94e7-3aa8b285e19e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e7e6b86c-a0ff-4c35-94e7-3aa8b285e19e",
      "name": "e7e6b86c-a0ff-4c35-94e7-3aa8b285e19e",
      "status": "Failed",
      "startTime": "2026-10-28T17:17:13.271950373Z",
      "endTime": "2026-10-28T17:36:19.733049446Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine x5b5y8h0f8t5k2t-autoscale-noaz-b47sx-574cm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-9k6kks4sqmhh--managed/providers/Microsoft.Compute/virtualMachines/x5b5y8h0f8t5k2t-autoscale-noaz-b47sx-574cm to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine x5b5y8h0f8t5k2t-autoscale-noaz-b47sx-574cm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-9k6kks4sqmhh--managed/providers/Microsoft.Compute/virtualMachines/x5b5y8h0f8t5k2t-autoscale-noaz-b47sx-574cm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5392000.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e7e6b86c-a0ff-4c35-94e7-3aa8b285e19e
            correlation ID: e5b7a290-af02-4ff7-9f0a-851a1b652878
            client request ID: a046022e-1fb3-43ea-9318-d95c8087e3da
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x1c827a968340>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-9k6kks4sqmhh" to finish creating: GET https://rp.j5392000.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e7e6b86c-a0ff-4c35-94e7-3aa8b285e19e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e7e6b86c-a0ff-4c35-94e7-3aa8b285e19e",
      "name": "e7e6b86c-a0ff-4c35-94e7-3aa8b285e19e",
      "status": "Failed",
      "startTime": "2026-10-28T17:17:13.271950373Z",
      "endTime": "2026-10-28T17:36:19.733049446Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine x5b5y8h0f8t5k2t-autoscale-noaz-b47sx-574cm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-9k6kks4sqmhh--managed/providers/Microsoft.Compute/virtualMachines/x5b5y8h0f8t5k2t-autoscale-noaz-b47sx-574cm to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine x5b5y8h0f8t5k2t-autoscale-noaz-b47sx-574cm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-9k6kks4sqmhh--managed/providers/Microsoft.Compute/virtualMachines/x5b5y8h0f8t5k2t-autoscale-noaz-b47sx-574cm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5392000.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e7e6b86c-a0ff-4c35-94e7-3aa8b285e19e
            correlation ID: e5b7a290-af02-4ff7-9f0a-851a1b652878
            client request ID: a046022e-1fb3-43ea-9318-d95c8087e3da
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-28 16:10	PR #7809	2115476335995392000

ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<u...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0x1ddd7d2831b8>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-zk7rbqmsp2zd" to finish creating: GET https://rp.j8902528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cb6f6658-c768-40a6-b3cf-8a5bd044305f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cb6f6658-c768-40a6-b3cf-8a5bd044305f",
      "name": "cb6f6658-c768-40a6-b3cf-8a5bd044305f",
      "status": "Failed",
      "startTime": "2026-10-29T00:42:47.491540846Z",
      "endTime": "2026-10-29T01:01:51.350365806Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-cilium-np-ztzhx-l97nr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-zk7rbqmsp2zd--managed/providers/Microsoft.Compute/virtualMachines/cluster-cilium-np-ztzhx-l97nr to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-cilium-np-ztzhx-547rj: Failed: interfaces failed to create or update. err: failed to get existing resource cni-cilium-zk7rbqmsp2zd--managed/cluster-cilium-np-ztzhx-547rj-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-zk7rbqmsp2zd--managed/providers/Microsoft.Network/networkInterfaces/cluster-cilium-np-ztzhx-547rj-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'b117cf0f-0276-4486-9691-4f00f5a90e1b' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/a... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8902528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cb6f6658-c768-40a6-b3cf-8a5bd044305f
            correlation ID: 52dc9816-9bf4-4c02-9a62-7105502ff251
            client request ID: 3591f936-ad0d-47b0-b46a-a9f82bf4e6ae
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0x1ddd7d2831b8>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-zk7rbqmsp2zd" to finish creating: GET https://rp.j8902528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cb6f6658-c768-40a6-b3cf-8a5bd044305f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cb6f6658-c768-40a6-b3cf-8a5bd044305f",
      "name": "cb6f6658-c768-40a6-b3cf-8a5bd044305f",
      "status": "Failed",
      "startTime": "2026-10-29T00:42:47.491540846Z",
      "endTime": "2026-10-29T01:01:51.350365806Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-cilium-np-ztzhx-l97nr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-zk7rbqmsp2zd--managed/providers/Microsoft.Compute/virtualMachines/cluster-cilium-np-ztzhx-l97nr to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-cilium-np-ztzhx-547rj: Failed: interfaces failed to create or update. err: failed to get existing resource cni-cilium-zk7rbqmsp2zd--managed/cluster-cilium-np-ztzhx-547rj-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-zk7rbqmsp2zd--managed/providers/Microsoft.Network/networkInterfaces/cluster-cilium-np-ztzhx-547rj-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'b117cf0f-0276-4486-9691-4f00f5a90e1b' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/a... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8902528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cb6f6658-c768-40a6-b3cf-8a5bd044305f
            correlation ID: 52dc9816-9bf4-4c02-9a62-7105502ff251
            client request ID: 3591f936-ad0d-47b0-b46a-a9f82bf4e6ae
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0x740ee223cf8>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-dmmfjzfldpm8" to finish creating: GET https://rp.j8717568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e51bb1e4-4996-454f-a81d-9354613a75ee
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e51bb1e4-4996-454f-a81d-9354613a75ee",
      "name": "e51bb1e4-4996-454f-a81d-9354613a75ee",
      "status": "Failed",
      "startTime": "2026-10-28T19:36:46.639467207Z",
      "endTime": "2026-10-28T19:55:53.152282953Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-8z842-x96hm: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cl-cilium-np-8z842-tlsx7: Failed: interfaces failed to create or update. err: failed to get existing resource cni-cilium-dmmfjzfldpm8--managed/cilium-cl-cilium-np-8z842-tlsx7-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-dmmfjzfldpm8--managed/providers/Microsoft.Network/networkInterfaces/cilium-cl-cilium-np-8z842-tlsx7-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '1d2f4929-515e-4e8d-8580-836e819f8d50' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8717568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e51bb1e4-4996-454f-a81d-9354613a75ee
            correlation ID: 4963ae88-5109-4671-873f-f28a9deccdac
            client request ID: 38a19ead-d2a2-4130-9e04-1364166950d2
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0x740ee223cf8>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-dmmfjzfldpm8" to finish creating: GET https://rp.j8717568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e51bb1e4-4996-454f-a81d-9354613a75ee
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e51bb1e4-4996-454f-a81d-9354613a75ee",
      "name": "e51bb1e4-4996-454f-a81d-9354613a75ee",
      "status": "Failed",
      "startTime": "2026-10-28T19:36:46.639467207Z",
      "endTime": "2026-10-28T19:55:53.152282953Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-8z842-x96hm: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cl-cilium-np-8z842-tlsx7: Failed: interfaces failed to create or update. err: failed to get existing resource cni-cilium-dmmfjzfldpm8--managed/cilium-cl-cilium-np-8z842-tlsx7-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-dmmfjzfldpm8--managed/providers/Microsoft.Network/networkInterfaces/cilium-cl-cilium-np-8z842-tlsx7-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '1d2f4929-515e-4e8d-8580-836e819f8d50' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8717568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e51bb1e4-4996-454f-a81d-9354613a75ee
            correlation ID: 4963ae88-5109-4671-873f-f28a9deccdac
            client request ID: 38a19ead-d2a2-4130-9e04-1364166950d2
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0x335ced2f1b40>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-kfwkzrbn6vbm" to finish creating: GET https://rp.j4206336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/77628145-9ee8-4d12-9dd2-5dced29b5ecb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/77628145-9ee8-4d12-9dd2-5dced29b5ecb",
      "name": "77628145-9ee8-4d12-9dd2-5dced29b5ecb",
      "status": "Failed",
      "startTime": "2026-10-28T16:28:12.935779531Z",
      "endTime": "2026-10-28T16:47:20.605052936Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j6k4t1t6z4j2k5z-np-1-5nm58-fsnz7: Failed: virtualmachine failed to create or update. err: failed to get existing resource fips-enabled-kfwkzrbn6vbm--managed/j6k4t1t6z4j2k5z-np-1-5nm58-fsnz7 (service: virtualmachine): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-kfwkzrbn6vbm--managed/providers/Microsoft.Compute/virtualMachines/j6k4t1t6z4j2k5z-np-1-5nm58-fsnz7\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'b117cf0f-0276-4486-9691-4f00f5a90e1b' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https:... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j4206336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/77628145-9ee8-4d12-9dd2-5dced29b5ecb
            correlation ID: c83d59d4-8297-41d1-87c1-fbe0352a6163
            client request ID: 4dcc2ce6-6521-4d7f-b86f-f0309a80ecee
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0x335ced2f1b40>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-kfwkzrbn6vbm" to finish creating: GET https://rp.j4206336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/77628145-9ee8-4d12-9dd2-5dced29b5ecb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/77628145-9ee8-4d12-9dd2-5dced29b5ecb",
      "name": "77628145-9ee8-4d12-9dd2-5dced29b5ecb",
      "status": "Failed",
      "startTime": "2026-10-28T16:28:12.935779531Z",
      "endTime": "2026-10-28T16:47:20.605052936Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j6k4t1t6z4j2k5z-np-1-5nm58-fsnz7: Failed: virtualmachine failed to create or update. err: failed to get existing resource fips-enabled-kfwkzrbn6vbm--managed/j6k4t1t6z4j2k5z-np-1-5nm58-fsnz7 (service: virtualmachine): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-kfwkzrbn6vbm--managed/providers/Microsoft.Compute/virtualMachines/j6k4t1t6z4j2k5z-np-1-5nm58-fsnz7\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'b117cf0f-0276-4486-9691-4f00f5a90e1b' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https:... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j4206336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/77628145-9ee8-4d12-9dd2-5dced29b5ecb
            correlation ID: c83d59d4-8297-41d1-87c1-fbe0352a6163
            client request ID: 4dcc2ce6-6521-4d7f-b86f-f0309a80ecee
occurred

Contributing tests (3)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a HCP cluster and use cilium CNI plugin	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster using v20260901preview API and verify cluster health	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property	1

Affected runs (4)

Date (UTC)	Associated PR	Prow job
2026-10-28 23:45	PR #7793	2115590796378902528
2026-10-28 18:33	PR #7764	2115512320468717568
2026-10-28 15:15	PR #7704	2115462359144206336
2026-10-28 05:51	PR #7795	2115320649005142016

alert [svc] KubePodCrashLooping fired

full failure pattern:

alert [svc] KubePodCrashLooping fired

Signal:

alert

none

Full failure examples (3)

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-28T13:27:25Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0229248-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.125:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0229248/providers/microsoft.monitor/accounts/services-j0229248", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-phdkm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="5471b0ac-6a29-48cb-ab86-2893cd772fbd"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-phdkm (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-28T13:29:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0229248-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.195:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0229248/providers/microsoft.monitor/accounts/services-j0229248", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-84g47", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="9462daaa-47d3-4b5e-b73b-01a9f857a137"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-84g47 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-28T13:29:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0229248-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.195:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0229248/providers/microsoft.monitor/accounts/services-j0229248", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-84g47", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="9462daaa-47d3-4b5e-b73b-01a9f857a137"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-84g47 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-28T13:32:24Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0229248-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.125:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0229248/providers/microsoft.monitor/accounts/services-j0229248", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-phdkm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="5471b0ac-6a29-48cb-ab86-2893cd772fbd"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-phdkm (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-28T10:42:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1445504-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.26:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1445504/providers/microsoft.monitor/accounts/services-j1445504", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-5dmcb", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="6520e185-94fe-45c4-99a9-813a1698d907"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-5dmcb (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-28T10:43:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1445504-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.97:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1445504/providers/microsoft.monitor/accounts/services-j1445504", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-ts4n7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="0a28f799-e0bd-4698-b612-8c6756c4b517"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-ts4n7 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-28T10:43:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1445504-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.97:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1445504/providers/microsoft.monitor/accounts/services-j1445504", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-ts4n7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="0a28f799-e0bd-4698-b612-8c6756c4b517"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-ts4n7 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-28T10:47:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1445504-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.26:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1445504/providers/microsoft.monitor/accounts/services-j1445504", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-5dmcb", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="6520e185-94fe-45c4-99a9-813a1698d907"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-5dmcb (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-28T08:13:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3004160-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.217:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3004160/providers/microsoft.monitor/accounts/services-j3004160", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-krfd7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="2298911c-0832-493f-a9ef-f65728646e11"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-krfd7 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-28T08:13:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3004160-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.217:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3004160/providers/microsoft.monitor/accounts/services-j3004160", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-krfd7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="2298911c-0832-493f-a9ef-f65728646e11"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-krfd7 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-28T08:15:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3004160-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.208:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3004160/providers/microsoft.monitor/accounts/services-j3004160", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-fgmvz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e461328c-3664-4b8e-907f-f8f42456ba46"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-fgmvz (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-28T08:15:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3004160-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.208:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3004160/providers/microsoft.monitor/accounts/services-j3004160", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-fgmvz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e461328c-3664-4b8e-907f-f8f42456ba46"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-fgmvz (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert KubePodCrashLooping does not fire	4

Affected runs (4)

Date (UTC)	Associated PR	Prow job
2026-10-28 12:14	PR #7662	2115416945330229248
2026-10-28 09:35	PR #7662	2115376885591445504
2026-10-28 07:07	PR #7662	2115339656903004160
2026-10-28 03:31	PR #7662	2115285263486816256

ERROR CODE: DeploymentFailed; detail code AzureKeyVaultKmsValidateIdentityPermissionCustomerError; detail message The identity does not have keys encrypt/decrypt permission on key...

full failure pattern:

ERROR CODE: DeploymentFailed; detail code AzureKeyVaultKmsValidateIdentityPermissionCustomerError; detail message The identity does not have keys encrypt/decrypt permission on key vault "<url>"; provider Microsoft.ContainerService

Signal:

provision

none

Full failure examples (3)

time=2026-10-29T00:10:51.309Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-29T00:10:53.106Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-29T00:10:54.792Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=3eb04a406861dadce17733f3106b46e7d01c706185214584bf436d2dff1602d4 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j4319744-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F3eb04a406861dadce17733f3106b46e7d01c706185214584bf436d2dff1602d4
time=2026-10-29T00:12:55.689Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster step=cluster error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4319744-svc/providers/Microsoft.Resources/deployments/3eb04a406861dadce17733f3106b46e7d01c706185214584bf436d2dff1602d4/operationStatuses/08584083742316437961\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4319744-svc/providers/Microsoft.Resources/deployments/cluster-6um4nhqkdznjc\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4319744-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j4319744-svc\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"AzureKeyVaultKmsValidateIdentityPermissionCustomerError\\\",\\r\\n                \\\"message\\\": \\\"The identity does not have keys encrypt/decrypt permission on key vault \\\\\\\"https://ah-ci01-se-j4319744-1.vault.azure.net\\\\\\\"\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
time=2026-10-29T00:14:55.689Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-29T00:14:56.876Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-29T00:14:58.481Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=3d3c29485896acfd7e15d9c09cc1476e363573f3512d196c907c860f06655a45 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j4319744-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F3d3c29485896acfd7e15d9c09cc1476e363573f3512d196c907c860f06655a45
time=2026-10-29T00:16:28.867Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4319744-svc/providers/Microsoft.Resources/deployments/3d3c29485896acfd7e15d9c09cc1476e363573f3512d196c907c860f06655a45/operationStatuses/08584083739878467732\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"Failed to obtain the resource body, because the resource is currently in a 'Failed' provisioning state. If this resource has not been modified by a prior step in this deployment, you may need to address this before retrying the deployment.\\\"\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

time=2026-10-28T16:43:28.511Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-28T16:43:31.273Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-28T16:43:32.448Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=f309630496c74d69129833a43742f2e9eec7ceb9d4411ea07b979bc73b89fd87 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j2862080-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Ff309630496c74d69129833a43742f2e9eec7ceb9d4411ea07b979bc73b89fd87
time=2026-10-28T16:45:32.811Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster step=cluster error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2862080-svc/providers/Microsoft.Resources/deployments/f309630496c74d69129833a43742f2e9eec7ceb9d4411ea07b979bc73b89fd87/operationStatuses/08584084010736719502\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2862080-svc/providers/Microsoft.Resources/deployments/cluster-rybldmoz73lrm\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2862080-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j2862080-svc\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"AzureKeyVaultKmsValidateIdentityPermissionCustomerError\\\",\\r\\n                \\\"message\\\": \\\"The identity does not have keys encrypt/decrypt permission on key vault \\\\\\\"https://ah-ci01-se-j2862080-1.vault.azure.net\\\\\\\"\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
time=2026-10-28T16:47:32.817Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-28T16:47:34.143Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-28T16:47:35.068Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=8fa925f61f96a8233f154e3da5c65d168518bb692c424a9f42720dad4bf60006 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j2862080-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F8fa925f61f96a8233f154e3da5c65d168518bb692c424a9f42720dad4bf60006
time=2026-10-28T16:49:05.326Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2862080-svc/providers/Microsoft.Resources/deployments/8fa925f61f96a8233f154e3da5c65d168518bb692c424a9f42720dad4bf60006/operationStatuses/08584084008309075646\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"Failed to obtain the resource body, because the resource is currently in a 'Failed' provisioning state. If this resource has not been modified by a prior step in this deployment, you may need to address this before retrying the deployment.\\\"\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

time=2026-10-28T15:32:48.082Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-28T15:32:50.394Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-28T15:32:51.656Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=51bb59b51713025e67da1bbb1f81509d9f0e687a1b531b2b697347d81ab333d6 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1862528-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F51bb59b51713025e67da1bbb1f81509d9f0e687a1b531b2b697347d81ab333d6
time=2026-10-28T15:35:22.185Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster step=cluster error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1862528-svc/providers/Microsoft.Resources/deployments/51bb59b51713025e67da1bbb1f81509d9f0e687a1b531b2b697347d81ab333d6/operationStatuses/08584084053144753487\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1862528-svc/providers/Microsoft.Resources/deployments/cluster-5xxgmkpcjnpl2\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1862528-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j1862528-svc\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"AzureKeyVaultKmsValidateIdentityPermissionCustomerError\\\",\\r\\n                \\\"message\\\": \\\"The identity does not have keys encrypt/decrypt permission on key vault \\\\\\\"https://ah-ci01-se-j1862528-1.vault.azure.net\\\\\\\"\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
time=2026-10-28T15:37:22.187Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-28T15:37:23.185Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-28T15:37:24.435Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=f330c62a337aea146d3097645390c7352ebc193053cfec3eadc02b161a2aba5f portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1862528-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Ff330c62a337aea146d3097645390c7352ebc193053cfec3eadc02b161a2aba5f
time=2026-10-28T15:38:54.741Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1862528-svc/providers/Microsoft.Resources/deployments/f330c62a337aea146d3097645390c7352ebc193053cfec3eadc02b161a2aba5f/operationStatuses/08584084050417029665\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"Failed to obtain the resource body, because the resource is currently in a 'Failed' provisioning state. If this resource has not been modified by a prior step in this deployment, you may need to address this before retrying the deployment.\\\"\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster	3

Affected runs (3)

Date (UTC)	Associated PR	Prow job
2026-10-28 23:45	PR #7814	2115590812124319744
2026-10-28 16:19	PR #7693	2115478468882862080
2026-10-28 15:05	PR #7781	2115460075651862528

ERROR CODE: ResourceGroupNotFound; provider Microsoft.AlertsManagement

full failure pattern:

ERROR CODE: ResourceGroupNotFound; provider Microsoft.AlertsManagement

Signal:

alert

none

Full failure examples (3)

alert collection incomplete
failed to fetch hcp alert rules: failed to list prometheus rule groups: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8739840/providers/Microsoft.AlertsManagement/prometheusRuleGroups
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceGroupNotFound",
    "message": "Resource group 'hcp-underlay-ci01-j8739840' could not be found."
  }
}
--------------------------------------------------------------------------------

alert collection incomplete
failed to fetch svc alert rules: failed to list prometheus rule groups: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8739840/providers/Microsoft.AlertsManagement/prometheusRuleGroups
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceGroupNotFound",
    "message": "Resource group 'hcp-underlay-ci01-j8739840' could not be found."
  }
}
--------------------------------------------------------------------------------

alert collection incomplete
failed to fetch hcp alert rules: failed to list prometheus rule groups: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2704640/providers/Microsoft.AlertsManagement/prometheusRuleGroups
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceGroupNotFound",
    "message": "Resource group 'hcp-underlay-ci01-j2704640' could not be found."
  }
}
--------------------------------------------------------------------------------

Contributing tests (2)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert collection is complete	2
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert collection is complete	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-28 14:30	PR #7793	2115451033688739840
2026-10-28 01:41	PR #7662	2115257563342704640

certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist

full failure pattern:

certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist

Signal:

e2e

STG

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:510]: Timed out after 600.004s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0x1212f5b25700>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:510]: Timed out after 600.004s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0x1212f5b25700>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:510]: Timed out after 600.000s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0x2ef66de74fd0>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:510]: Timed out after 600.000s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0x2ef66de74fd0>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }

Contributing tests (2)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 4.22	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 5.0	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-28 22:37	PR #7651	2115573733652434944
2026-10-28 22:33	PR #7823	2115572620593205248

VerifyAllAPIServicesAvailable failed: apiservice/v1beta1

full failure pattern:

VerifyAllAPIServicesAvailable failed: apiservice/v1beta1

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:183]: cluster viability check failed
Expected success, but got an error:
    <*errors.joinError | 0x311de884c030>: 
    VerifyAllAPIServicesAvailable failed: apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.129.0.21:10250/apis/metrics.k8s.io/v1beta1: Get "https://10.129.0.21:10250/apis/metrics.k8s.io/v1beta1": tls: failed to verify certificate: x509: certificate signed by unknown authority
    {
        errs: [
            <*fmt.wrapError | 0x311de7428000>{
                msg: "VerifyAllAPIServicesAvailable failed: apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.129.0.21:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.129.0.21:10250/apis/metrics.k8s.io/v1beta1\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
                err: <*errors.joinError | 0x311de82372a8>{
                    errs: [
                        <*errors.errorString | 0x311de8b96230>{
                            s: "apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.129.0.21:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.129.0.21:10250/apis/metrics.k8s.io/v1beta1\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
                        },
                    ],
                },
            },
        ],
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:183]: cluster viability check failed
Expected success, but got an error:
    <*errors.joinError | 0x311de884c030>: 
    VerifyAllAPIServicesAvailable failed: apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.129.0.21:10250/apis/metrics.k8s.io/v1beta1: Get "https://10.129.0.21:10250/apis/metrics.k8s.io/v1beta1": tls: failed to verify certificate: x509: certificate signed by unknown authority
    {
        errs: [
            <*fmt.wrapError | 0x311de7428000>{
                msg: "VerifyAllAPIServicesAvailable failed: apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.129.0.21:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.129.0.21:10250/apis/metrics.k8s.io/v1beta1\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
                err: <*errors.joinError | 0x311de82372a8>{
                    errs: [
                        <*errors.errorString | 0x311de8b96230>{
                            s: "apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.129.0.21:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.129.0.21:10250/apis/metrics.k8s.io/v1beta1\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
                        },
                    ],
                },
            },
        ],
    }

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:183]: cluster viability check failed
Expected success, but got an error:
    <*errors.joinError | 0xc83633402b8>: 
    VerifyAllAPIServicesAvailable failed: apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.128.0.29:10250/apis/metrics.k8s.io/v1beta1: Get "https://10.128.0.29:10250/apis/metrics.k8s.io/v1beta1": tls: failed to verify certificate: x509: certificate signed by unknown authority
    {
        errs: [
            <*fmt.wrapError | 0xc83630bcee0>{
                msg: "VerifyAllAPIServicesAvailable failed: apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.128.0.29:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.128.0.29:10250/apis/metrics.k8s.io/v1beta1\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
                err: <*errors.joinError | 0xc8362e9fb78>{
                    errs: [
                        <*errors.errorString | 0xc83633b5610>{
                            s: "apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.128.0.29:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.128.0.29:10250/apis/metrics.k8s.io/v1beta1\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
                        },
                    ],
                },
            },
        ],
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:183]: cluster viability check failed
Expected success, but got an error:
    <*errors.joinError | 0xc83633402b8>: 
    VerifyAllAPIServicesAvailable failed: apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.128.0.29:10250/apis/metrics.k8s.io/v1beta1: Get "https://10.128.0.29:10250/apis/metrics.k8s.io/v1beta1": tls: failed to verify certificate: x509: certificate signed by unknown authority
    {
        errs: [
            <*fmt.wrapError | 0xc83630bcee0>{
                msg: "VerifyAllAPIServicesAvailable failed: apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.128.0.29:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.128.0.29:10250/apis/metrics.k8s.io/v1beta1\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
                err: <*errors.joinError | 0xc8362e9fb78>{
                    errs: [
                        <*errors.errorString | 0xc83633b5610>{
                            s: "apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.128.0.29:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.128.0.29:10250/apis/metrics.k8s.io/v1beta1\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
                        },
                    ],
                },
            },
        ],
    }

Contributing tests (2)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 5.0	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 5.1	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-28 23:23	PR #7670	2115585270475657216
2026-10-28 12:14	PR #7662	2115416945330229248

ERROR CODE: ResourceGroupNotFound; provider Microsoft.Insights

full failure pattern:

ERROR CODE: ResourceGroupNotFound; provider Microsoft.Insights

Signal:

alert

none

Full failure examples (2)

alert collection incomplete
failed to fetch metric alert rules for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8739840: failed to list metric alert rules: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8739840/providers/Microsoft.Insights/metricAlerts
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceGroupNotFound",
    "message": "Resource group 'hcp-underlay-ci01-j8739840' could not be found."
  }
}
--------------------------------------------------------------------------------

alert collection incomplete
failed to fetch metric alert rules for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2704640: failed to list metric alert rules: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2704640/providers/Microsoft.Insights/metricAlerts
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceGroupNotFound",
    "message": "Resource group 'hcp-underlay-ci01-j2704640' could not be found."
  }
}
--------------------------------------------------------------------------------

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [infra] alert collection is complete	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-28 14:30	PR #7793	2115451033688739840
2026-10-28 01:41	PR #7662	2115257563342704640

VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node <node> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "<node>": nodes "<node>" ...

full failure pattern:

VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node <node> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "<node>": nodes "<node>" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-1-nxx9xr"
Unexpected error:
    <*errors.joinError | 0x138a2d1f8048>: 
    VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb": nodes "e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
    {
        errs: [
            <*fmt.wrapError | 0x138a2cdf5020>{
                msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\": nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                err: <*fmt.wrapError | 0x138a2cdf5000>{
                    msg: "failed to label node e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\": nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                    err: <*fmt.wrapError | 0x138a2cdf4fe0>{
                        msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\": nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                        err: <*errors.StatusError | 0x138a2cd0dcc0>{
                            ErrStatus: {
                                TypeMeta: {Kind: "", APIVersion: ""},
                                ListMeta: {
                                    SelfLink: "",
                                    ResourceVersion: "",
                                    Continue: "",
                                    RemainingItemCount: nil,
                                    ShardInfo: nil,
                                },
                                Status: "Failure",
                                Message: "nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                                Reason: "Forbidden",
                                Details: {
                                    Name: "e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb",
                                    Group: "",
                                    Kind: "nodes",
                                    UID: "",
                                    Causes: nil,
                                    RetryAfterSeconds: 0,
                                },
                                Code: 403,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-1-nxx9xr"
Unexpected error:
    <*errors.joinError | 0x138a2d1f8048>: 
    VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb": nodes "e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
    {
        errs: [
            <*fmt.wrapError | 0x138a2cdf5020>{
                msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\": nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                err: <*fmt.wrapError | 0x138a2cdf5000>{
                    msg: "failed to label node e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\": nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                    err: <*fmt.wrapError | 0x138a2cdf4fe0>{
                        msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\": nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                        err: <*errors.StatusError | 0x138a2cd0dcc0>{
                            ErrStatus: {
                                TypeMeta: {Kind: "", APIVersion: ""},
                                ListMeta: {
                                    SelfLink: "",
                                    ResourceVersion: "",
                                    Continue: "",
                                    RemainingItemCount: nil,
                                    ShardInfo: nil,
                                },
                                Status: "Failure",
                                Message: "nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                                Reason: "Forbidden",
                                Details: {
                                    Name: "e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb",
                                    Group: "",
                                    Kind: "nodes",
                                    UID: "",
                                    Causes: nil,
                                    RetryAfterSeconds: 0,
                                },
                                Code: 403,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-1-x4d58d"
Unexpected error:
    <*errors.joinError | 0x2a036e0a6030>: 
    VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc": nodes "t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
    {
        errs: [
            <*fmt.wrapError | 0x2a036cc0d0a0>{
                msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\": nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                err: <*fmt.wrapError | 0x2a036cc0d080>{
                    msg: "failed to label node t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\": nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                    err: <*fmt.wrapError | 0x2a036cc0d060>{
                        msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\": nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                        err: <*errors.StatusError | 0x2a036cc3c780>{
                            ErrStatus: {
                                TypeMeta: {Kind: "", APIVersion: ""},
                                ListMeta: {
                                    SelfLink: "",
                                    ResourceVersion: "",
                                    Continue: "",
                                    RemainingItemCount: nil,
                                    ShardInfo: nil,
                                },
                                Status: "Failure",
                                Message: "nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                                Reason: "Forbidden",
                                Details: {
                                    Name: "t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc",
                                    Group: "",
                                    Kind: "nodes",
                                    UID: "",
                                    Causes: nil,
                                    RetryAfterSeconds: 0,
                                },
                                Code: 403,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-1-x4d58d"
Unexpected error:
    <*errors.joinError | 0x2a036e0a6030>: 
    VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc": nodes "t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
    {
        errs: [
            <*fmt.wrapError | 0x2a036cc0d0a0>{
                msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\": nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                err: <*fmt.wrapError | 0x2a036cc0d080>{
                    msg: "failed to label node t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\": nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                    err: <*fmt.wrapError | 0x2a036cc0d060>{
                        msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\": nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                        err: <*errors.StatusError | 0x2a036cc3c780>{
                            ErrStatus: {
                                TypeMeta: {Kind: "", APIVersion: ""},
                                ListMeta: {
                                    SelfLink: "",
                                    ResourceVersion: "",
                                    Continue: "",
                                    RemainingItemCount: nil,
                                    ShardInfo: nil,
                                },
                                Status: "Failure",
                                Message: "nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                                Reason: "Forbidden",
                                Details: {
                                    Name: "t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc",
                                    Group: "",
                                    Kind: "nodes",
                                    UID: "",
                                    Causes: nil,
                                    RetryAfterSeconds: 0,
                                },
                                Code: 403,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.1	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-28 18:33	PR #7816	2115512185487626240
2026-10-28 11:14	PR #7786	2115401780928974848

DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline...

full failure pattern:

DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline exceeded

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:116]: failed to verify simple web app runs on cluster "cluster-hs-pre-krfsmn"
Unexpected error:
    <*fmt.wrapError | 0x34ab4bc30760>: 
    DNS for route host agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x34ab4c426760>{
            msg: "DNS for agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:116]: failed to verify simple web app runs on cluster "cluster-hs-pre-krfsmn"
Unexpected error:
    <*fmt.wrapError | 0x34ab4bc30760>: 
    DNS for route host agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x34ab4c426760>{
            msg: "DNS for agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:116]: failed to verify simple web app runs on cluster "cluster-hs-pre-5pj7h6"
Unexpected error:
    <*fmt.wrapError | 0x2590ea2143e0>: 
    DNS for route host agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x2590e99482e0>{
            msg: "DNS for agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:116]: failed to verify simple web app runs on cluster "cluster-hs-pre-5pj7h6"
Unexpected error:
    <*fmt.wrapError | 0x2590ea2143e0>: 
    DNS for route host agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x2590e99482e0>{
            msg: "DNS for agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-28 11:14	PR #7786	2115401780928974848
2026-10-28 07:14	PR #7797	2115341484919099392

ERROR CODE: DeploymentFailed; detail code AssociationFailedAtResourceProvider; detail message Association provisioning failed at resource provider; provider Microsoft.Network

full failure pattern:

ERROR CODE: DeploymentFailed; detail code AssociationFailedAtResourceProvider; detail message Association provisioning failed at resource provider; provider Microsoft.Network

Signal:

provision

none

Full failure examples (2)

time=2026-10-28T22:46:09.268Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=1 description="Step nsp\n  Kind: ARM\n  Template: templates/mgmt-nsp.bicep\n  Parameters: configurations/mgmt-nsp.tmpl.bicepparam"
time=2026-10-28T22:46:09.760Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=1
time=2026-10-28T22:46:10.194Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=1 deployment=351403f99eccc84e000639428bd1b6b39bdb6ecf570569167f5f1b17cdb7aee2 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j9857152-mgmt-1%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F351403f99eccc84e000639428bd1b6b39bdb6ecf570569167f5f1b17cdb7aee2
time=2026-10-28T22:47:10.427Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=1 err="stamp 1: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9857152-mgmt-1/providers/Microsoft.Resources/deployments/351403f99eccc84e000639428bd1b6b39bdb6ecf570569167f5f1b17cdb7aee2/operationStatuses/08584083793155131248\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9857152-mgmt-1/providers/Microsoft.Resources/deployments/profile-ufpn5r2pwmb4i\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 824ad833-5117-4ca4-a2e6-1002e46997d8\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9857152-mgmt-1/providers/Microsoft.Network/networkSecurityPerimeters/nsp-j9857152-mgmt-1/resourceAssociations/nsp-j9857152-mgmt-1-ra-67927217-e211-550f-abbf-b3e73881a01a\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"AssociationFailedAtResourceProvider\\\",\\r\\n                \\\"message\\\": \\\"Association provisioning failed at resource provider\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

time=2026-10-28T13:56:43.120Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=2 description="Step nsp\n  Kind: ARM\n  Template: templates/mgmt-nsp.bicep\n  Parameters: configurations/mgmt-nsp.tmpl.bicepparam"
time=2026-10-28T13:56:43.546Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=2
time=2026-10-28T13:56:43.936Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=2 deployment=27369930d49e6b87bdd6e0a76bb435d03ceef9149d6c362d06dfe1d47f9f5244 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j7323392-mgmt-2%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F27369930d49e6b87bdd6e0a76bb435d03ceef9149d6c362d06dfe1d47f9f5244
time=2026-10-28T13:58:14.248Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=2 err="stamp 2: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7323392-mgmt-2/providers/Microsoft.Resources/deployments/27369930d49e6b87bdd6e0a76bb435d03ceef9149d6c362d06dfe1d47f9f5244/operationStatuses/08584084110817295270\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7323392-mgmt-2/providers/Microsoft.Resources/deployments/profile-sxizw5tprl6fg\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7323392-mgmt-2/providers/Microsoft.Network/networkSecurityPerimeters/nsp-j7323392-mgmt-2/resourceAssociations/nsp-j7323392-mgmt-2-ra-4bf3dc7e-4183-5ff8-b0f7-92d18b1331a1\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"AssociationFailedAtResourceProvider\\\",\\r\\n                \\\"message\\\": \\\"Association provisioning failed at resource provider\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

Contributing tests (2)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/nsp (stamp=1)	1
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/nsp (stamp=2)	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-28 21:44	PR #6688	2115560273719857152
2026-10-28 13:13	PR #7783	2115431844907323392

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down pod network instances, pod networks, namespa...

full failure pattern:

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down pod network instances, pod networks, namespaces; [descendantResources] remaining resources: applydesires, serviceProviderClusters; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:321]: Unexpected error:
    <*errors.joinError | 0xf5ad11741b0>: 
    resource group "rg-candidate-5-0-lf2cjh6fwp8g": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-5-0-28vlzn" in resourcegroup="rg-candidate-5-0-lf2cjh6fwp8g" to finish deleting: GET https://rp.j6816256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ee641f30-7b7c-4096-8a0a-8e212ff6df0d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ee641f30-7b7c-4096-8a0a-8e212ff6df0d",
      "name": "ee641f30-7b7c-4096-8a0a-8e212ff6df0d",
      "status": "Failed",
      "startTime": "2026-10-28T04:57:47.524111456Z",
      "endTime": "2026-10-28T05:21:51.687390361Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6816256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ee641f30-7b7c-4096-8a0a-8e212ff6df0d
            correlation ID: 57653937-379d-4e91-8cdc-72bb96f14f99
            client request ID: da17b50d-815e-4f7b-b6a1-511dd538a3b6
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:321]: Unexpected error:
    <*errors.joinError | 0xf5ad11741b0>: 
    resource group "rg-candidate-5-0-lf2cjh6fwp8g": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-5-0-28vlzn" in resourcegroup="rg-candidate-5-0-lf2cjh6fwp8g" to finish deleting: GET https://rp.j6816256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ee641f30-7b7c-4096-8a0a-8e212ff6df0d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ee641f30-7b7c-4096-8a0a-8e212ff6df0d",
      "name": "ee641f30-7b7c-4096-8a0a-8e212ff6df0d",
      "status": "Failed",
      "startTime": "2026-10-28T04:57:47.524111456Z",
      "endTime": "2026-10-28T05:21:51.687390361Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6816256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ee641f30-7b7c-4096-8a0a-8e212ff6df0d
            correlation ID: 57653937-379d-4e91-8cdc-72bb96f14f99
            client request ID: da17b50d-815e-4f7b-b6a1-511dd538a3b6
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:321]: Unexpected error:
    <*errors.joinError | 0x2fb9767520c0>: 
    resource group "rg-candidate-4-23-8gsj9lgrgthl": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-23-x7cmb2" in resourcegroup="rg-candidate-4-23-8gsj9lgrgthl" to finish deleting: GET https://rp.j6816256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c5e082c0-738b-4be7-9ea3-e60e834b86c8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c5e082c0-738b-4be7-9ea3-e60e834b86c8",
      "name": "c5e082c0-738b-4be7-9ea3-e60e834b86c8",
      "status": "Failed",
      "startTime": "2026-10-28T04:49:53.842986291Z",
      "endTime": "2026-10-28T05:14:02.592266097Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6816256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c5e082c0-738b-4be7-9ea3-e60e834b86c8
            correlation ID: 4e7a26de-7852-4792-8175-683cd23ec9a8
            client request ID: a8483675-393f-439a-880f-cb8cebaae1a8
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:321]: Unexpected error:
    <*errors.joinError | 0x2fb9767520c0>: 
    resource group "rg-candidate-4-23-8gsj9lgrgthl": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-23-x7cmb2" in resourcegroup="rg-candidate-4-23-8gsj9lgrgthl" to finish deleting: GET https://rp.j6816256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c5e082c0-738b-4be7-9ea3-e60e834b86c8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c5e082c0-738b-4be7-9ea3-e60e834b86c8",
      "name": "c5e082c0-738b-4be7-9ea3-e60e834b86c8",
      "status": "Failed",
      "startTime": "2026-10-28T04:49:53.842986291Z",
      "endTime": "2026-10-28T05:14:02.592266097Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6816256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c5e082c0-738b-4be7-9ea3-e60e834b86c8
            correlation ID: 4e7a26de-7852-4792-8175-683cd23ec9a8
            client request ID: a8483675-393f-439a-880f-cb8cebaae1a8
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:321]: Unexpected error:
    <*errors.joinError | 0x19fdcb2b4318>: 
    resource group "rg-candidate-4-22-b4lvcfvhwx52": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-22-jsq2fr" in resourcegroup="rg-candidate-4-22-b4lvcfvhwx52" to finish deleting: GET https://rp.j6816256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6f392de4-c202-4b15-9624-974c650cb629
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6f392de4-c202-4b15-9624-974c650cb629",
      "name": "6f392de4-c202-4b15-9624-974c650cb629",
      "status": "Failed",
      "startTime": "2026-10-28T04:51:16.927309848Z",
      "endTime": "2026-10-28T05:15:21.671423449Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6816256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6f392de4-c202-4b15-9624-974c650cb629
            correlation ID: 6a7c51a3-41c5-4f0f-a544-ee94e68fe93c
            client request ID: cb15b9e0-1085-4828-b59c-20b42a78c8ff
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:321]: Unexpected error:
    <*errors.joinError | 0x19fdcb2b4318>: 
    resource group "rg-candidate-4-22-b4lvcfvhwx52": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-22-jsq2fr" in resourcegroup="rg-candidate-4-22-b4lvcfvhwx52" to finish deleting: GET https://rp.j6816256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6f392de4-c202-4b15-9624-974c650cb629
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6f392de4-c202-4b15-9624-974c650cb629",
      "name": "6f392de4-c202-4b15-9624-974c650cb629",
      "status": "Failed",
      "startTime": "2026-10-28T04:51:16.927309848Z",
      "endTime": "2026-10-28T05:15:21.671423449Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6816256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6f392de4-c202-4b15-9624-974c650cb629
            correlation ID: 6a7c51a3-41c5-4f0f-a544-ee94e68fe93c
            client request ID: cb15b9e0-1085-4828-b59c-20b42a78c8ff
occurred

Contributing tests (3)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-28 03:31	PR #7662	2115285263486816256

alert [hcp] KubeDeploymentRolloutStuck fired

full failure pattern:

alert [hcp] KubeDeploymentRolloutStuck fired

Signal:

alert

none

Full failure examples (1)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-28T13:22:31Z
  Ended:       2026-10-28T13:28:32Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j6945280-mgmt-1", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.108:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6945280/providers/microsoft.monitor/accounts/hcps-j6945280", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-fct9q", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k/capi-provider is not progressing for longer than 30 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubeDeploymentRolloutStuck does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-28 11:45	PR #7782	2115409675166945280

error code: CANCEL

full failure pattern:

error code: CANCEL

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-th8csc"
Unexpected error:
    <*errors.joinError | 0x28585f5aa1e0>: 
    VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.004s: PVC e2e-csi-disk-5zpk7/e2e-csi-pvc phase is "Pending", expected Bound; pod phase is "Pending"
    [storageclass] name=managed-csi provisioner=disk.csi.azure.com
    [pvc] e2e-csi-disk-5zpk7/e2e-csi-pvc phase=Bound volumeName=pvc-ec65a1b9-74c2-42c0-961c-c9b41647402f storageClass=managed-csi
    [pod] e2e-csi-disk-5zpk7/e2e-csi-pod phase=Pending reason= message=
    [pod] condition type=PodReadyToStartContainers status=False reason= message=
    [pod] condition type=Initialized status=True reason= message=
    [pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]
    [pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]
    [pod] condition type=PodScheduled status=True reason= message=
    [pod] container volume-check ready=false restartCount=0
    [pod]   waiting reason=ContainerCreating message=
    [events] e2e-csi-disk-5zpk7 Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-5zpk7/e2e-csi-pod to t8z9t1e8w4t1q5i-np-th8csc-h6n6d-49dfg (reason=Scheduled, count=0, age=12s)
    [events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim "e2e-csi-disk-5zpk7/e2e-csi-pvc" (reason=Provisioning, count=10, age=36s)
    [events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node t8z9t1e8w4t1q5i-np-th8csc-h6n6d-49dfg (reason=ProvisioningFailed, count=5, age=9m33s)
    [events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=3m48s)
    [events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL (reason=ProvisioningFailed, count=1, age=8m47s)
    [events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=3, age=3m33s)
    [events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-ec65a1b9-74c2-42c0-961c-c9b41647402f (reason=ProvisioningSucceeded, count=1, age=13s)
    {
        errs: [
            <*fmt.wrapError | 0x28585f462380>{
                msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.004s: PVC e2e-csi-disk-5zpk7/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-5zpk7/e2e-csi-pvc phase=Bound volumeName=pvc-ec65a1b9-74c2-42c0-961c-c9b41647402f storageClass=managed-csi\n[pod] e2e-csi-disk-5zpk7/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodReadyToStartContainers status=False reason= message=\n[pod] condition type=Initialized status=True reason= message=\n[pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=PodScheduled status=True reason= message=\n[pod] container volume-check ready=false restartCount=0\n[pod]   waiting reason=ContainerCreating message=\n[events] e2e-csi-disk-5zpk7 Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-5zpk7/e2e-csi-pod to t8z9t1e8w4t1q5i-np-th8csc-h6n6d-49dfg (reason=Scheduled, count=0, age=12s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-5zpk7/e2e-csi-pvc\" (reason=Provisioning, count=10, age=36s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node t8z9t1e8w4t1q5i-np-th8csc-h6n6d-49dfg (reason=ProvisioningFailed, count=5, age=9m33s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=3m48s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL (reason=ProvisioningFailed, count=1, age=8m47s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=3, age=3m33s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-ec65a1b9-74c2-42c0-961c-c9b41647402f (reason=ProvisioningSucceeded, count=1, age=13s)",
                err: <*fmt.wrapError | 0x28585e8a4800>{
                    msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.004s: PVC e2e-csi-disk-5zpk7/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-5zpk7/e2e-csi-pvc phase=Bound volumeName=pvc-ec65a1b9-74c2-42c0-961c-c9b41647402f storageClass=managed-csi\n[pod] e2e-csi-disk-5zpk7/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodReadyToStartContainers status=False reason= message=\n[pod] condition type=Initialized status=True reason= message=\n[pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=PodScheduled status=True reason= message=\n[pod] container volume-check ready=false restartCount=0\n[pod]   waiting reason=ContainerCreating message=\n[events] e2e-csi-disk-5zpk7 Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-5zpk7/e2e-csi-pod to t8z9t1e8w4t1q5i-np-th8csc-h6n6d-49dfg (reason=Scheduled, count=0, age=12s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-5zpk7/e2e-csi-pvc\" (reason=Provisioning, count=10, age=36s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node t8z9t1e8w4t1q5i-np-th8csc-h6n6d-49dfg (reason=ProvisioningFailed, count=5, age=9m33s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=3m48s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL (reason=ProvisioningFailed, count=1, age=8m47s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=3, age=3m33s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-ec65a1b9-74c2-42c0-961c-c9b41647402f (reason=ProvisioningSucceeded, count=1, age=13s)",
                    err: <*verifiers.pollObservationError | 0x28585e8a43c0>{
                        observation: <*errors.errorString | 0x28585ee0a670>{
                            s: "PVC e2e-csi-disk-5zpk7/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"",
                        },
                        interruption: <*fmt.wrapError | 0x28585e8a4320>{
                            msg: "failed to get PVC e2e-csi-disk-5zpk7/e2e-csi-pvc: context deadline exceeded",
                            err: <context.deadlineExceededError>{},
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-th8csc"
Unexpected error:
    <*errors.joinError | 0x28585f5aa1e0>: 
    VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.004s: PVC e2e-csi-disk-5zpk7/e2e-csi-pvc phase is "Pending", expected Bound; pod phase is "Pending"
    [storageclass] name=managed-csi provisioner=disk.csi.azure.com
    [pvc] e2e-csi-disk-5zpk7/e2e-csi-pvc phase=Bound volumeName=pvc-ec65a1b9-74c2-42c0-961c-c9b41647402f storageClass=managed-csi
    [pod] e2e-csi-disk-5zpk7/e2e-csi-pod phase=Pending reason= message=
    [pod] condition type=PodReadyToStartContainers status=False reason= message=
    [pod] condition type=Initialized status=True reason= message=
    [pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]
    [pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]
    [pod] condition type=PodScheduled status=True reason= message=
    [pod] container volume-check ready=false restartCount=0
    [pod]   waiting reason=ContainerCreating message=
    [events] e2e-csi-disk-5zpk7 Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-5zpk7/e2e-csi-pod to t8z9t1e8w4t1q5i-np-th8csc-h6n6d-49dfg (reason=Scheduled, count=0, age=12s)
    [events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim "e2e-csi-disk-5zpk7/e2e-csi-pvc" (reason=Provisioning, count=10, age=36s)
    [events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node t8z9t1e8w4t1q5i-np-th8csc-h6n6d-49dfg (reason=ProvisioningFailed, count=5, age=9m33s)
    [events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=3m48s)
    [events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL (reason=ProvisioningFailed, count=1, age=8m47s)
    [events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=3, age=3m33s)
    [events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-ec65a1b9-74c2-42c0-961c-c9b41647402f (reason=ProvisioningSucceeded, count=1, age=13s)
    {
        errs: [
            <*fmt.wrapError | 0x28585f462380>{
                msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.004s: PVC e2e-csi-disk-5zpk7/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-5zpk7/e2e-csi-pvc phase=Bound volumeName=pvc-ec65a1b9-74c2-42c0-961c-c9b41647402f storageClass=managed-csi\n[pod] e2e-csi-disk-5zpk7/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodReadyToStartContainers status=False reason= message=\n[pod] condition type=Initialized status=True reason= message=\n[pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=PodScheduled status=True reason= message=\n[pod] container volume-check ready=false restartCount=0\n[pod]   waiting reason=ContainerCreating message=\n[events] e2e-csi-disk-5zpk7 Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-5zpk7/e2e-csi-pod to t8z9t1e8w4t1q5i-np-th8csc-h6n6d-49dfg (reason=Scheduled, count=0, age=12s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-5zpk7/e2e-csi-pvc\" (reason=Provisioning, count=10, age=36s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node t8z9t1e8w4t1q5i-np-th8csc-h6n6d-49dfg (reason=ProvisioningFailed, count=5, age=9m33s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=3m48s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL (reason=ProvisioningFailed, count=1, age=8m47s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=3, age=3m33s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-ec65a1b9-74c2-42c0-961c-c9b41647402f (reason=ProvisioningSucceeded, count=1, age=13s)",
                err: <*fmt.wrapError | 0x28585e8a4800>{
                    msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.004s: PVC e2e-csi-disk-5zpk7/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-5zpk7/e2e-csi-pvc phase=Bound volumeName=pvc-ec65a1b9-74c2-42c0-961c-c9b41647402f storageClass=managed-csi\n[pod] e2e-csi-disk-5zpk7/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodReadyToStartContainers status=False reason= message=\n[pod] condition type=Initialized status=True reason= message=\n[pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=PodScheduled status=True reason= message=\n[pod] container volume-check ready=false restartCount=0\n[pod]   waiting reason=ContainerCreating message=\n[events] e2e-csi-disk-5zpk7 Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-5zpk7/e2e-csi-pod to t8z9t1e8w4t1q5i-np-th8csc-h6n6d-49dfg (reason=Scheduled, count=0, age=12s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-5zpk7/e2e-csi-pvc\" (reason=Provisioning, count=10, age=36s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node t8z9t1e8w4t1q5i-np-th8csc-h6n6d-49dfg (reason=ProvisioningFailed, count=5, age=9m33s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=3m48s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL (reason=ProvisioningFailed, count=1, age=8m47s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=3, age=3m33s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-ec65a1b9-74c2-42c0-961c-c9b41647402f (reason=ProvisioningSucceeded, count=1, age=13s)",
                    err: <*verifiers.pollObservationError | 0x28585e8a43c0>{
                        observation: <*errors.errorString | 0x28585ee0a670>{
                            s: "PVC e2e-csi-disk-5zpk7/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"",
                        },
                        interruption: <*fmt.wrapError | 0x28585e8a4320>{
                            msg: "failed to get PVC e2e-csi-disk-5zpk7/e2e-csi-pvc: context deadline exceeded",
                            err: <context.deadlineExceededError>{},
                        },
                    },
                },
            },
        ],
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-28 11:46	PR #7802	2115409868675354624

ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<u...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try ag... message truncated; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x3f06fbed83a0>: 
    failed to create NodePool swift-workers: failed waiting for nodepool="swift-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-4lmptk5vpx7s" to finish creating: GET https://rp.j7626240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f4b98636-6475-41ec-a151-007f2ef479ea
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f4b98636-6475-41ec-a151-007f2ef479ea",
      "name": "f4b98636-6475-41ec-a151-007f2ef479ea",
      "status": "Failed",
      "startTime": "2026-10-28T19:44:01.234316083Z",
      "endTime": "2026-10-28T20:03:07.974415443Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l3j9w2r9y4j1z4u-swift-workers-4vqnr-xznbb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-4lmptk5vpx7s--managed/providers/Microsoft.Compute/virtualMachines/l3j9w2r9y4j1z4u-swift-workers-4vqnr-xznbb to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l3j9w2r9y4j1z4u-swift-workers-4vqnr-xznbb: Failed: availabilitysets failed to create or update. err: failed to get existing resource kusto-logs-4lmptk5vpx7s--managed/2tnvvns3jn8rgfrctgfo79mkpouiqrqc_l3j9w2r9y4j1z4u-swift-workers-as (service: availabilitysets): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-4lmptk5vpx7s--managed/providers/Microsoft.Compute/availabilitySets/2tnvvns3jn8rgfrctgfo79mkpouiqrqc_l3j9w2r9y4j1z4u-swift-workers-as\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '1868e77e-aae0-4284-940e-e4cd46ae5baa' exceeded. Please try ag... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7626240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f4b98636-6475-41ec-a151-007f2ef479ea
            correlation ID: 1f8c9a64-07bf-44f3-bca5-a4308cdb0f56
            client request ID: 49aeda1c-19cc-4647-af7d-1d78f910596d
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x3f06fbed83a0>: 
    failed to create NodePool swift-workers: failed waiting for nodepool="swift-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-4lmptk5vpx7s" to finish creating: GET https://rp.j7626240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f4b98636-6475-41ec-a151-007f2ef479ea
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f4b98636-6475-41ec-a151-007f2ef479ea",
      "name": "f4b98636-6475-41ec-a151-007f2ef479ea",
      "status": "Failed",
      "startTime": "2026-10-28T19:44:01.234316083Z",
      "endTime": "2026-10-28T20:03:07.974415443Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l3j9w2r9y4j1z4u-swift-workers-4vqnr-xznbb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-4lmptk5vpx7s--managed/providers/Microsoft.Compute/virtualMachines/l3j9w2r9y4j1z4u-swift-workers-4vqnr-xznbb to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l3j9w2r9y4j1z4u-swift-workers-4vqnr-xznbb: Failed: availabilitysets failed to create or update. err: failed to get existing resource kusto-logs-4lmptk5vpx7s--managed/2tnvvns3jn8rgfrctgfo79mkpouiqrqc_l3j9w2r9y4j1z4u-swift-workers-as (service: availabilitysets): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-4lmptk5vpx7s--managed/providers/Microsoft.Compute/availabilitySets/2tnvvns3jn8rgfrctgfo79mkpouiqrqc_l3j9w2r9y4j1z4u-swift-workers-as\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '1868e77e-aae0-4284-940e-e4cd46ae5baa' exceeded. Please try ag... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7626240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f4b98636-6475-41ec-a151-007f2ef479ea
            correlation ID: 1f8c9a64-07bf-44f3-bca5-a4308cdb0f56
            client request ID: 49aeda1c-19cc-4647-af7d-1d78f910596d
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-28 18:33	PR #7816	2115512185487626240

ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<u...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional t... message truncated; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:96]: failed to create node pool "nodepool-128GiB" with 128GiB osDisk
Unexpected error:
    <*fmt.wrapError | 0x4218f58d8c0>: 
    failed to create NodePool nodepool-128GiB: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-8q6szjwkq6rk" to finish creating: GET https://rp.j0183936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/78051d0a-b2ad-4647-99d4-120efa3f6a99
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/78051d0a-b2ad-4647-99d4-120efa3f6a99",
      "name": "78051d0a-b2ad-4647-99d4-120efa3f6a99",
      "status": "Failed",
      "startTime": "2026-10-28T21:13:11.691142567Z",
      "endTime": "2026-10-28T21:32:17.541374372Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine a7p5v6a9s2f0e3j-nodepool-128gib-brx87-m46c2: Failed: interfaces failed to create or update. err: failed to get existing resource clusternp128-8q6szjwkq6rk--managed/a7p5v6a9s2f0e3j-nodepool-128gib-brx87-m46c2-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-8q6szjwkq6rk--managed/providers/Microsoft.Network/networkInterfaces/a7p5v6a9s2f0e3j-nodepool-128gib-brx87-m46c2-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '3996e871-9d8c-4863-9b56-cf4685281cdc' exceeded. Please try again after '10' seconds after additional t... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j0183936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/78051d0a-b2ad-4647-99d4-120efa3f6a99
            correlation ID: 7ef5aa4f-453a-4950-9b06-46a903ca993d
            client request ID: a9dcefcf-ba4d-4111-9c80-6dbba9abd781
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:96]: failed to create node pool "nodepool-128GiB" with 128GiB osDisk
Unexpected error:
    <*fmt.wrapError | 0x4218f58d8c0>: 
    failed to create NodePool nodepool-128GiB: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-8q6szjwkq6rk" to finish creating: GET https://rp.j0183936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/78051d0a-b2ad-4647-99d4-120efa3f6a99
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/78051d0a-b2ad-4647-99d4-120efa3f6a99",
      "name": "78051d0a-b2ad-4647-99d4-120efa3f6a99",
      "status": "Failed",
      "startTime": "2026-10-28T21:13:11.691142567Z",
      "endTime": "2026-10-28T21:32:17.541374372Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine a7p5v6a9s2f0e3j-nodepool-128gib-brx87-m46c2: Failed: interfaces failed to create or update. err: failed to get existing resource clusternp128-8q6szjwkq6rk--managed/a7p5v6a9s2f0e3j-nodepool-128gib-brx87-m46c2-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-8q6szjwkq6rk--managed/providers/Microsoft.Network/networkInterfaces/a7p5v6a9s2f0e3j-nodepool-128gib-brx87-m46c2-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '3996e871-9d8c-4863-9b56-cf4685281cdc' exceeded. Please try again after '10' seconds after additional t... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j0183936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/78051d0a-b2ad-4647-99d4-120efa3f6a99
            correlation ID: 7ef5aa4f-453a-4950-9b06-46a903ca993d
            client request ID: a9dcefcf-ba4d-4111-9c80-6dbba9abd781
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and custom node pool osDisk size	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-28 20:11	PR #7809	2115536999510183936

ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<u...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available. Refer ... message truncated; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:178]: failed to create node pool np-1 for pull secret cluster
Unexpected error:
    <*fmt.wrapError | 0x315531e198c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-6tc6qf9c7brh" to finish creating: GET https://rp.j6407296.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4ebc64ca-e27a-4569-a3bf-84d10df15019
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4ebc64ca-e27a-4569-a3bf-84d10df15019",
      "name": "4ebc64ca-e27a-4569-a3bf-84d10df15019",
      "status": "Failed",
      "startTime": "2026-10-28T11:51:14.858832489Z",
      "endTime": "2026-10-28T12:10:21.145603579Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine u9f3h2h7j3o5r2t-np-1-frcss-9d6tp: Failed: interfaces failed to create or update. err: failed to get existing resource pullsecret-test-6tc6qf9c7brh--managed/u9f3h2h7j3o5r2t-np-1-frcss-9d6tp-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-6tc6qf9c7brh--managed/providers/Microsoft.Network/networkInterfaces/u9f3h2h7j3o5r2t-np-1-frcss-9d6tp-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'f8be0ede-39df-41cf-aed7-7f3626f23a5a' exceeded. Please try again after '10' seconds after additional tokens are available. Refer ... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6407296.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4ebc64ca-e27a-4569-a3bf-84d10df15019
            correlation ID: 80c82865-2a09-4afd-85b7-df3733e6ccd3
            client request ID: 14042924-6e78-44be-a159-f0ae94071a00
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:178]: failed to create node pool np-1 for pull secret cluster
Unexpected error:
    <*fmt.wrapError | 0x315531e198c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-6tc6qf9c7brh" to finish creating: GET https://rp.j6407296.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4ebc64ca-e27a-4569-a3bf-84d10df15019
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4ebc64ca-e27a-4569-a3bf-84d10df15019",
      "name": "4ebc64ca-e27a-4569-a3bf-84d10df15019",
      "status": "Failed",
      "startTime": "2026-10-28T11:51:14.858832489Z",
      "endTime": "2026-10-28T12:10:21.145603579Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine u9f3h2h7j3o5r2t-np-1-frcss-9d6tp: Failed: interfaces failed to create or update. err: failed to get existing resource pullsecret-test-6tc6qf9c7brh--managed/u9f3h2h7j3o5r2t-np-1-frcss-9d6tp-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-6tc6qf9c7brh--managed/providers/Microsoft.Network/networkInterfaces/u9f3h2h7j3o5r2t-np-1-frcss-9d6tp-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'f8be0ede-39df-41cf-aed7-7f3626f23a5a' exceeded. Please try again after '10' seconds after additional tokens are available. Refer ... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6407296.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4ebc64ca-e27a-4569-a3bf-84d10df15019
            correlation ID: 80c82865-2a09-4afd-85b7-df3733e6ccd3
            client request ID: 14042924-6e78-44be-a159-f0ae94071a00
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-28 10:45	PR #7783	2115394496836407296

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: NotReady: AzureMachine status.initialization.provisioned is false; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
    <*fmt.wrapError | 0x5aaaa2c7760>: 
    failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-sbb47qkh4z4z" to finish creating: GET https://rp.j3205248.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0944ef7a-112a-4da6-8fcb-95326da6c09a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0944ef7a-112a-4da6-8fcb-95326da6c09a",
      "name": "0944ef7a-112a-4da6-8fcb-95326da6c09a",
      "status": "Failed",
      "startTime": "2026-10-28T23:56:10.615228789Z",
      "endTime": "2026-10-29T00:15:11.126153584Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine r0r9z1j6z3s7b4w-cordon-np-n2bgq-dtwls: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine r0r9z1j6z3s7b4w-cordon-np-n2bgq-dtwls: NotReady: AzureMachine status.initialization.provisioned is false\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3205248.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0944ef7a-112a-4da6-8fcb-95326da6c09a
            correlation ID: 57d4995c-1d2b-4698-92e2-470cb921ad86
            client request ID: 07a0e266-a0ed-4565-bed7-5b18b3a11045
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
    <*fmt.wrapError | 0x5aaaa2c7760>: 
    failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-sbb47qkh4z4z" to finish creating: GET https://rp.j3205248.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0944ef7a-112a-4da6-8fcb-95326da6c09a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0944ef7a-112a-4da6-8fcb-95326da6c09a",
      "name": "0944ef7a-112a-4da6-8fcb-95326da6c09a",
      "status": "Failed",
      "startTime": "2026-10-28T23:56:10.615228789Z",
      "endTime": "2026-10-29T00:15:11.126153584Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine r0r9z1j6z3s7b4w-cordon-np-n2bgq-dtwls: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine r0r9z1j6z3s7b4w-cordon-np-n2bgq-dtwls: NotReady: AzureMachine status.initialization.provisioned is false\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3205248.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0944ef7a-112a-4da6-8fcb-95326da6c09a
            correlation ID: 57d4995c-1d2b-4698-92e2-470cb921ad86
            client request ID: 07a0e266-a0ed-4565-bed7-5b18b3a11045
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to cordon, drain, and uncordon a node in an HCP cluster	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-28 22:33	PR #7823	2115572620593205248

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down namespaces; [descendantResources] remaining ...

full failure pattern:

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down namespaces; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:321]: Unexpected error:
    <*errors.joinError | 0x19a135b620a8>: 
    resource group "feature-aggregation-c29rndpv7j5c": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-c29rndpv7j5c" to finish deleting: GET https://rp.j6816256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/541f32be-fa69-4e38-80a2-84651b9746cc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/541f32be-fa69-4e38-80a2-84651b9746cc",
      "name": "541f32be-fa69-4e38-80a2-84651b9746cc",
      "status": "Failed",
      "startTime": "2026-10-28T04:58:24.120121666Z",
      "endTime": "2026-10-28T05:22:31.687389775Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6816256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/541f32be-fa69-4e38-80a2-84651b9746cc
            correlation ID: efcef01e-78c0-4c18-a8a7-6bed6c2a5d41
            client request ID: 23dd3444-0c3c-438b-a21b-7fd27b67a689
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:321]: Unexpected error:
    <*errors.joinError | 0x19a135b620a8>: 
    resource group "feature-aggregation-c29rndpv7j5c": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-c29rndpv7j5c" to finish deleting: GET https://rp.j6816256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/541f32be-fa69-4e38-80a2-84651b9746cc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/541f32be-fa69-4e38-80a2-84651b9746cc",
      "name": "541f32be-fa69-4e38-80a2-84651b9746cc",
      "status": "Failed",
      "startTime": "2026-10-28T04:58:24.120121666Z",
      "endTime": "2026-10-28T05:22:31.687389775Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6816256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/541f32be-fa69-4e38-80a2-84651b9746cc
            correlation ID: efcef01e-78c0-4c18-a8a7-6bed6c2a5d41
            client request ID: 23dd3444-0c3c-438b-a21b-7fd27b67a689
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-28 03:31	PR #7662	2115285263486816256

ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-lpggdd' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...

full failure pattern:

ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-lpggdd' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'.; provider Microsoft.Network

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-21-tdqwb6"
Unexpected error:
    <*errors.joinError | 0x122a895e0150>: 
    VerifyIngressDNSRecord(aro.r1n2g0k6g2s3t3a.soq5.j8963712.hcp.osadev.cloud) failed: VerifyIngressDNSRecord(aro.r1n2g0k6g2s3t3a.soq5.j8963712.hcp.osadev.cloud) timed out after 10m0.001s: DNS A record "*.e2e-dns-lpggdd" not yet present in zone rg-candidate-4-21-qhznhv86fzbg-candidate-tdqwb6--managed/aro.r1n2g0k6g2s3t3a.soq5.j8963712.hcp.osadev.cloud: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qhznhv86fzbg-candidate-tdqwb6--managed/providers/Microsoft.Network/dnsZones/aro.r1n2g0k6g2s3t3a.soq5.j8963712.hcp.osadev.cloud/A/*.e2e-dns-lpggdd
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: NotFound
    --------------------------------------------------------------------------------
    {
      "code": "NotFound",
      "message": "The resource record '*.e2e-dns-lpggdd' does not exist in resource group 'rg-candidate-4-21-qhznhv86fzbg-candidate-tdqwb6--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qhznhv86fzbg-candidate-tdqwb6--managed/providers/Microsoft.Network/dnsZones/aro.r1n2g0k6g2s3t3a.soq5.j8963712.hcp.osadev.cloud/A/*.e2e-dns-lpggdd
            correlation ID: fc55c026-0ddc-4f29-bb3d-f393fa0f3cd6
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-21-tdqwb6"
Unexpected error:
    <*errors.joinError | 0x122a895e0150>: 
    VerifyIngressDNSRecord(aro.r1n2g0k6g2s3t3a.soq5.j8963712.hcp.osadev.cloud) failed: VerifyIngressDNSRecord(aro.r1n2g0k6g2s3t3a.soq5.j8963712.hcp.osadev.cloud) timed out after 10m0.001s: DNS A record "*.e2e-dns-lpggdd" not yet present in zone rg-candidate-4-21-qhznhv86fzbg-candidate-tdqwb6--managed/aro.r1n2g0k6g2s3t3a.soq5.j8963712.hcp.osadev.cloud: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qhznhv86fzbg-candidate-tdqwb6--managed/providers/Microsoft.Network/dnsZones/aro.r1n2g0k6g2s3t3a.soq5.j8963712.hcp.osadev.cloud/A/*.e2e-dns-lpggdd
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: NotFound
    --------------------------------------------------------------------------------
    {
      "code": "NotFound",
      "message": "The resource record '*.e2e-dns-lpggdd' does not exist in resource group 'rg-candidate-4-21-qhznhv86fzbg-candidate-tdqwb6--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qhznhv86fzbg-candidate-tdqwb6--managed/providers/Microsoft.Network/dnsZones/aro.r1n2g0k6g2s3t3a.soq5.j8963712.hcp.osadev.cloud/A/*.e2e-dns-lpggdd
            correlation ID: fc55c026-0ddc-4f29-bb3d-f393fa0f3cd6
            client request ID: unavailable
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-28 07:47	PR #7786	2115349685118963712

ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-p2v629' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...

full failure pattern:

ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-p2v629' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'.; provider Microsoft.Network

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-r5cltn"
Unexpected error:
    <*errors.joinError | 0x263db2a8cd50>: 
    VerifyIngressDNSRecord(aro.u8t8x8m6x2i2q9i.jz74.j3603072.hcp.osadev.cloud) failed: VerifyIngressDNSRecord(aro.u8t8x8m6x2i2q9i.jz74.j3603072.hcp.osadev.cloud) timed out after 10m0.003s: DNS A record "*.e2e-dns-p2v629" not yet present in zone rg-candidate-4-23-6vt99sh5dqhr-candidate-r5cltn--managed/aro.u8t8x8m6x2i2q9i.jz74.j3603072.hcp.osadev.cloud: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-6vt99sh5dqhr-candidate-r5cltn--managed/providers/Microsoft.Network/dnsZones/aro.u8t8x8m6x2i2q9i.jz74.j3603072.hcp.osadev.cloud/A/*.e2e-dns-p2v629
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: NotFound
    --------------------------------------------------------------------------------
    {
      "code": "NotFound",
      "message": "The resource record '*.e2e-dns-p2v629' does not exist in resource group 'rg-candidate-4-23-6vt99sh5dqhr-candidate-r5cltn--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-6vt99sh5dqhr-candidate-r5cltn--managed/providers/Microsoft.Network/dnsZones/aro.u8t8x8m6x2i2q9i.jz74.j3603072.hcp.osadev.cloud/A/*.e2e-dns-p2v629
            correlation ID: 0eb3f5e1-8fdc-4f54-ad01-280d07fdef7f
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-r5cltn"
Unexpected error:
    <*errors.joinError | 0x263db2a8cd50>: 
    VerifyIngressDNSRecord(aro.u8t8x8m6x2i2q9i.jz74.j3603072.hcp.osadev.cloud) failed: VerifyIngressDNSRecord(aro.u8t8x8m6x2i2q9i.jz74.j3603072.hcp.osadev.cloud) timed out after 10m0.003s: DNS A record "*.e2e-dns-p2v629" not yet present in zone rg-candidate-4-23-6vt99sh5dqhr-candidate-r5cltn--managed/aro.u8t8x8m6x2i2q9i.jz74.j3603072.hcp.osadev.cloud: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-6vt99sh5dqhr-candidate-r5cltn--managed/providers/Microsoft.Network/dnsZones/aro.u8t8x8m6x2i2q9i.jz74.j3603072.hcp.osadev.cloud/A/*.e2e-dns-p2v629
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: NotFound
    --------------------------------------------------------------------------------
    {
      "code": "NotFound",
      "message": "The resource record '*.e2e-dns-p2v629' does not exist in resource group 'rg-candidate-4-23-6vt99sh5dqhr-candidate-r5cltn--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-6vt99sh5dqhr-candidate-r5cltn--managed/providers/Microsoft.Network/dnsZones/aro.u8t8x8m6x2i2q9i.jz74.j3603072.hcp.osadev.cloud/A/*.e2e-dns-p2v629
            correlation ID: 0eb3f5e1-8fdc-4f54-ad01-280d07fdef7f
            client request ID: unavailable
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-28 17:45	PR #6688	2115500092713603072

ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-swrng7' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...

full failure pattern:

ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-swrng7' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'.; provider Microsoft.Network

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-v9km2q"
Unexpected error:
    <*errors.joinError | 0x1f131b624198>: 
    VerifyIngressDNSRecord(aro.r6b2z1c7z7d0i9t.kvy2.j8974848.hcp.osadev.cloud) failed: VerifyIngressDNSRecord(aro.r6b2z1c7z7d0i9t.kvy2.j8974848.hcp.osadev.cloud) timed out after 10m0.003s: DNS A record "*.e2e-dns-swrng7" not yet present in zone rg-candidate-4-23-2f2k8zzmfdnf-candidate-v9km2q--managed/aro.r6b2z1c7z7d0i9t.kvy2.j8974848.hcp.osadev.cloud: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-2f2k8zzmfdnf-candidate-v9km2q--managed/providers/Microsoft.Network/dnsZones/aro.r6b2z1c7z7d0i9t.kvy2.j8974848.hcp.osadev.cloud/A/*.e2e-dns-swrng7
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: NotFound
    --------------------------------------------------------------------------------
    {
      "code": "NotFound",
      "message": "The resource record '*.e2e-dns-swrng7' does not exist in resource group 'rg-candidate-4-23-2f2k8zzmfdnf-candidate-v9km2q--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-2f2k8zzmfdnf-candidate-v9km2q--managed/providers/Microsoft.Network/dnsZones/aro.r6b2z1c7z7d0i9t.kvy2.j8974848.hcp.osadev.cloud/A/*.e2e-dns-swrng7
            correlation ID: f5ba2020-eed4-41d4-be4e-2d7ff36da168
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-v9km2q"
Unexpected error:
    <*errors.joinError | 0x1f131b624198>: 
    VerifyIngressDNSRecord(aro.r6b2z1c7z7d0i9t.kvy2.j8974848.hcp.osadev.cloud) failed: VerifyIngressDNSRecord(aro.r6b2z1c7z7d0i9t.kvy2.j8974848.hcp.osadev.cloud) timed out after 10m0.003s: DNS A record "*.e2e-dns-swrng7" not yet present in zone rg-candidate-4-23-2f2k8zzmfdnf-candidate-v9km2q--managed/aro.r6b2z1c7z7d0i9t.kvy2.j8974848.hcp.osadev.cloud: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-2f2k8zzmfdnf-candidate-v9km2q--managed/providers/Microsoft.Network/dnsZones/aro.r6b2z1c7z7d0i9t.kvy2.j8974848.hcp.osadev.cloud/A/*.e2e-dns-swrng7
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: NotFound
    --------------------------------------------------------------------------------
    {
      "code": "NotFound",
      "message": "The resource record '*.e2e-dns-swrng7' does not exist in resource group 'rg-candidate-4-23-2f2k8zzmfdnf-candidate-v9km2q--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-2f2k8zzmfdnf-candidate-v9km2q--managed/providers/Microsoft.Network/dnsZones/aro.r6b2z1c7z7d0i9t.kvy2.j8974848.hcp.osadev.cloud/A/*.e2e-dns-swrng7
            correlation ID: f5ba2020-eed4-41d4-be4e-2d7ff36da168
            client request ID: unavailable
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-28 11:14	PR #7786	2115401780928974848

VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...

full failure pattern:

VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j0183936-mgmt-1/ocm-arohcpci01-<id>: expected

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hcf7x894jgzg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x2424493382e0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hcf7x894jgzg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j0183936-mgmt-1/ocm-arohcpci01-2to1b0ie1a4878pqeni53tn93ag3ur6i-t8h4i5b2e7a9g5f: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hcf7x894jgzg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j0183936-mgmt-1/ocm-arohcpci01-2to1b0ie1a4878pqeni53tn93ag3ur6i-t8h4i5b2e7a9g5f: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        err: <*errors.errorString | 0x2424493345b0>{
            s: "dev/centralus/ci01-j0183936-mgmt-1/ocm-arohcpci01-2to1b0ie1a4878pqeni53tn93ag3ur6i-t8h4i5b2e7a9g5f: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hcf7x894jgzg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x2424493382e0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hcf7x894jgzg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j0183936-mgmt-1/ocm-arohcpci01-2to1b0ie1a4878pqeni53tn93ag3ur6i-t8h4i5b2e7a9g5f: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hcf7x894jgzg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j0183936-mgmt-1/ocm-arohcpci01-2to1b0ie1a4878pqeni53tn93ag3ur6i-t8h4i5b2e7a9g5f: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        err: <*errors.errorString | 0x2424493345b0>{
            s: "dev/centralus/ci01-j0183936-mgmt-1/ocm-arohcpci01-2to1b0ie1a4878pqeni53tn93ag3ur6i-t8h4i5b2e7a9g5f: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-28 20:11	PR #7809	2115536999510183936

VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...

full failure pattern:

VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: no customer resource ID -> live ReadDesire HostedCluster ->

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-w96mrp6lpnz7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x67403d76ae0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-w96mrp6lpnz7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-w96mrp6lpnz7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
        err: <*verifiers.pollObservationError | 0x67403d76ac0>{
            observation: <*errors.errorString | 0x674032428b0>{
                s: "no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
            },
            interruption: <*fmt.wrapError | 0x67403d76aa0>{
                msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j5001984-mgmt-2\"; let ns = \"ocm-arohcpci01-2tnvsselnr4pgg1e0n2skr2pggevm4b6-w8s3q8g2g8v3s0d\"; let start = datetime(2026-10-28T19:24:48.1549118Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                err: <*errors.Error | 0x674036d06e0>{
                    Op: 1,
                    Kind: 7,
                    Err: <*fmt.wrapError | 0x67403d76a80>{
                        msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j5001984-mgmt-2\"; let ns = \"ocm-arohcpci01-2tnvsselnr4pgg1e0n2skr2pggevm4b6-w8s3q8g2g8v3s0d\"; let start = datetime(2026-10-28T19:24:48.1549118Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                        err: <*url.Error | 0x67404ab4ba0>{
                            Op: "Post",
                            URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                            Err: <context.deadlineExceededError>{},
                        },
                    },
                    restErrMsg: nil,
                    decoded: nil,
                    permanent: false,
                    inner: nil,
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-w96mrp6lpnz7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x67403d76ae0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-w96mrp6lpnz7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-w96mrp6lpnz7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
        err: <*verifiers.pollObservationError | 0x67403d76ac0>{
            observation: <*errors.errorString | 0x674032428b0>{
                s: "no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
            },
            interruption: <*fmt.wrapError | 0x67403d76aa0>{
                msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j5001984-mgmt-2\"; let ns = \"ocm-arohcpci01-2tnvsselnr4pgg1e0n2skr2pggevm4b6-w8s3q8g2g8v3s0d\"; let start = datetime(2026-10-28T19:24:48.1549118Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                err: <*errors.Error | 0x674036d06e0>{
                    Op: 1,
                    Kind: 7,
                    Err: <*fmt.wrapError | 0x67403d76a80>{
                        msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j5001984-mgmt-2\"; let ns = \"ocm-arohcpci01-2tnvsselnr4pgg1e0n2skr2pggevm4b6-w8s3q8g2g8v3s0d\"; let start = datetime(2026-10-28T19:24:48.1549118Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                        err: <*url.Error | 0x67404ab4ba0>{
                            Op: "Post",
                            URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                            Err: <context.deadlineExceededError>{},
                        },
                    },
                    restErrMsg: nil,
                    decoded: nil,
                    permanent: false,
                    inner: nil,
                },
            },
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-28 18:22	PR #7814	2115509651545001984

VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...

full failure pattern:

VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.005s: no customer resource ID -> live ReadDesire HostedCluster ->

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-7qgq6lcld5tj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x31615201e9a0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-7qgq6lcld5tj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.005s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-7qgq6lcld5tj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.005s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
        err: <*verifiers.pollObservationError | 0x31615201e980>{
            observation: <*errors.errorString | 0x316153080da0>{
                s: "no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
            },
            interruption: <*fmt.wrapError | 0x31615201e960>{
                msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j9822336-mgmt-1\"; let ns = \"ocm-arohcpci01-2tnvqtddfc0hnmn52ol3g2lfbm6i22ri-i7y4b4h2z7o4a5h\"; let start = datetime(2026-10-28T19:21:43.9974261Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                err: <*errors.Error | 0x316150f60aa0>{
                    Op: 1,
                    Kind: 7,
                    Err: <*fmt.wrapError | 0x31615201e920>{
                        msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j9822336-mgmt-1\"; let ns = \"ocm-arohcpci01-2tnvqtddfc0hnmn52ol3g2lfbm6i22ri-i7y4b4h2z7o4a5h\"; let start = datetime(2026-10-28T19:21:43.9974261Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                        err: <*url.Error | 0x316152c10a80>{
                            Op: "Post",
                            URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                            Err: <context.deadlineExceededError>{},
                        },
                    },
                    restErrMsg: nil,
                    decoded: nil,
                    permanent: false,
                    inner: nil,
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-7qgq6lcld5tj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x31615201e9a0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-7qgq6lcld5tj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.005s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-7qgq6lcld5tj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.005s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
        err: <*verifiers.pollObservationError | 0x31615201e980>{
            observation: <*errors.errorString | 0x316153080da0>{
                s: "no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
            },
            interruption: <*fmt.wrapError | 0x31615201e960>{
                msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j9822336-mgmt-1\"; let ns = \"ocm-arohcpci01-2tnvqtddfc0hnmn52ol3g2lfbm6i22ri-i7y4b4h2z7o4a5h\"; let start = datetime(2026-10-28T19:21:43.9974261Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                err: <*errors.Error | 0x316150f60aa0>{
                    Op: 1,
                    Kind: 7,
                    Err: <*fmt.wrapError | 0x31615201e920>{
                        msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j9822336-mgmt-1\"; let ns = \"ocm-arohcpci01-2tnvqtddfc0hnmn52ol3g2lfbm6i22ri-i7y4b4h2z7o4a5h\"; let start = datetime(2026-10-28T19:21:43.9974261Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                        err: <*url.Error | 0x316152c10a80>{
                            Op: "Post",
                            URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                            Err: <context.deadlineExceededError>{},
                        },
                    },
                    restErrMsg: nil,
                    decoded: nil,
                    permanent: false,
                    inner: nil,
                },
            },
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-28 18:12	PR #7704	2115506901079822336

VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...

full failure pattern:

VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.006s: no customer resource ID -> live ReadDesire HostedCluster ->

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-8jc7r25jfq6n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xd26a6364d80>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-8jc7r25jfq6n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.006s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-8jc7r25jfq6n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.006s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
        err: <*verifiers.pollObservationError | 0xd26a6364d60>{
            observation: <*errors.errorString | 0xd26a57e61b0>{
                s: "no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
            },
            interruption: <*fmt.wrapError | 0xd26a6364d40>{
                msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j6172544-mgmt-2\"; let ns = \"ocm-arohcpci01-2tnpie08b6e3m8ktaflq60p9p59bkmrn-q5i5v4f4b9e4k5j\"; let start = datetime(2026-10-28T12:14:00.5191321Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                err: <*errors.Error | 0xd26a5820dc0>{
                    Op: 1,
                    Kind: 7,
                    Err: <*fmt.wrapError | 0xd26a6364d20>{
                        msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j6172544-mgmt-2\"; let ns = \"ocm-arohcpci01-2tnpie08b6e3m8ktaflq60p9p59bkmrn-q5i5v4f4b9e4k5j\"; let start = datetime(2026-10-28T12:14:00.5191321Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                        err: <*url.Error | 0xd26a719e570>{
                            Op: "Post",
                            URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                            Err: <context.deadlineExceededError>{},
                        },
                    },
                    restErrMsg: nil,
                    decoded: nil,
                    permanent: false,
                    inner: nil,
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-8jc7r25jfq6n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xd26a6364d80>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-8jc7r25jfq6n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.006s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-8jc7r25jfq6n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.006s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
        err: <*verifiers.pollObservationError | 0xd26a6364d60>{
            observation: <*errors.errorString | 0xd26a57e61b0>{
                s: "no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
            },
            interruption: <*fmt.wrapError | 0xd26a6364d40>{
                msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j6172544-mgmt-2\"; let ns = \"ocm-arohcpci01-2tnpie08b6e3m8ktaflq60p9p59bkmrn-q5i5v4f4b9e4k5j\"; let start = datetime(2026-10-28T12:14:00.5191321Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                err: <*errors.Error | 0xd26a5820dc0>{
                    Op: 1,
                    Kind: 7,
                    Err: <*fmt.wrapError | 0xd26a6364d20>{
                        msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j6172544-mgmt-2\"; let ns = \"ocm-arohcpci01-2tnpie08b6e3m8ktaflq60p9p59bkmrn-q5i5v4f4b9e4k5j\"; let start = datetime(2026-10-28T12:14:00.5191321Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                        err: <*url.Error | 0xd26a719e570>{
                            Op: "Post",
                            URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                            Err: <context.deadlineExceededError>{},
                        },
                    },
                    restErrMsg: nil,
                    decoded: nil,
                    permanent: false,
                    inner: nil,
                },
            },
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-28 11:13	PR #7704	2115401772536172544

VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...

full failure pattern:

VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: dev/westus3/ci01-j5676416-mgmt-2/ocm-arohcpci01-<id>

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-dx26gclbnqgv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x3756ccac8820>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-dx26gclbnqgv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: dev/westus3/ci01-j5676416-mgmt-2/ocm-arohcpci01-2tnufqipa4peho2pme6jiu9n7kj695rj-r8q8e3h9t3h8q1o router=8004b169-b285-4616-88a5-3129e6cda029 node=aks-userswft2-37523794-vmss000000: expected matching summary after test start and within 8m; summaryUID=8004b169-b285-4616-88a5-3129e6cda029 node=aks-userswft2-37523794-vmss000000 pass=5eb81438-b4f4-4d92-b379-a83081a8b2cc at=2026-10-28T18:18:30Z
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-dx26gclbnqgv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: dev/westus3/ci01-j5676416-mgmt-2/ocm-arohcpci01-2tnufqipa4peho2pme6jiu9n7kj695rj-r8q8e3h9t3h8q1o router=8004b169-b285-4616-88a5-3129e6cda029 node=aks-userswft2-37523794-vmss000000: expected matching summary after test start and within 8m; summaryUID=8004b169-b285-4616-88a5-3129e6cda029 node=aks-userswft2-37523794-vmss000000 pass=5eb81438-b4f4-4d92-b379-a83081a8b2cc at=2026-10-28T18:18:30Z",
        err: <*errors.errorString | 0x3756cb62d010>{
            s: "dev/westus3/ci01-j5676416-mgmt-2/ocm-arohcpci01-2tnufqipa4peho2pme6jiu9n7kj695rj-r8q8e3h9t3h8q1o router=8004b169-b285-4616-88a5-3129e6cda029 node=aks-userswft2-37523794-vmss000000: expected matching summary after test start and within 8m; summaryUID=8004b169-b285-4616-88a5-3129e6cda029 node=aks-userswft2-37523794-vmss000000 pass=5eb81438-b4f4-4d92-b379-a83081a8b2cc at=2026-10-28T18:18:30Z",
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-dx26gclbnqgv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x3756ccac8820>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-dx26gclbnqgv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: dev/westus3/ci01-j5676416-mgmt-2/ocm-arohcpci01-2tnufqipa4peho2pme6jiu9n7kj695rj-r8q8e3h9t3h8q1o router=8004b169-b285-4616-88a5-3129e6cda029 node=aks-userswft2-37523794-vmss000000: expected matching summary after test start and within 8m; summaryUID=8004b169-b285-4616-88a5-3129e6cda029 node=aks-userswft2-37523794-vmss000000 pass=5eb81438-b4f4-4d92-b379-a83081a8b2cc at=2026-10-28T18:18:30Z
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-dx26gclbnqgv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: dev/westus3/ci01-j5676416-mgmt-2/ocm-arohcpci01-2tnufqipa4peho2pme6jiu9n7kj695rj-r8q8e3h9t3h8q1o router=8004b169-b285-4616-88a5-3129e6cda029 node=aks-userswft2-37523794-vmss000000: expected matching summary after test start and within 8m; summaryUID=8004b169-b285-4616-88a5-3129e6cda029 node=aks-userswft2-37523794-vmss000000 pass=5eb81438-b4f4-4d92-b379-a83081a8b2cc at=2026-10-28T18:18:30Z",
        err: <*errors.errorString | 0x3756cb62d010>{
            s: "dev/westus3/ci01-j5676416-mgmt-2/ocm-arohcpci01-2tnufqipa4peho2pme6jiu9n7kj695rj-r8q8e3h9t3h8q1o router=8004b169-b285-4616-88a5-3129e6cda029 node=aks-userswft2-37523794-vmss000000: expected matching summary after test start and within 8m; summaryUID=8004b169-b285-4616-88a5-3129e6cda029 node=aks-userswft2-37523794-vmss000000 pass=5eb81438-b4f4-4d92-b379-a83081a8b2cc at=2026-10-28T18:18:30Z",
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-28 16:49	PR #7808	2115486053375676416

VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...

full failure pattern:

VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: dev/westus3/ci01-j9298304-mgmt-2/ocm-arohcpci01-<id>: expected

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-wbzzkmqrv5j9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x6750650e460>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-wbzzkmqrv5j9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: dev/westus3/ci01-j9298304-mgmt-2/ocm-arohcpci01-2tnnvmiqd0ho3pieph8vtukoridbmhhh-t4k1z6w0y4s5q4h: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-wbzzkmqrv5j9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: dev/westus3/ci01-j9298304-mgmt-2/ocm-arohcpci01-2tnnvmiqd0ho3pieph8vtukoridbmhhh-t4k1z6w0y4s5q4h: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        err: <*verifiers.pollObservationError | 0x6750650e400>{
            observation: <*errors.errorString | 0x6750816aa40>{
                s: "dev/westus3/ci01-j9298304-mgmt-2/ocm-arohcpci01-2tnnvmiqd0ho3pieph8vtukoridbmhhh-t4k1z6w0y4s5q4h: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
            },
            interruption: <*fmt.wrapError | 0x6750650e3e0>{
                msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j9298304-mgmt-2\"; let ns = \"ocm-arohcpci01-2tnnvmiqd0ho3pieph8vtukoridbmhhh-t4k1z6w0y4s5q4h\"; let start = datetime(2026-10-28T10:25:27.2654774Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                err: <*errors.Error | 0x67506232820>{
                    Op: 1,
                    Kind: 7,
                    Err: <*fmt.wrapError | 0x6750650e3c0>{
                        msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j9298304-mgmt-2\"; let ns = \"ocm-arohcpci01-2tnnvmiqd0ho3pieph8vtukoridbmhhh-t4k1z6w0y4s5q4h\"; let start = datetime(2026-10-28T10:25:27.2654774Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                        err: <*url.Error | 0x67507892480>{
                            Op: "Post",
                            URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                            Err: <context.deadlineExceededError>{},
                        },
                    },
                    restErrMsg: nil,
                    decoded: nil,
                    permanent: false,
                    inner: nil,
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-wbzzkmqrv5j9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x6750650e460>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-wbzzkmqrv5j9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: dev/westus3/ci01-j9298304-mgmt-2/ocm-arohcpci01-2tnnvmiqd0ho3pieph8vtukoridbmhhh-t4k1z6w0y4s5q4h: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-wbzzkmqrv5j9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: dev/westus3/ci01-j9298304-mgmt-2/ocm-arohcpci01-2tnnvmiqd0ho3pieph8vtukoridbmhhh-t4k1z6w0y4s5q4h: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        err: <*verifiers.pollObservationError | 0x6750650e400>{
            observation: <*errors.errorString | 0x6750816aa40>{
                s: "dev/westus3/ci01-j9298304-mgmt-2/ocm-arohcpci01-2tnnvmiqd0ho3pieph8vtukoridbmhhh-t4k1z6w0y4s5q4h: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
            },
            interruption: <*fmt.wrapError | 0x6750650e3e0>{
                msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j9298304-mgmt-2\"; let ns = \"ocm-arohcpci01-2tnnvmiqd0ho3pieph8vtukoridbmhhh-t4k1z6w0y4s5q4h\"; let start = datetime(2026-10-28T10:25:27.2654774Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                err: <*errors.Error | 0x67506232820>{
                    Op: 1,
                    Kind: 7,
                    Err: <*fmt.wrapError | 0x6750650e3c0>{
                        msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j9298304-mgmt-2\"; let ns = \"ocm-arohcpci01-2tnnvmiqd0ho3pieph8vtukoridbmhhh-t4k1z6w0y4s5q4h\"; let start = datetime(2026-10-28T10:25:27.2654774Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                        err: <*url.Error | 0x67507892480>{
                            Op: "Post",
                            URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                            Err: <context.deadlineExceededError>{},
                        },
                    },
                    restErrMsg: nil,
                    decoded: nil,
                    permanent: false,
                    inner: nil,
                },
            },
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-28 09:34	PR #6718	2115376655219298304

VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...

full failure pattern:

VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/westus3/ci01-j2262528-mgmt-1/ocm-arohcpci01-<id>: expected all

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-8qcj8gl78j5j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x2b13ff3f4aa0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-8qcj8gl78j5j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/westus3/ci01-j2262528-mgmt-1/ocm-arohcpci01-2tnnatgpnn1ff4v4ufusun418oclrbge-r6d7p9b2f9o5i3m: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-8qcj8gl78j5j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/westus3/ci01-j2262528-mgmt-1/ocm-arohcpci01-2tnnatgpnn1ff4v4ufusun418oclrbge-r6d7p9b2f9o5i3m: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1",
        err: <*errors.errorString | 0x2b13ffa1a410>{
            s: "dev/westus3/ci01-j2262528-mgmt-1/ocm-arohcpci01-2tnnatgpnn1ff4v4ufusun418oclrbge-r6d7p9b2f9o5i3m: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1",
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-8qcj8gl78j5j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x2b13ff3f4aa0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-8qcj8gl78j5j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/westus3/ci01-j2262528-mgmt-1/ocm-arohcpci01-2tnnatgpnn1ff4v4ufusun418oclrbge-r6d7p9b2f9o5i3m: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-8qcj8gl78j5j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/westus3/ci01-j2262528-mgmt-1/ocm-arohcpci01-2tnnatgpnn1ff4v4ufusun418oclrbge-r6d7p9b2f9o5i3m: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1",
        err: <*errors.errorString | 0x2b13ffa1a410>{
            s: "dev/westus3/ci01-j2262528-mgmt-1/ocm-arohcpci01-2tnnatgpnn1ff4v4ufusun418oclrbge-r6d7p9b2f9o5i3m: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1",
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-28 08:46	PR #7704	2115364484842262528

[events] e2e-csi-disk-<id> PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node <node> (reason=ProvisioningFailed, count=...

full failure pattern:

[events] e2e-csi-disk-<id> PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node <node> (reason=ProvisioningFailed, count=8, age=7m53s)

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-cptgtj"
Unexpected error:
    <*errors.joinError | 0x199ff3994258>: 
    VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.006s: pod e2e-csi-disk-s4t9m/e2e-csi-pod phase is "Pending", expected Succeeded; PVC is Bound
    [storageclass] name=managed-csi provisioner=disk.csi.azure.com
    [pvc] e2e-csi-disk-s4t9m/e2e-csi-pvc phase=Bound volumeName=pvc-742751d0-4b4e-4c69-b7d6-859886d48626 storageClass=managed-csi
    [pod] e2e-csi-disk-s4t9m/e2e-csi-pod phase=Pending reason= message=
    [pod] condition type=PodReadyToStartContainers status=False reason= message=
    [pod] condition type=Initialized status=True reason= message=
    [pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]
    [pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]
    [pod] condition type=PodScheduled status=True reason= message=
    [pod] container volume-check ready=false restartCount=0
    [pod]   waiting reason=ContainerCreating message=
    [events] e2e-csi-disk-s4t9m Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-s4t9m/e2e-csi-pod to l1m3f0i9q3m4q6d-np-cptgtj-w9vkt-7rxb5 (reason=Scheduled, count=0, age=5m24s)
    [events] e2e-csi-disk-s4t9m Pod/e2e-csi-pod: AttachVolume.Attach failed for volume "pvc-742751d0-4b4e-4c69-b7d6-859886d48626" : rpc error: code = Internal desc = Attach volume pvc-742751d0-4b4e-4c69-b7d6-859886d48626 to instance l1m3f0i9q3m4q6d-np-cptgtj-w9vkt-7rxb5 failed with throttled due to too many requests (reason=FailedAttachVolume, count=9, age=2m0s)
    [events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)
    [events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=19, age=5m33s)
    [events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim "e2e-csi-disk-s4t9m/e2e-csi-pvc" (reason=Provisioning, count=9, age=5m45s)
    [events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node l1m3f0i9q3m4q6d-np-cptgtj-w9vkt-7rxb5 (reason=ProvisioningFailed, count=8, age=7m53s)
    [events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-742751d0-4b4e-4c69-b7d6-859886d48626 (reason=ProvisioningSucceeded, count=1, age=5m24s)
    {
        errs: [
            <*fmt.wrapError | 0x199ff3d7a160>{
                msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.006s: pod e2e-csi-disk-s4t9m/e2e-csi-pod phase is \"Pending\", expected Succeeded; PVC is Bound\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-s4t9m/e2e-csi-pvc phase=Bound volumeName=pvc-742751d0-4b4e-4c69-b7d6-859886d48626 storageClass=managed-csi\n[pod] e2e-csi-disk-s4t9m/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodReadyToStartContainers status=False reason= message=\n[pod] condition type=Initialized status=True reason= message=\n[pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=PodScheduled status=True reason= message=\n[pod] container volume-check ready=false restartCount=0\n[pod]   waiting reason=ContainerCreating message=\n[events] e2e-csi-disk-s4t9m Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-s4t9m/e2e-csi-pod to l1m3f0i9q3m4q6d-np-cptgtj-w9vkt-7rxb5 (reason=Scheduled, count=0, age=5m24s)\n[events] e2e-csi-disk-s4t9m Pod/e2e-csi-pod: AttachVolume.Attach failed for volume \"pvc-742751d0-4b4e-4c69-b7d6-859886d48626\" : rpc error: code = Internal desc = Attach volume pvc-742751d0-4b4e-4c69-b7d6-859886d48626 to instance l1m3f0i9q3m4q6d-np-cptgtj-w9vkt-7rxb5 failed with throttled due to too many requests (reason=FailedAttachVolume, count=9, age=2m0s)\n[events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)\n[events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=19, age=5m33s)\n[events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-s4t9m/e2e-csi-pvc\" (reason=Provisioning, count=9, age=5m45s)\n[events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node l1m3f0i9q3m4q6d-np-cptgtj-w9vkt-7rxb5 (reason=ProvisioningFailed, count=8, age=7m53s)\n[events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-742751d0-4b4e-4c69-b7d6-859886d48626 (reason=ProvisioningSucceeded, count=1, age=5m24s)",
                err: <*fmt.wrapError | 0x199ff3d7a020>{
                    msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.006s: pod e2e-csi-disk-s4t9m/e2e-csi-pod phase is \"Pending\", expected Succeeded; PVC is Bound\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-s4t9m/e2e-csi-pvc phase=Bound volumeName=pvc-742751d0-4b4e-4c69-b7d6-859886d48626 storageClass=managed-csi\n[pod] e2e-csi-disk-s4t9m/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodReadyToStartContainers status=False reason= message=\n[pod] condition type=Initialized status=True reason= message=\n[pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=PodScheduled status=True reason= message=\n[pod] container volume-check ready=false restartCount=0\n[pod]   waiting reason=ContainerCreating message=\n[events] e2e-csi-disk-s4t9m Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-s4t9m/e2e-csi-pod to l1m3f0i9q3m4q6d-np-cptgtj-w9vkt-7rxb5 (reason=Scheduled, count=0, age=5m24s)\n[events] e2e-csi-disk-s4t9m Pod/e2e-csi-pod: AttachVolume.Attach failed for volume \"pvc-742751d0-4b4e-4c69-b7d6-859886d48626\" : rpc error: code = Internal desc = Attach volume pvc-742751d0-4b4e-4c69-b7d6-859886d48626 to instance l1m3f0i9q3m4q6d-np-cptgtj-w9vkt-7rxb5 failed with throttled due to too many requests (reason=FailedAttachVolume, count=9, age=2m0s)\n[events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)\n[events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=19, age=5m33s)\n[events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-s4t9m/e2e-csi-pvc\" (reason=Provisioning, count=9, age=5m45s)\n[events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node l1m3f0i9q3m4q6d-np-cptgtj-w9vkt-7rxb5 (reason=ProvisioningFailed, count=8, age=7m53s)\n[events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-742751d0-4b4e-4c69-b7d6-859886d48626 (reason=ProvisioningSucceeded, count=1, age=5m24s)",
                    err: <*verifiers.pollObservationError | 0x199ff3d7a680>{
                        observation: <*errors.errorString | 0x199ff36faf30>{
                            s: "pod e2e-csi-disk-s4t9m/e2e-csi-pod phase is \"Pending\", expected Succeeded; PVC is Bound",
                        },
                        interruption: <*fmt.wrapError | 0x199ff3d7a660>{
                            msg: "failed to get PVC e2e-csi-disk-s4t9m/e2e-csi-pvc: context deadline exceeded",
                            err: <context.deadlineExceededError>{},
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-cptgtj"
Unexpected error:
    <*errors.joinError | 0x199ff3994258>: 
    VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.006s: pod e2e-csi-disk-s4t9m/e2e-csi-pod phase is "Pending", expected Succeeded; PVC is Bound
    [storageclass] name=managed-csi provisioner=disk.csi.azure.com
    [pvc] e2e-csi-disk-s4t9m/e2e-csi-pvc phase=Bound volumeName=pvc-742751d0-4b4e-4c69-b7d6-859886d48626 storageClass=managed-csi
    [pod] e2e-csi-disk-s4t9m/e2e-csi-pod phase=Pending reason= message=
    [pod] condition type=PodReadyToStartContainers status=False reason= message=
    [pod] condition type=Initialized status=True reason= message=
    [pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]
    [pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]
    [pod] condition type=PodScheduled status=True reason= message=
    [pod] container volume-check ready=false restartCount=0
    [pod]   waiting reason=ContainerCreating message=
    [events] e2e-csi-disk-s4t9m Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-s4t9m/e2e-csi-pod to l1m3f0i9q3m4q6d-np-cptgtj-w9vkt-7rxb5 (reason=Scheduled, count=0, age=5m24s)
    [events] e2e-csi-disk-s4t9m Pod/e2e-csi-pod: AttachVolume.Attach failed for volume "pvc-742751d0-4b4e-4c69-b7d6-859886d48626" : rpc error: code = Internal desc = Attach volume pvc-742751d0-4b4e-4c69-b7d6-859886d48626 to instance l1m3f0i9q3m4q6d-np-cptgtj-w9vkt-7rxb5 failed with throttled due to too many requests (reason=FailedAttachVolume, count=9, age=2m0s)
    [events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)
    [events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=19, age=5m33s)
    [events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim "e2e-csi-disk-s4t9m/e2e-csi-pvc" (reason=Provisioning, count=9, age=5m45s)
    [events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node l1m3f0i9q3m4q6d-np-cptgtj-w9vkt-7rxb5 (reason=ProvisioningFailed, count=8, age=7m53s)
    [events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-742751d0-4b4e-4c69-b7d6-859886d48626 (reason=ProvisioningSucceeded, count=1, age=5m24s)
    {
        errs: [
            <*fmt.wrapError | 0x199ff3d7a160>{
                msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.006s: pod e2e-csi-disk-s4t9m/e2e-csi-pod phase is \"Pending\", expected Succeeded; PVC is Bound\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-s4t9m/e2e-csi-pvc phase=Bound volumeName=pvc-742751d0-4b4e-4c69-b7d6-859886d48626 storageClass=managed-csi\n[pod] e2e-csi-disk-s4t9m/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodReadyToStartContainers status=False reason= message=\n[pod] condition type=Initialized status=True reason= message=\n[pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=PodScheduled status=True reason= message=\n[pod] container volume-check ready=false restartCount=0\n[pod]   waiting reason=ContainerCreating message=\n[events] e2e-csi-disk-s4t9m Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-s4t9m/e2e-csi-pod to l1m3f0i9q3m4q6d-np-cptgtj-w9vkt-7rxb5 (reason=Scheduled, count=0, age=5m24s)\n[events] e2e-csi-disk-s4t9m Pod/e2e-csi-pod: AttachVolume.Attach failed for volume \"pvc-742751d0-4b4e-4c69-b7d6-859886d48626\" : rpc error: code = Internal desc = Attach volume pvc-742751d0-4b4e-4c69-b7d6-859886d48626 to instance l1m3f0i9q3m4q6d-np-cptgtj-w9vkt-7rxb5 failed with throttled due to too many requests (reason=FailedAttachVolume, count=9, age=2m0s)\n[events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)\n[events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=19, age=5m33s)\n[events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-s4t9m/e2e-csi-pvc\" (reason=Provisioning, count=9, age=5m45s)\n[events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node l1m3f0i9q3m4q6d-np-cptgtj-w9vkt-7rxb5 (reason=ProvisioningFailed, count=8, age=7m53s)\n[events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-742751d0-4b4e-4c69-b7d6-859886d48626 (reason=ProvisioningSucceeded, count=1, age=5m24s)",
                err: <*fmt.wrapError | 0x199ff3d7a020>{
                    msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.006s: pod e2e-csi-disk-s4t9m/e2e-csi-pod phase is \"Pending\", expected Succeeded; PVC is Bound\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-s4t9m/e2e-csi-pvc phase=Bound volumeName=pvc-742751d0-4b4e-4c69-b7d6-859886d48626 storageClass=managed-csi\n[pod] e2e-csi-disk-s4t9m/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodReadyToStartContainers status=False reason= message=\n[pod] condition type=Initialized status=True reason= message=\n[pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=PodScheduled status=True reason= message=\n[pod] container volume-check ready=false restartCount=0\n[pod]   waiting reason=ContainerCreating message=\n[events] e2e-csi-disk-s4t9m Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-s4t9m/e2e-csi-pod to l1m3f0i9q3m4q6d-np-cptgtj-w9vkt-7rxb5 (reason=Scheduled, count=0, age=5m24s)\n[events] e2e-csi-disk-s4t9m Pod/e2e-csi-pod: AttachVolume.Attach failed for volume \"pvc-742751d0-4b4e-4c69-b7d6-859886d48626\" : rpc error: code = Internal desc = Attach volume pvc-742751d0-4b4e-4c69-b7d6-859886d48626 to instance l1m3f0i9q3m4q6d-np-cptgtj-w9vkt-7rxb5 failed with throttled due to too many requests (reason=FailedAttachVolume, count=9, age=2m0s)\n[events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)\n[events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=19, age=5m33s)\n[events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-s4t9m/e2e-csi-pvc\" (reason=Provisioning, count=9, age=5m45s)\n[events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node l1m3f0i9q3m4q6d-np-cptgtj-w9vkt-7rxb5 (reason=ProvisioningFailed, count=8, age=7m53s)\n[events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-742751d0-4b4e-4c69-b7d6-859886d48626 (reason=ProvisioningSucceeded, count=1, age=5m24s)",
                    err: <*verifiers.pollObservationError | 0x199ff3d7a680>{
                        observation: <*errors.errorString | 0x199ff36faf30>{
                            s: "pod e2e-csi-disk-s4t9m/e2e-csi-pod phase is \"Pending\", expected Succeeded; PVC is Bound",
                        },
                        interruption: <*fmt.wrapError | 0x199ff3d7a660>{
                            msg: "failed to get PVC e2e-csi-disk-s4t9m/e2e-csi-pvc: context deadline exceeded",
                            err: <context.deadlineExceededError>{},
                        },
                    },
                },
            },
        ],
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-28 15:14	PR #7802	2115462331566657536

alert [svc] hypershiftOperatorPodsDown fired

full failure pattern:

alert [svc] hypershiftOperatorPodsDown fired

Signal:

alert

none

Full failure examples (1)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-29T00:34:30Z
  Ended:       2026-10-29T00:40:28Z
  Severity:    Sev2
  Labels:      alertname="hypershiftOperatorPodsDown", cluster="ci01-j8902528-mgmt-2", component="slo", service="hypershift-operator", severity="2", slo="hypershift-operator-availability"
  Description: 2 hypershift-operator pods have been unavailable for 5 minutes.
Management Cluster: ci01-j8902528-mgmt-2

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert hypershiftOperatorPodsDown does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-28 23:45	PR #7793	2115590796378902528