CIHealth Failure Patterns

Environment: DEV

Failure patterns: 65 · Failures matched: 120 · Total runs: 113 · Runs affected: 73

Failure pattern

Failed at

Trend

Also in

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-<id> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-<id> to exist; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x26678ea60858>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-f8mftsvbkmbp" to finish creating: GET https://rp.j2959360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/95ed2027-596a-4016-be69-40cc2e92cf3f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/95ed2027-596a-4016-be69-40cc2e92cf3f",
      "name": "95ed2027-596a-4016-be69-40cc2e92cf3f",
      "status": "Failed",
      "startTime": "2026-10-27T01:20:09.676863434Z",
      "endTime": "2026-10-27T01:39:10.703576848Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-j8lrv-nn5dn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-f8mftsvbkmbp--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-j8lrv-nn5dn to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-j8lrv-nn5dn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-f8mftsvbkmbp--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-j8lrv-nn5dn to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2959360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/95ed2027-596a-4016-be69-40cc2e92cf3f
            correlation ID: 7f67a06a-5085-4c8e-9661-1df8e8a79a0a
            client request ID: 9930c3d6-bc48-4ffb-b0a4-b32c7c60a790
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x26678ea60858>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-f8mftsvbkmbp" to finish creating: GET https://rp.j2959360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/95ed2027-596a-4016-be69-40cc2e92cf3f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/95ed2027-596a-4016-be69-40cc2e92cf3f",
      "name": "95ed2027-596a-4016-be69-40cc2e92cf3f",
      "status": "Failed",
      "startTime": "2026-10-27T01:20:09.676863434Z",
      "endTime": "2026-10-27T01:39:10.703576848Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-j8lrv-nn5dn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-f8mftsvbkmbp--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-j8lrv-nn5dn to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-j8lrv-nn5dn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-f8mftsvbkmbp--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-j8lrv-nn5dn to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2959360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/95ed2027-596a-4016-be69-40cc2e92cf3f
            correlation ID: 7f67a06a-5085-4c8e-9661-1df8e8a79a0a
            client request ID: 9930c3d6-bc48-4ffb-b0a4-b32c7c60a790
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-27 00:03	PR #7177	2114870474402959360

ERROR CODE: DeploymentFailed; detail code InvalidRequest; detail message Call to Microsoft.EventGrid/namespaces failed. Error message: Required property 'linkIdentifier' not found...

full failure pattern:

ERROR CODE: DeploymentFailed; detail code InvalidRequest; detail message Call to Microsoft.EventGrid/namespaces failed. Error message: Required property 'linkIdentifier' not found in JSON. Path 'remotePrivateEndpoint.connectionDetails[0]', line 1, position 932.; provider Microsoft.EventGrid

Signal:

provision

none

Full failure examples (1)

time=2026-10-27T07:31:38.140Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 description="Step cluster\n  Kind: ARM\n  Template: templates/mgmt-cluster.bicep\n  Parameters: configurations/mgmt-cluster.tmpl.bicepparam"
time=2026-10-27T07:31:38.545Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1
time=2026-10-27T07:31:39.254Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 deployment=95d11f391192e4ffeb516d68e892ddaaf99516942ada8720f17881ddeb56eda0 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1948032-mgmt-1%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F95d11f391192e4ffeb516d68e892ddaaf99516942ada8720f17881ddeb56eda0
time=2026-10-27T07:33:09.610Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 err="stamp 1: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1948032-mgmt-1/providers/Microsoft.Resources/deployments/95d11f391192e4ffeb516d68e892ddaaf99516942ada8720f17881ddeb56eda0/operationStatuses/08584085205866172617\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1948032-mgmt-1/providers/Microsoft.Resources/deployments/eventGridPrivateEndpoint\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"InvalidRequest\\\",\\r\\n            \\\"message\\\": \\\"Call to Microsoft.EventGrid/namespaces failed. Error message: Required property 'linkIdentifier' not found in JSON. Path 'remotePrivateEndpoint.connectionDetails[0]', line 1, position 932.\\\",\\r\\n            \\\"details\\\": []\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster (stamp=1)	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-27 06:52	PR #6364	2114973576971948032

ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-4sk7fq' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...

full failure pattern:

ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-4sk7fq' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'.; provider Microsoft.Network

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-jz7tkx"
Unexpected error:
    <*errors.joinError | 0x3490511d0108>: 
    VerifyIngressDNSRecord(aro.v8z3q8n3u9i1v9l.eahs.j1816448.hcp.osadev.cloud) failed: VerifyIngressDNSRecord(aro.v8z3q8n3u9i1v9l.eahs.j1816448.hcp.osadev.cloud) timed out after 11m45.097s: DNS A record "*.e2e-dns-4sk7fq" not yet present in zone rg-candidate-4-23-5sw7d6kw8f6m-candidate-jz7tkx--managed/aro.v8z3q8n3u9i1v9l.eahs.j1816448.hcp.osadev.cloud: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-5sw7d6kw8f6m-candidate-jz7tkx--managed/providers/Microsoft.Network/dnsZones/aro.v8z3q8n3u9i1v9l.eahs.j1816448.hcp.osadev.cloud/A/*.e2e-dns-4sk7fq
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: NotFound
    --------------------------------------------------------------------------------
    {
      "code": "NotFound",
      "message": "The resource record '*.e2e-dns-4sk7fq' does not exist in resource group 'rg-candidate-4-23-5sw7d6kw8f6m-candidate-jz7tkx--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-5sw7d6kw8f6m-candidate-jz7tkx--managed/providers/Microsoft.Network/dnsZones/aro.v8z3q8n3u9i1v9l.eahs.j1816448.hcp.osadev.cloud/A/*.e2e-dns-4sk7fq
            correlation ID: 7140a367-bc3d-43f2-b136-ffe5167138f0
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-jz7tkx"
Unexpected error:
    <*errors.joinError | 0x3490511d0108>: 
    VerifyIngressDNSRecord(aro.v8z3q8n3u9i1v9l.eahs.j1816448.hcp.osadev.cloud) failed: VerifyIngressDNSRecord(aro.v8z3q8n3u9i1v9l.eahs.j1816448.hcp.osadev.cloud) timed out after 11m45.097s: DNS A record "*.e2e-dns-4sk7fq" not yet present in zone rg-candidate-4-23-5sw7d6kw8f6m-candidate-jz7tkx--managed/aro.v8z3q8n3u9i1v9l.eahs.j1816448.hcp.osadev.cloud: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-5sw7d6kw8f6m-candidate-jz7tkx--managed/providers/Microsoft.Network/dnsZones/aro.v8z3q8n3u9i1v9l.eahs.j1816448.hcp.osadev.cloud/A/*.e2e-dns-4sk7fq
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: NotFound
    --------------------------------------------------------------------------------
    {
      "code": "NotFound",
      "message": "The resource record '*.e2e-dns-4sk7fq' does not exist in resource group 'rg-candidate-4-23-5sw7d6kw8f6m-candidate-jz7tkx--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-5sw7d6kw8f6m-candidate-jz7tkx--managed/providers/Microsoft.Network/dnsZones/aro.v8z3q8n3u9i1v9l.eahs.j1816448.hcp.osadev.cloud/A/*.e2e-dns-4sk7fq
            correlation ID: 7140a367-bc3d-43f2-b136-ffe5167138f0
            client request ID: unavailable
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-27 03:15	PR #6364	2114918973651816448

ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-fs2vtm' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...

full failure pattern:

ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-fs2vtm' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'.; provider Microsoft.Network

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-9pvn6m"
Unexpected error:
    <*errors.joinError | 0x6c70d0280d8>: 
    VerifyIngressDNSRecord(aro.k7l7c8r8l9b3j7g.kbo1.j5963776.hcp.osadev.cloud) failed: VerifyIngressDNSRecord(aro.k7l7c8r8l9b3j7g.kbo1.j5963776.hcp.osadev.cloud) timed out after 10m0.004s: DNS A record "*.e2e-dns-fs2vtm" not yet present in zone rg-candidate-4-23-kkv4f4xfm4kx-candidate-9pvn6m--managed/aro.k7l7c8r8l9b3j7g.kbo1.j5963776.hcp.osadev.cloud: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-kkv4f4xfm4kx-candidate-9pvn6m--managed/providers/Microsoft.Network/dnsZones/aro.k7l7c8r8l9b3j7g.kbo1.j5963776.hcp.osadev.cloud/A/*.e2e-dns-fs2vtm
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: NotFound
    --------------------------------------------------------------------------------
    {
      "code": "NotFound",
      "message": "The resource record '*.e2e-dns-fs2vtm' does not exist in resource group 'rg-candidate-4-23-kkv4f4xfm4kx-candidate-9pvn6m--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-kkv4f4xfm4kx-candidate-9pvn6m--managed/providers/Microsoft.Network/dnsZones/aro.k7l7c8r8l9b3j7g.kbo1.j5963776.hcp.osadev.cloud/A/*.e2e-dns-fs2vtm
            correlation ID: 850b8d7d-f572-4c32-8b7c-4052d22cd119
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-9pvn6m"
Unexpected error:
    <*errors.joinError | 0x6c70d0280d8>: 
    VerifyIngressDNSRecord(aro.k7l7c8r8l9b3j7g.kbo1.j5963776.hcp.osadev.cloud) failed: VerifyIngressDNSRecord(aro.k7l7c8r8l9b3j7g.kbo1.j5963776.hcp.osadev.cloud) timed out after 10m0.004s: DNS A record "*.e2e-dns-fs2vtm" not yet present in zone rg-candidate-4-23-kkv4f4xfm4kx-candidate-9pvn6m--managed/aro.k7l7c8r8l9b3j7g.kbo1.j5963776.hcp.osadev.cloud: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-kkv4f4xfm4kx-candidate-9pvn6m--managed/providers/Microsoft.Network/dnsZones/aro.k7l7c8r8l9b3j7g.kbo1.j5963776.hcp.osadev.cloud/A/*.e2e-dns-fs2vtm
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: NotFound
    --------------------------------------------------------------------------------
    {
      "code": "NotFound",
      "message": "The resource record '*.e2e-dns-fs2vtm' does not exist in resource group 'rg-candidate-4-23-kkv4f4xfm4kx-candidate-9pvn6m--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-kkv4f4xfm4kx-candidate-9pvn6m--managed/providers/Microsoft.Network/dnsZones/aro.k7l7c8r8l9b3j7g.kbo1.j5963776.hcp.osadev.cloud/A/*.e2e-dns-fs2vtm
            correlation ID: 850b8d7d-f572-4c32-8b7c-4052d22cd119
            client request ID: unavailable
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-27 12:07	PR #7373	2115052847685963776

ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-gkfmqf' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...

full failure pattern:

ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-gkfmqf' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'.; provider Microsoft.Network

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-n5f5ld"
Unexpected error:
    <*errors.joinError | 0x29f2244004b0>: 
    VerifyIngressDNSRecord(aro.l9k5u3r1j0f0v1z.l4tl.j1933056.hcp.osadev.cloud) failed: VerifyIngressDNSRecord(aro.l9k5u3r1j0f0v1z.l4tl.j1933056.hcp.osadev.cloud) timed out after 10m0.005s: DNS A record "*.e2e-dns-gkfmqf" not yet present in zone rg-candidate-4-23-tpslzg6rj8xj-candidate-n5f5ld--managed/aro.l9k5u3r1j0f0v1z.l4tl.j1933056.hcp.osadev.cloud: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-tpslzg6rj8xj-candidate-n5f5ld--managed/providers/Microsoft.Network/dnsZones/aro.l9k5u3r1j0f0v1z.l4tl.j1933056.hcp.osadev.cloud/A/*.e2e-dns-gkfmqf
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: NotFound
    --------------------------------------------------------------------------------
    {
      "code": "NotFound",
      "message": "The resource record '*.e2e-dns-gkfmqf' does not exist in resource group 'rg-candidate-4-23-tpslzg6rj8xj-candidate-n5f5ld--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-tpslzg6rj8xj-candidate-n5f5ld--managed/providers/Microsoft.Network/dnsZones/aro.l9k5u3r1j0f0v1z.l4tl.j1933056.hcp.osadev.cloud/A/*.e2e-dns-gkfmqf
            correlation ID: d271f6f6-6cb4-472d-b6b5-9e1294932892
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-n5f5ld"
Unexpected error:
    <*errors.joinError | 0x29f2244004b0>: 
    VerifyIngressDNSRecord(aro.l9k5u3r1j0f0v1z.l4tl.j1933056.hcp.osadev.cloud) failed: VerifyIngressDNSRecord(aro.l9k5u3r1j0f0v1z.l4tl.j1933056.hcp.osadev.cloud) timed out after 10m0.005s: DNS A record "*.e2e-dns-gkfmqf" not yet present in zone rg-candidate-4-23-tpslzg6rj8xj-candidate-n5f5ld--managed/aro.l9k5u3r1j0f0v1z.l4tl.j1933056.hcp.osadev.cloud: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-tpslzg6rj8xj-candidate-n5f5ld--managed/providers/Microsoft.Network/dnsZones/aro.l9k5u3r1j0f0v1z.l4tl.j1933056.hcp.osadev.cloud/A/*.e2e-dns-gkfmqf
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: NotFound
    --------------------------------------------------------------------------------
    {
      "code": "NotFound",
      "message": "The resource record '*.e2e-dns-gkfmqf' does not exist in resource group 'rg-candidate-4-23-tpslzg6rj8xj-candidate-n5f5ld--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-tpslzg6rj8xj-candidate-n5f5ld--managed/providers/Microsoft.Network/dnsZones/aro.l9k5u3r1j0f0v1z.l4tl.j1933056.hcp.osadev.cloud/A/*.e2e-dns-gkfmqf
            correlation ID: d271f6f6-6cb4-472d-b6b5-9e1294932892
            client request ID: unavailable
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-27 22:20	PR #6924	2115207073061933056

ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-qg48vc' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...

full failure pattern:

ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-qg48vc' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'.; provider Microsoft.Network

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-458kh7"
Unexpected error:
    <*errors.joinError | 0x1b0cdceb6450>: 
    VerifyIngressDNSRecord(aro.g7r4k9t5x4l0l6s.4rfh.j1218048.hcp.osadev.cloud) failed: VerifyIngressDNSRecord(aro.g7r4k9t5x4l0l6s.4rfh.j1218048.hcp.osadev.cloud) timed out after 10m0.005s: DNS A record "*.e2e-dns-qg48vc" not yet present in zone rg-candidate-4-23-m5tx8lx5j5bn-candidate-458kh7--managed/aro.g7r4k9t5x4l0l6s.4rfh.j1218048.hcp.osadev.cloud: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-m5tx8lx5j5bn-candidate-458kh7--managed/providers/Microsoft.Network/dnsZones/aro.g7r4k9t5x4l0l6s.4rfh.j1218048.hcp.osadev.cloud/A/*.e2e-dns-qg48vc
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: NotFound
    --------------------------------------------------------------------------------
    {
      "code": "NotFound",
      "message": "The resource record '*.e2e-dns-qg48vc' does not exist in resource group 'rg-candidate-4-23-m5tx8lx5j5bn-candidate-458kh7--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-m5tx8lx5j5bn-candidate-458kh7--managed/providers/Microsoft.Network/dnsZones/aro.g7r4k9t5x4l0l6s.4rfh.j1218048.hcp.osadev.cloud/A/*.e2e-dns-qg48vc
            correlation ID: 1332f9fe-dcac-4684-bfa0-1d5d13e9cf06
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-458kh7"
Unexpected error:
    <*errors.joinError | 0x1b0cdceb6450>: 
    VerifyIngressDNSRecord(aro.g7r4k9t5x4l0l6s.4rfh.j1218048.hcp.osadev.cloud) failed: VerifyIngressDNSRecord(aro.g7r4k9t5x4l0l6s.4rfh.j1218048.hcp.osadev.cloud) timed out after 10m0.005s: DNS A record "*.e2e-dns-qg48vc" not yet present in zone rg-candidate-4-23-m5tx8lx5j5bn-candidate-458kh7--managed/aro.g7r4k9t5x4l0l6s.4rfh.j1218048.hcp.osadev.cloud: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-m5tx8lx5j5bn-candidate-458kh7--managed/providers/Microsoft.Network/dnsZones/aro.g7r4k9t5x4l0l6s.4rfh.j1218048.hcp.osadev.cloud/A/*.e2e-dns-qg48vc
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: NotFound
    --------------------------------------------------------------------------------
    {
      "code": "NotFound",
      "message": "The resource record '*.e2e-dns-qg48vc' does not exist in resource group 'rg-candidate-4-23-m5tx8lx5j5bn-candidate-458kh7--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-m5tx8lx5j5bn-candidate-458kh7--managed/providers/Microsoft.Network/dnsZones/aro.g7r4k9t5x4l0l6s.4rfh.j1218048.hcp.osadev.cloud/A/*.e2e-dns-qg48vc
            correlation ID: 1332f9fe-dcac-4684-bfa0-1d5d13e9cf06
            client request ID: unavailable
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-27 06:56	PR #7662	2114974410921218048

VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node cluster-np-prdbsm-<id> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-...

full failure pattern:

VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node cluster-np-prdbsm-<id> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-prdbsm-<id>": nodes "cluster-np-prdbsm-<id>" is forbidden

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-0-prdbsm"
Unexpected error:
    <*errors.joinError | 0xfaf21d3c618>: 
    VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-prdbsm-dxj9f-p58c2 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-prdbsm-dxj9f-p58c2": nodes "cluster-np-prdbsm-dxj9f-p58c2" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
    {
        errs: [
            <*fmt.wrapError | 0xfaf223bc0a0>{
                msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-prdbsm-dxj9f-p58c2 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-prdbsm-dxj9f-p58c2\": nodes \"cluster-np-prdbsm-dxj9f-p58c2\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                err: <*fmt.wrapError | 0xfaf223bc080>{
                    msg: "failed to label node cluster-np-prdbsm-dxj9f-p58c2 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-prdbsm-dxj9f-p58c2\": nodes \"cluster-np-prdbsm-dxj9f-p58c2\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                    err: <*fmt.wrapError | 0xfaf223bc060>{
                        msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-prdbsm-dxj9f-p58c2\": nodes \"cluster-np-prdbsm-dxj9f-p58c2\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                        err: <*errors.StatusError | 0xfaf21fc43c0>{
                            ErrStatus: {
                                TypeMeta: {Kind: "", APIVersion: ""},
                                ListMeta: {
                                    SelfLink: "",
                                    ResourceVersion: "",
                                    Continue: "",
                                    RemainingItemCount: nil,
                                    ShardInfo: nil,
                                },
                                Status: "Failure",
                                Message: "nodes \"cluster-np-prdbsm-dxj9f-p58c2\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                                Reason: "Forbidden",
                                Details: {
                                    Name: "cluster-np-prdbsm-dxj9f-p58c2",
                                    Group: "",
                                    Kind: "nodes",
                                    UID: "",
                                    Causes: nil,
                                    RetryAfterSeconds: 0,
                                },
                                Code: 403,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-0-prdbsm"
Unexpected error:
    <*errors.joinError | 0xfaf21d3c618>: 
    VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-prdbsm-dxj9f-p58c2 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-prdbsm-dxj9f-p58c2": nodes "cluster-np-prdbsm-dxj9f-p58c2" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
    {
        errs: [
            <*fmt.wrapError | 0xfaf223bc0a0>{
                msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-prdbsm-dxj9f-p58c2 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-prdbsm-dxj9f-p58c2\": nodes \"cluster-np-prdbsm-dxj9f-p58c2\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                err: <*fmt.wrapError | 0xfaf223bc080>{
                    msg: "failed to label node cluster-np-prdbsm-dxj9f-p58c2 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-prdbsm-dxj9f-p58c2\": nodes \"cluster-np-prdbsm-dxj9f-p58c2\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                    err: <*fmt.wrapError | 0xfaf223bc060>{
                        msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-prdbsm-dxj9f-p58c2\": nodes \"cluster-np-prdbsm-dxj9f-p58c2\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                        err: <*errors.StatusError | 0xfaf21fc43c0>{
                            ErrStatus: {
                                TypeMeta: {Kind: "", APIVersion: ""},
                                ListMeta: {
                                    SelfLink: "",
                                    ResourceVersion: "",
                                    Continue: "",
                                    RemainingItemCount: nil,
                                    ShardInfo: nil,
                                },
                                Status: "Failure",
                                Message: "nodes \"cluster-np-prdbsm-dxj9f-p58c2\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                                Reason: "Forbidden",
                                Details: {
                                    Name: "cluster-np-prdbsm-dxj9f-p58c2",
                                    Group: "",
                                    Kind: "nodes",
                                    UID: "",
                                    Causes: nil,
                                    RetryAfterSeconds: 0,
                                },
                                Code: 403,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-27 22:25	PR #7787	2115208311295971328

VerifyClusterVisibleInConsole timed out after 20m0.02s: failed to query OCM clusters API for external_id "<uuid>": Get "<url>": can't get access token: invalid_grant: Invalid refr...

full failure pattern:

VerifyClusterVisibleInConsole timed out after 20m0.02s: failed to query OCM clusters API for external_id "<uuid>": Get "<url>": can't get access token: invalid_grant: Invalid refresh token

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_insights_operator.go:159]: cluster did not become visible in console.redhat.com
Unexpected error:
    <*fmt.wrapError | 0x1f4ce1090c80>: 
    VerifyClusterVisibleInConsole timed out after 20m0.02s: failed to query OCM clusters API for external_id "462efae8-075b-4d74-bf89-a1bd7f9d3b6f": Get "https://api.openshift.com/api/clusters_mgmt/v1/clusters?search=external_id+%3D+%27462efae8-075b-4d74-bf89-a1bd7f9d3b6f%27&size=1": can't get access token: invalid_grant: Invalid refresh token
    {
        msg: "VerifyClusterVisibleInConsole timed out after 20m0.02s: failed to query OCM clusters API for external_id \"462efae8-075b-4d74-bf89-a1bd7f9d3b6f\": Get \"https://api.openshift.com/api/clusters_mgmt/v1/clusters?search=external_id+%3D+%27462efae8-075b-4d74-bf89-a1bd7f9d3b6f%27&size=1\": can't get access token: invalid_grant: Invalid refresh token",
        err: <*fmt.wrapError | 0x1f4ce110ccc0>{
            msg: "failed to query OCM clusters API for external_id \"462efae8-075b-4d74-bf89-a1bd7f9d3b6f\": Get \"https://api.openshift.com/api/clusters_mgmt/v1/clusters?search=external_id+%3D+%27462efae8-075b-4d74-bf89-a1bd7f9d3b6f%27&size=1\": can't get access token: invalid_grant: Invalid refresh token",
            err: <*url.Error | 0x1f4ce0ce8a80>{
                Op: "Get",
                URL: "https://api.openshift.com/api/clusters_mgmt/v1/clusters?search=external_id+%3D+%27462efae8-075b-4d74-bf89-a1bd7f9d3b6f%27&size=1",
                Err: <*fmt.wrapError | 0x1f4ce110cca0>{
                    msg: "can't get access token: invalid_grant: Invalid refresh token",
                    err: <*errors.errorString | 0x1f4ce1105790>{
                        s: "invalid_grant: Invalid refresh token",
                    },
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_insights_operator.go:159]: cluster did not become visible in console.redhat.com
Unexpected error:
    <*fmt.wrapError | 0x1f4ce1090c80>: 
    VerifyClusterVisibleInConsole timed out after 20m0.02s: failed to query OCM clusters API for external_id "462efae8-075b-4d74-bf89-a1bd7f9d3b6f": Get "https://api.openshift.com/api/clusters_mgmt/v1/clusters?search=external_id+%3D+%27462efae8-075b-4d74-bf89-a1bd7f9d3b6f%27&size=1": can't get access token: invalid_grant: Invalid refresh token
    {
        msg: "VerifyClusterVisibleInConsole timed out after 20m0.02s: failed to query OCM clusters API for external_id \"462efae8-075b-4d74-bf89-a1bd7f9d3b6f\": Get \"https://api.openshift.com/api/clusters_mgmt/v1/clusters?search=external_id+%3D+%27462efae8-075b-4d74-bf89-a1bd7f9d3b6f%27&size=1\": can't get access token: invalid_grant: Invalid refresh token",
        err: <*fmt.wrapError | 0x1f4ce110ccc0>{
            msg: "failed to query OCM clusters API for external_id \"462efae8-075b-4d74-bf89-a1bd7f9d3b6f\": Get \"https://api.openshift.com/api/clusters_mgmt/v1/clusters?search=external_id+%3D+%27462efae8-075b-4d74-bf89-a1bd7f9d3b6f%27&size=1\": can't get access token: invalid_grant: Invalid refresh token",
            err: <*url.Error | 0x1f4ce0ce8a80>{
                Op: "Get",
                URL: "https://api.openshift.com/api/clusters_mgmt/v1/clusters?search=external_id+%3D+%27462efae8-075b-4d74-bf89-a1bd7f9d3b6f%27&size=1",
                Err: <*fmt.wrapError | 0x1f4ce110cca0>{
                    msg: "can't get access token: invalid_grant: Invalid refresh token",
                    err: <*errors.errorString | 0x1f4ce1105790>{
                        s: "invalid_grant: Invalid refresh token",
                    },
                },
            },
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to see an HCP cluster registered with console.redhat.com via the Insights Operator	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-27 15:06	PR #6954	2115097821592948736

VerifyInsightsOperatorDisabledNoToken timed out after 10m0.005s: ClusterOperator "insights" has no Disabled condition yet ()

full failure pattern:

VerifyInsightsOperatorDisabledNoToken timed out after 10m0.005s: ClusterOperator "insights" has no Disabled condition yet ()

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_insights_operator.go:117]: Insights Operator was not deployed or did not reach Disabled/NoToken state
Unexpected error:
    <*fmt.wrapError | 0xa7745440900>: 
    VerifyInsightsOperatorDisabledNoToken timed out after 10m0.005s: ClusterOperator "insights" has no Disabled condition yet ()
    {
        msg: "VerifyInsightsOperatorDisabledNoToken timed out after 10m0.005s: ClusterOperator \"insights\" has no Disabled condition yet ()",
        err: <*verifiers.pollObservationError | 0xa77454408e0>{
            observation: <*errors.errorString | 0xa77456c4580>{
                s: "ClusterOperator \"insights\" has no Disabled condition yet ()",
            },
            interruption: <*fmt.wrapError | 0xa77454408c0>{
                msg: "failed to get ClusterOperator \"insights\": context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_insights_operator.go:117]: Insights Operator was not deployed or did not reach Disabled/NoToken state
Unexpected error:
    <*fmt.wrapError | 0xa7745440900>: 
    VerifyInsightsOperatorDisabledNoToken timed out after 10m0.005s: ClusterOperator "insights" has no Disabled condition yet ()
    {
        msg: "VerifyInsightsOperatorDisabledNoToken timed out after 10m0.005s: ClusterOperator \"insights\" has no Disabled condition yet ()",
        err: <*verifiers.pollObservationError | 0xa77454408e0>{
            observation: <*errors.errorString | 0xa77456c4580>{
                s: "ClusterOperator \"insights\" has no Disabled condition yet ()",
            },
            interruption: <*fmt.wrapError | 0xa77454408c0>{
                msg: "failed to get ClusterOperator \"insights\": context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to see an HCP cluster registered with console.redhat.com via the Insights Operator	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-27 11:29	PR #6954	2115043139386347520

VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...

full failure pattern:

VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/centralus/ci01-j4637312-mgmt-2/ocm-arohcpci01-<id>: expected

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-w7kwr6bpq57t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x2874138e2000>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-w7kwr6bpq57t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/centralus/ci01-j4637312-mgmt-2/ocm-arohcpci01-2tn2f6qu0skm7r1cd4s42o9104lkbkll-m9o9d5o9m2l7k4w: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-w7kwr6bpq57t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/centralus/ci01-j4637312-mgmt-2/ocm-arohcpci01-2tn2f6qu0skm7r1cd4s42o9104lkbkll-m9o9d5o9m2l7k4w: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1",
        err: <*errors.errorString | 0x2874134f7c90>{
            s: "dev/centralus/ci01-j4637312-mgmt-2/ocm-arohcpci01-2tn2f6qu0skm7r1cd4s42o9104lkbkll-m9o9d5o9m2l7k4w: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1",
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-w7kwr6bpq57t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x2874138e2000>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-w7kwr6bpq57t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/centralus/ci01-j4637312-mgmt-2/ocm-arohcpci01-2tn2f6qu0skm7r1cd4s42o9104lkbkll-m9o9d5o9m2l7k4w: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-w7kwr6bpq57t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/centralus/ci01-j4637312-mgmt-2/ocm-arohcpci01-2tn2f6qu0skm7r1cd4s42o9104lkbkll-m9o9d5o9m2l7k4w: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1",
        err: <*errors.errorString | 0x2874134f7c90>{
            s: "dev/centralus/ci01-j4637312-mgmt-2/ocm-arohcpci01-2tn2f6qu0skm7r1cd4s42o9104lkbkll-m9o9d5o9m2l7k4w: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1",
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-27 09:10	PR #7514	2115008102154637312

VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...

full failure pattern:

VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: no customer resource ID -> live ReadDesire HostedCluster ->

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-9xwk6zxxz7m6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x2680bac6c1a0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-9xwk6zxxz7m6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-9xwk6zxxz7m6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
        err: <*verifiers.pollObservationError | 0x2680bac6c180>{
            observation: <*errors.errorString | 0x2680b95b3860>{
                s: "no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
            },
            interruption: <*fmt.wrapError | 0x2680bac6c160>{
                msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j1816448-mgmt-2\"; let ns = \"ocm-arohcpci01-2tmt6fqch3btra8cfaickj0t85ovobt1-d4f6x6g2r2p5t9d\"; let start = datetime(2026-10-27T03:57:03.7385865Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                err: <*errors.Error | 0x2680b9c4c190>{
                    Op: 1,
                    Kind: 7,
                    Err: <*fmt.wrapError | 0x2680bac6c140>{
                        msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j1816448-mgmt-2\"; let ns = \"ocm-arohcpci01-2tmt6fqch3btra8cfaickj0t85ovobt1-d4f6x6g2r2p5t9d\"; let start = datetime(2026-10-27T03:57:03.7385865Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                        err: <*url.Error | 0x2680bb4eaab0>{
                            Op: "Post",
                            URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                            Err: <context.deadlineExceededError>{},
                        },
                    },
                    restErrMsg: nil,
                    decoded: nil,
                    permanent: false,
                    inner: nil,
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-9xwk6zxxz7m6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x2680bac6c1a0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-9xwk6zxxz7m6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-9xwk6zxxz7m6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
        err: <*verifiers.pollObservationError | 0x2680bac6c180>{
            observation: <*errors.errorString | 0x2680b95b3860>{
                s: "no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
            },
            interruption: <*fmt.wrapError | 0x2680bac6c160>{
                msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j1816448-mgmt-2\"; let ns = \"ocm-arohcpci01-2tmt6fqch3btra8cfaickj0t85ovobt1-d4f6x6g2r2p5t9d\"; let start = datetime(2026-10-27T03:57:03.7385865Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                err: <*errors.Error | 0x2680b9c4c190>{
                    Op: 1,
                    Kind: 7,
                    Err: <*fmt.wrapError | 0x2680bac6c140>{
                        msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j1816448-mgmt-2\"; let ns = \"ocm-arohcpci01-2tmt6fqch3btra8cfaickj0t85ovobt1-d4f6x6g2r2p5t9d\"; let start = datetime(2026-10-27T03:57:03.7385865Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                        err: <*url.Error | 0x2680bb4eaab0>{
                            Op: "Post",
                            URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                            Err: <context.deadlineExceededError>{},
                        },
                    },
                    restErrMsg: nil,
                    decoded: nil,
                    permanent: false,
                    inner: nil,
                },
            },
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-27 03:15	PR #6364	2114918973651816448

VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...

full failure pattern:

VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-spr5zsf6h48p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x910473e2000>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-spr5zsf6h48p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/westus3/ci01-j5971328-mgmt-2/ocm-arohcpci01-2tne6fmsv15g1h9j1vkcqdkjq8bfs0ld-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-spr5zsf6h48p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/westus3/ci01-j5971328-mgmt-2/ocm-arohcpci01-2tne6fmsv15g1h9j1vkcqdkjq8bfs0ld-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        err: <*errors.errorString | 0x91047780d20>{
            s: "dev/westus3/ci01-j5971328-mgmt-2/ocm-arohcpci01-2tne6fmsv15g1h9j1vkcqdkjq8bfs0ld-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-spr5zsf6h48p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x910473e2000>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-spr5zsf6h48p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/westus3/ci01-j5971328-mgmt-2/ocm-arohcpci01-2tne6fmsv15g1h9j1vkcqdkjq8bfs0ld-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-spr5zsf6h48p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/westus3/ci01-j5971328-mgmt-2/ocm-arohcpci01-2tne6fmsv15g1h9j1vkcqdkjq8bfs0ld-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        err: <*errors.errorString | 0x91047780d20>{
            s: "dev/westus3/ci01-j5971328-mgmt-2/ocm-arohcpci01-2tne6fmsv15g1h9j1vkcqdkjq8bfs0ld-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-27 22:25	PR #7787	2115208311295971328

VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...

full failure pattern:

VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/westus3/ci01-j7087616-mgmt-2/ocm-arohcpci01-<id>: expected all

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-95ktkd6tw5dw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x2555ba2ee820>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-95ktkd6tw5dw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/westus3/ci01-j7087616-mgmt-2/ocm-arohcpci01-2tn11bbfhosljf97vunimgc88lhlamlu-u6q6o6h9w2m2h0f: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-95ktkd6tw5dw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/westus3/ci01-j7087616-mgmt-2/ocm-arohcpci01-2tn11bbfhosljf97vunimgc88lhlamlu-u6q6o6h9w2m2h0f: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        err: <*verifiers.pollObservationError | 0x2555ba2ee800>{
            observation: <*errors.errorString | 0x2555b9ce7460>{
                s: "dev/westus3/ci01-j7087616-mgmt-2/ocm-arohcpci01-2tn11bbfhosljf97vunimgc88lhlamlu-u6q6o6h9w2m2h0f: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
            },
            interruption: <*fmt.wrapError | 0x2555ba2ee7e0>{
                msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j7087616-mgmt-2\"; let ns = \"ocm-arohcpci01-2tn11bbfhosljf97vunimgc88lhlamlu-u6q6o6h9w2m2h0f\"; let start = datetime(2026-10-27T08:19:08.7156738Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                err: <*errors.Error | 0x2555b95a8370>{
                    Op: 1,
                    Kind: 7,
                    Err: <*fmt.wrapError | 0x2555ba2ee7c0>{
                        msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j7087616-mgmt-2\"; let ns = \"ocm-arohcpci01-2tn11bbfhosljf97vunimgc88lhlamlu-u6q6o6h9w2m2h0f\"; let start = datetime(2026-10-27T08:19:08.7156738Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                        err: <*url.Error | 0x2555baea86f0>{
                            Op: "Post",
                            URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                            Err: <context.deadlineExceededError>{},
                        },
                    },
                    restErrMsg: nil,
                    decoded: nil,
                    permanent: false,
                    inner: nil,
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-95ktkd6tw5dw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x2555ba2ee820>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-95ktkd6tw5dw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/westus3/ci01-j7087616-mgmt-2/ocm-arohcpci01-2tn11bbfhosljf97vunimgc88lhlamlu-u6q6o6h9w2m2h0f: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-95ktkd6tw5dw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/westus3/ci01-j7087616-mgmt-2/ocm-arohcpci01-2tn11bbfhosljf97vunimgc88lhlamlu-u6q6o6h9w2m2h0f: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        err: <*verifiers.pollObservationError | 0x2555ba2ee800>{
            observation: <*errors.errorString | 0x2555b9ce7460>{
                s: "dev/westus3/ci01-j7087616-mgmt-2/ocm-arohcpci01-2tn11bbfhosljf97vunimgc88lhlamlu-u6q6o6h9w2m2h0f: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
            },
            interruption: <*fmt.wrapError | 0x2555ba2ee7e0>{
                msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j7087616-mgmt-2\"; let ns = \"ocm-arohcpci01-2tn11bbfhosljf97vunimgc88lhlamlu-u6q6o6h9w2m2h0f\"; let start = datetime(2026-10-27T08:19:08.7156738Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                err: <*errors.Error | 0x2555b95a8370>{
                    Op: 1,
                    Kind: 7,
                    Err: <*fmt.wrapError | 0x2555ba2ee7c0>{
                        msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j7087616-mgmt-2\"; let ns = \"ocm-arohcpci01-2tn11bbfhosljf97vunimgc88lhlamlu-u6q6o6h9w2m2h0f\"; let start = datetime(2026-10-27T08:19:08.7156738Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                        err: <*url.Error | 0x2555baea86f0>{
                            Op: "Post",
                            URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                            Err: <context.deadlineExceededError>{},
                        },
                    },
                    restErrMsg: nil,
                    decoded: nil,
                    permanent: false,
                    inner: nil,
                },
            },
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-27 07:29	PR #7598	2114982798447087616

[events] e2e-csi-disk-<id> PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node <node> (reason=ProvisioningFailed, count=...

full failure pattern:

[events] e2e-csi-disk-<id> PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node <node> (reason=ProvisioningFailed, count=8, age=7m53s)

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-94pk4p"
Unexpected error:
    <*errors.joinError | 0x380579046d8>: 
    VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.001s: pod e2e-csi-disk-fj4vw/e2e-csi-pod phase is "Pending", expected Succeeded; PVC is Bound
    [storageclass] name=managed-csi provisioner=disk.csi.azure.com
    [pvc] e2e-csi-disk-fj4vw/e2e-csi-pvc phase=Bound volumeName=pvc-9120227e-fe48-4e13-8ec5-8e95f21af7be storageClass=managed-csi
    [pod] e2e-csi-disk-fj4vw/e2e-csi-pod phase=Pending reason= message=
    [pod] condition type=PodReadyToStartContainers status=False reason= message=
    [pod] condition type=Initialized status=True reason= message=
    [pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]
    [pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]
    [pod] condition type=PodScheduled status=True reason= message=
    [pod] container volume-check ready=false restartCount=0
    [pod]   waiting reason=ContainerCreating message=
    [events] e2e-csi-disk-fj4vw Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-fj4vw/e2e-csi-pod to s8m2i8d5f5w8n1d-np-94pk4p-q9sxk-psnks (reason=Scheduled, count=0, age=5m21s)
    [events] e2e-csi-disk-fj4vw Pod/e2e-csi-pod: AttachVolume.Attach failed for volume "pvc-9120227e-fe48-4e13-8ec5-8e95f21af7be" : rpc error: code = Internal desc = could not get disk lun for volume /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-mdwfnk5tx96c-candidate-94pk4p--managed/providers/Microsoft.Compute/disks/pvc-9120227e-fe48-4e13-8ec5-8e95f21af7be: getNodeVMSet: failed to check the node s8m2i8d5f5w8n1d-np-94pk4p-q9sxk-psnks management type: getter function of nonVmssUniformNodesCache: failed to list vms in the resource group rg-candidate-4-23-mdwfnk5tx96c-candidate-94pk4p--managed: throttled due to too many requests (reason=FailedAttachVolume, count=9, age=1m58s)
    [events] e2e-csi-disk-fj4vw PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m1s)
    [events] e2e-csi-disk-fj4vw PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=1, age=10m0s)
    [events] e2e-csi-disk-fj4vw PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim "e2e-csi-disk-fj4vw/e2e-csi-pvc" (reason=Provisioning, count=9, age=5m45s)
    [events] e2e-csi-disk-fj4vw PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node s8m2i8d5f5w8n1d-np-94pk4p-q9sxk-psnks (reason=ProvisioningFailed, count=8, age=7m53s)
    [events] e2e-csi-disk-fj4vw PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=17, age=5m36s)
    [events] e2e-csi-disk-fj4vw PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-9120227e-fe48-4e13-8ec5-8e95f21af7be (reason=ProvisioningSucceeded, count=1, age=5m22s)
    {
        errs: [
            <*fmt.wrapError | 0x38056a72920>{
                msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.001s: pod e2e-csi-disk-fj4vw/e2e-csi-pod phase is \"Pending\", expected Succeeded; PVC is Bound\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-fj4vw/e2e-csi-pvc phase=Bound volumeName=pvc-9120227e-fe48-4e13-8ec5-8e95f21af7be storageClass=managed-csi\n[pod] e2e-csi-disk-fj4vw/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodReadyToStartContainers status=False reason= message=\n[pod] condition type=Initialized status=True reason= message=\n[pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=PodScheduled status=True reason= message=\n[pod] container volume-check ready=false restartCount=0\n[pod]   waiting reason=ContainerCreating message=\n[events] e2e-csi-disk-fj4vw Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-fj4vw/e2e-csi-pod to s8m2i8d5f5w8n1d-np-94pk4p-q9sxk-psnks (reason=Scheduled, count=0, age=5m21s)\n[events] e2e-csi-disk-fj4vw Pod/e2e-csi-pod: AttachVolume.Attach failed for volume \"pvc-9120227e-fe48-4e13-8ec5-8e95f21af7be\" : rpc error: code = Internal desc = could not get disk lun for volume /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-mdwfnk5tx96c-candidate-94pk4p--managed/providers/Microsoft.Compute/disks/pvc-9120227e-fe48-4e13-8ec5-8e95f21af7be: getNodeVMSet: failed to check the node s8m2i8d5f5w8n1d-np-94pk4p-q9sxk-psnks management type: getter function of nonVmssUniformNodesCache: failed to list vms in the resource group rg-candidate-4-23-mdwfnk5tx96c-candidate-94pk4p--managed: throttled due to too many requests (reason=FailedAttachVolume, count=9, age=1m58s)\n[events] e2e-csi-disk-fj4vw PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m1s)\n[events] e2e-csi-disk-fj4vw PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=1, age=10m0s)\n[events] e2e-csi-disk-fj4vw PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-fj4vw/e2e-csi-pvc\" (reason=Provisioning, count=9, age=5m45s)\n[events] e2e-csi-disk-fj4vw PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node s8m2i8d5f5w8n1d-np-94pk4p-q9sxk-psnks (reason=ProvisioningFailed, count=8, age=7m53s)\n[events] e2e-csi-disk-fj4vw PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=17, age=5m36s)\n[events] e2e-csi-disk-fj4vw PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-9120227e-fe48-4e13-8ec5-8e95f21af7be (reason=ProvisioningSucceeded, count=1, age=5m22s)",
                err: <*fmt.wrapError | 0x38056fdc300>{
                    msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.001s: pod e2e-csi-disk-fj4vw/e2e-csi-pod phase is \"Pending\", expected Succeeded; PVC is Bound\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-fj4vw/e2e-csi-pvc phase=Bound volumeName=pvc-9120227e-fe48-4e13-8ec5-8e95f21af7be storageClass=managed-csi\n[pod] e2e-csi-disk-fj4vw/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodReadyToStartContainers status=False reason= message=\n[pod] condition type=Initialized status=True reason= message=\n[pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=PodScheduled status=True reason= message=\n[pod] container volume-check ready=false restartCount=0\n[pod]   waiting reason=ContainerCreating message=\n[events] e2e-csi-disk-fj4vw Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-fj4vw/e2e-csi-pod to s8m2i8d5f5w8n1d-np-94pk4p-q9sxk-psnks (reason=Scheduled, count=0, age=5m21s)\n[events] e2e-csi-disk-fj4vw Pod/e2e-csi-pod: AttachVolume.Attach failed for volume \"pvc-9120227e-fe48-4e13-8ec5-8e95f21af7be\" : rpc error: code = Internal desc = could not get disk lun for volume /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-mdwfnk5tx96c-candidate-94pk4p--managed/providers/Microsoft.Compute/disks/pvc-9120227e-fe48-4e13-8ec5-8e95f21af7be: getNodeVMSet: failed to check the node s8m2i8d5f5w8n1d-np-94pk4p-q9sxk-psnks management type: getter function of nonVmssUniformNodesCache: failed to list vms in the resource group rg-candidate-4-23-mdwfnk5tx96c-candidate-94pk4p--managed: throttled due to too many requests (reason=FailedAttachVolume, count=9, age=1m58s)\n[events] e2e-csi-disk-fj4vw PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m1s)\n[events] e2e-csi-disk-fj4vw PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=1, age=10m0s)\n[events] e2e-csi-disk-fj4vw PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-fj4vw/e2e-csi-pvc\" (reason=Provisioning, count=9, age=5m45s)\n[events] e2e-csi-disk-fj4vw PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node s8m2i8d5f5w8n1d-np-94pk4p-q9sxk-psnks (reason=ProvisioningFailed, count=8, age=7m53s)\n[events] e2e-csi-disk-fj4vw PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=17, age=5m36s)\n[events] e2e-csi-disk-fj4vw PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-9120227e-fe48-4e13-8ec5-8e95f21af7be (reason=ProvisioningSucceeded, count=1, age=5m22s)",
                    err: <*verifiers.pollObservationError | 0x38056fb6b80>{
                        observation: <*errors.errorString | 0x380577528d0>{
                            s: "pod e2e-csi-disk-fj4vw/e2e-csi-pod phase is \"Pending\", expected Succeeded; PVC is Bound",
                        },
                        interruption: <*fmt.wrapError | 0x38056fb6b20>{
                            msg: "failed to get PVC e2e-csi-disk-fj4vw/e2e-csi-pvc: context deadline exceeded",
                            err: <context.deadlineExceededError>{},
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-94pk4p"
Unexpected error:
    <*errors.joinError | 0x380579046d8>: 
    VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.001s: pod e2e-csi-disk-fj4vw/e2e-csi-pod phase is "Pending", expected Succeeded; PVC is Bound
    [storageclass] name=managed-csi provisioner=disk.csi.azure.com
    [pvc] e2e-csi-disk-fj4vw/e2e-csi-pvc phase=Bound volumeName=pvc-9120227e-fe48-4e13-8ec5-8e95f21af7be storageClass=managed-csi
    [pod] e2e-csi-disk-fj4vw/e2e-csi-pod phase=Pending reason= message=
    [pod] condition type=PodReadyToStartContainers status=False reason= message=
    [pod] condition type=Initialized status=True reason= message=
    [pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]
    [pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]
    [pod] condition type=PodScheduled status=True reason= message=
    [pod] container volume-check ready=false restartCount=0
    [pod]   waiting reason=ContainerCreating message=
    [events] e2e-csi-disk-fj4vw Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-fj4vw/e2e-csi-pod to s8m2i8d5f5w8n1d-np-94pk4p-q9sxk-psnks (reason=Scheduled, count=0, age=5m21s)
    [events] e2e-csi-disk-fj4vw Pod/e2e-csi-pod: AttachVolume.Attach failed for volume "pvc-9120227e-fe48-4e13-8ec5-8e95f21af7be" : rpc error: code = Internal desc = could not get disk lun for volume /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-mdwfnk5tx96c-candidate-94pk4p--managed/providers/Microsoft.Compute/disks/pvc-9120227e-fe48-4e13-8ec5-8e95f21af7be: getNodeVMSet: failed to check the node s8m2i8d5f5w8n1d-np-94pk4p-q9sxk-psnks management type: getter function of nonVmssUniformNodesCache: failed to list vms in the resource group rg-candidate-4-23-mdwfnk5tx96c-candidate-94pk4p--managed: throttled due to too many requests (reason=FailedAttachVolume, count=9, age=1m58s)
    [events] e2e-csi-disk-fj4vw PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m1s)
    [events] e2e-csi-disk-fj4vw PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=1, age=10m0s)
    [events] e2e-csi-disk-fj4vw PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim "e2e-csi-disk-fj4vw/e2e-csi-pvc" (reason=Provisioning, count=9, age=5m45s)
    [events] e2e-csi-disk-fj4vw PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node s8m2i8d5f5w8n1d-np-94pk4p-q9sxk-psnks (reason=ProvisioningFailed, count=8, age=7m53s)
    [events] e2e-csi-disk-fj4vw PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=17, age=5m36s)
    [events] e2e-csi-disk-fj4vw PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-9120227e-fe48-4e13-8ec5-8e95f21af7be (reason=ProvisioningSucceeded, count=1, age=5m22s)
    {
        errs: [
            <*fmt.wrapError | 0x38056a72920>{
                msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.001s: pod e2e-csi-disk-fj4vw/e2e-csi-pod phase is \"Pending\", expected Succeeded; PVC is Bound\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-fj4vw/e2e-csi-pvc phase=Bound volumeName=pvc-9120227e-fe48-4e13-8ec5-8e95f21af7be storageClass=managed-csi\n[pod] e2e-csi-disk-fj4vw/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodReadyToStartContainers status=False reason= message=\n[pod] condition type=Initialized status=True reason= message=\n[pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=PodScheduled status=True reason= message=\n[pod] container volume-check ready=false restartCount=0\n[pod]   waiting reason=ContainerCreating message=\n[events] e2e-csi-disk-fj4vw Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-fj4vw/e2e-csi-pod to s8m2i8d5f5w8n1d-np-94pk4p-q9sxk-psnks (reason=Scheduled, count=0, age=5m21s)\n[events] e2e-csi-disk-fj4vw Pod/e2e-csi-pod: AttachVolume.Attach failed for volume \"pvc-9120227e-fe48-4e13-8ec5-8e95f21af7be\" : rpc error: code = Internal desc = could not get disk lun for volume /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-mdwfnk5tx96c-candidate-94pk4p--managed/providers/Microsoft.Compute/disks/pvc-9120227e-fe48-4e13-8ec5-8e95f21af7be: getNodeVMSet: failed to check the node s8m2i8d5f5w8n1d-np-94pk4p-q9sxk-psnks management type: getter function of nonVmssUniformNodesCache: failed to list vms in the resource group rg-candidate-4-23-mdwfnk5tx96c-candidate-94pk4p--managed: throttled due to too many requests (reason=FailedAttachVolume, count=9, age=1m58s)\n[events] e2e-csi-disk-fj4vw PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m1s)\n[events] e2e-csi-disk-fj4vw PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=1, age=10m0s)\n[events] e2e-csi-disk-fj4vw PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-fj4vw/e2e-csi-pvc\" (reason=Provisioning, count=9, age=5m45s)\n[events] e2e-csi-disk-fj4vw PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node s8m2i8d5f5w8n1d-np-94pk4p-q9sxk-psnks (reason=ProvisioningFailed, count=8, age=7m53s)\n[events] e2e-csi-disk-fj4vw PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=17, age=5m36s)\n[events] e2e-csi-disk-fj4vw PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-9120227e-fe48-4e13-8ec5-8e95f21af7be (reason=ProvisioningSucceeded, count=1, age=5m22s)",
                err: <*fmt.wrapError | 0x38056fdc300>{
                    msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.001s: pod e2e-csi-disk-fj4vw/e2e-csi-pod phase is \"Pending\", expected Succeeded; PVC is Bound\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-fj4vw/e2e-csi-pvc phase=Bound volumeName=pvc-9120227e-fe48-4e13-8ec5-8e95f21af7be storageClass=managed-csi\n[pod] e2e-csi-disk-fj4vw/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodReadyToStartContainers status=False reason= message=\n[pod] condition type=Initialized status=True reason= message=\n[pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=PodScheduled status=True reason= message=\n[pod] container volume-check ready=false restartCount=0\n[pod]   waiting reason=ContainerCreating message=\n[events] e2e-csi-disk-fj4vw Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-fj4vw/e2e-csi-pod to s8m2i8d5f5w8n1d-np-94pk4p-q9sxk-psnks (reason=Scheduled, count=0, age=5m21s)\n[events] e2e-csi-disk-fj4vw Pod/e2e-csi-pod: AttachVolume.Attach failed for volume \"pvc-9120227e-fe48-4e13-8ec5-8e95f21af7be\" : rpc error: code = Internal desc = could not get disk lun for volume /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-mdwfnk5tx96c-candidate-94pk4p--managed/providers/Microsoft.Compute/disks/pvc-9120227e-fe48-4e13-8ec5-8e95f21af7be: getNodeVMSet: failed to check the node s8m2i8d5f5w8n1d-np-94pk4p-q9sxk-psnks management type: getter function of nonVmssUniformNodesCache: failed to list vms in the resource group rg-candidate-4-23-mdwfnk5tx96c-candidate-94pk4p--managed: throttled due to too many requests (reason=FailedAttachVolume, count=9, age=1m58s)\n[events] e2e-csi-disk-fj4vw PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m1s)\n[events] e2e-csi-disk-fj4vw PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=1, age=10m0s)\n[events] e2e-csi-disk-fj4vw PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-fj4vw/e2e-csi-pvc\" (reason=Provisioning, count=9, age=5m45s)\n[events] e2e-csi-disk-fj4vw PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node s8m2i8d5f5w8n1d-np-94pk4p-q9sxk-psnks (reason=ProvisioningFailed, count=8, age=7m53s)\n[events] e2e-csi-disk-fj4vw PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=17, age=5m36s)\n[events] e2e-csi-disk-fj4vw PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-9120227e-fe48-4e13-8ec5-8e95f21af7be (reason=ProvisioningSucceeded, count=1, age=5m22s)",
                    err: <*verifiers.pollObservationError | 0x38056fb6b80>{
                        observation: <*errors.errorString | 0x380577528d0>{
                            s: "pod e2e-csi-disk-fj4vw/e2e-csi-pod phase is \"Pending\", expected Succeeded; PVC is Bound",
                        },
                        interruption: <*fmt.wrapError | 0x38056fb6b20>{
                            msg: "failed to get PVC e2e-csi-disk-fj4vw/e2e-csi-pvc: context deadline exceeded",
                            err: <context.deadlineExceededError>{},
                        },
                    },
                },
            },
        ],
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-27 14:36	PR #7705	2115090302321561600

[events] e2e-csi-disk-<id> PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=3, age=10s)

full failure pattern:

[events] e2e-csi-disk-<id> PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=3, age=10s)

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-dj4ptk"
Unexpected error:
    <*errors.joinError | 0xd5290aa85b8>: 
    VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.001s: PVC e2e-csi-disk-8ql64/e2e-csi-pvc phase is "Pending", expected Bound; pod phase is "Pending"
    [storageclass] name=managed-csi provisioner=disk.csi.azure.com
    [pvc] e2e-csi-disk-8ql64/e2e-csi-pvc phase=Pending volumeName= storageClass=managed-csi
    [pod] e2e-csi-disk-8ql64/e2e-csi-pod phase=Pending reason= message=
    [pod] condition type=PodScheduled status=False reason=SchedulerError message=running PreBind plugin "VolumeBinding": binding volumes: context deadline exceeded
    [events] e2e-csi-disk-8ql64 Pod/e2e-csi-pod: running PreBind plugin "VolumeBinding": binding volumes: context deadline exceeded (reason=FailedScheduling, count=0, age=0s)
    [events] e2e-csi-disk-8ql64 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim "e2e-csi-disk-8ql64/e2e-csi-pvc" (reason=Provisioning, count=11, age=10s)
    [events] e2e-csi-disk-8ql64 PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node c9x9s6y3p8f9j2i-np-dj4ptk-xtdjf-gwssm (reason=ProvisioningFailed, count=7, age=8m44s)
    [events] e2e-csi-disk-8ql64 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=3m32s)
    [events] e2e-csi-disk-8ql64 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=3, age=10s)
    {
        errs: [
            <*fmt.wrapError | 0xd5290a963a0>{
                msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.001s: PVC e2e-csi-disk-8ql64/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-8ql64/e2e-csi-pvc phase=Pending volumeName= storageClass=managed-csi\n[pod] e2e-csi-disk-8ql64/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodScheduled status=False reason=SchedulerError message=running PreBind plugin \"VolumeBinding\": binding volumes: context deadline exceeded\n[events] e2e-csi-disk-8ql64 Pod/e2e-csi-pod: running PreBind plugin \"VolumeBinding\": binding volumes: context deadline exceeded (reason=FailedScheduling, count=0, age=0s)\n[events] e2e-csi-disk-8ql64 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-8ql64/e2e-csi-pvc\" (reason=Provisioning, count=11, age=10s)\n[events] e2e-csi-disk-8ql64 PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node c9x9s6y3p8f9j2i-np-dj4ptk-xtdjf-gwssm (reason=ProvisioningFailed, count=7, age=8m44s)\n[events] e2e-csi-disk-8ql64 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=3m32s)\n[events] e2e-csi-disk-8ql64 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=3, age=10s)",
                err: <*fmt.wrapError | 0xd5290356ce0>{
                    msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.001s: PVC e2e-csi-disk-8ql64/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-8ql64/e2e-csi-pvc phase=Pending volumeName= storageClass=managed-csi\n[pod] e2e-csi-disk-8ql64/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodScheduled status=False reason=SchedulerError message=running PreBind plugin \"VolumeBinding\": binding volumes: context deadline exceeded\n[events] e2e-csi-disk-8ql64 Pod/e2e-csi-pod: running PreBind plugin \"VolumeBinding\": binding volumes: context deadline exceeded (reason=FailedScheduling, count=0, age=0s)\n[events] e2e-csi-disk-8ql64 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-8ql64/e2e-csi-pvc\" (reason=Provisioning, count=11, age=10s)\n[events] e2e-csi-disk-8ql64 PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node c9x9s6y3p8f9j2i-np-dj4ptk-xtdjf-gwssm (reason=ProvisioningFailed, count=7, age=8m44s)\n[events] e2e-csi-disk-8ql64 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=3m32s)\n[events] e2e-csi-disk-8ql64 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=3, age=10s)",
                    err: <*verifiers.pollObservationError | 0xd5290356540>{
                        observation: <*errors.errorString | 0xd52904125d0>{
                            s: "PVC e2e-csi-disk-8ql64/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"",
                        },
                        interruption: <*fmt.wrapError | 0xd5290356520>{
                            msg: "failed to get PVC e2e-csi-disk-8ql64/e2e-csi-pvc: context deadline exceeded",
                            err: <context.deadlineExceededError>{},
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-dj4ptk"
Unexpected error:
    <*errors.joinError | 0xd5290aa85b8>: 
    VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.001s: PVC e2e-csi-disk-8ql64/e2e-csi-pvc phase is "Pending", expected Bound; pod phase is "Pending"
    [storageclass] name=managed-csi provisioner=disk.csi.azure.com
    [pvc] e2e-csi-disk-8ql64/e2e-csi-pvc phase=Pending volumeName= storageClass=managed-csi
    [pod] e2e-csi-disk-8ql64/e2e-csi-pod phase=Pending reason= message=
    [pod] condition type=PodScheduled status=False reason=SchedulerError message=running PreBind plugin "VolumeBinding": binding volumes: context deadline exceeded
    [events] e2e-csi-disk-8ql64 Pod/e2e-csi-pod: running PreBind plugin "VolumeBinding": binding volumes: context deadline exceeded (reason=FailedScheduling, count=0, age=0s)
    [events] e2e-csi-disk-8ql64 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim "e2e-csi-disk-8ql64/e2e-csi-pvc" (reason=Provisioning, count=11, age=10s)
    [events] e2e-csi-disk-8ql64 PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node c9x9s6y3p8f9j2i-np-dj4ptk-xtdjf-gwssm (reason=ProvisioningFailed, count=7, age=8m44s)
    [events] e2e-csi-disk-8ql64 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=3m32s)
    [events] e2e-csi-disk-8ql64 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=3, age=10s)
    {
        errs: [
            <*fmt.wrapError | 0xd5290a963a0>{
                msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.001s: PVC e2e-csi-disk-8ql64/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-8ql64/e2e-csi-pvc phase=Pending volumeName= storageClass=managed-csi\n[pod] e2e-csi-disk-8ql64/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodScheduled status=False reason=SchedulerError message=running PreBind plugin \"VolumeBinding\": binding volumes: context deadline exceeded\n[events] e2e-csi-disk-8ql64 Pod/e2e-csi-pod: running PreBind plugin \"VolumeBinding\": binding volumes: context deadline exceeded (reason=FailedScheduling, count=0, age=0s)\n[events] e2e-csi-disk-8ql64 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-8ql64/e2e-csi-pvc\" (reason=Provisioning, count=11, age=10s)\n[events] e2e-csi-disk-8ql64 PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node c9x9s6y3p8f9j2i-np-dj4ptk-xtdjf-gwssm (reason=ProvisioningFailed, count=7, age=8m44s)\n[events] e2e-csi-disk-8ql64 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=3m32s)\n[events] e2e-csi-disk-8ql64 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=3, age=10s)",
                err: <*fmt.wrapError | 0xd5290356ce0>{
                    msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.001s: PVC e2e-csi-disk-8ql64/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-8ql64/e2e-csi-pvc phase=Pending volumeName= storageClass=managed-csi\n[pod] e2e-csi-disk-8ql64/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodScheduled status=False reason=SchedulerError message=running PreBind plugin \"VolumeBinding\": binding volumes: context deadline exceeded\n[events] e2e-csi-disk-8ql64 Pod/e2e-csi-pod: running PreBind plugin \"VolumeBinding\": binding volumes: context deadline exceeded (reason=FailedScheduling, count=0, age=0s)\n[events] e2e-csi-disk-8ql64 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-8ql64/e2e-csi-pvc\" (reason=Provisioning, count=11, age=10s)\n[events] e2e-csi-disk-8ql64 PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node c9x9s6y3p8f9j2i-np-dj4ptk-xtdjf-gwssm (reason=ProvisioningFailed, count=7, age=8m44s)\n[events] e2e-csi-disk-8ql64 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=3m32s)\n[events] e2e-csi-disk-8ql64 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=3, age=10s)",
                    err: <*verifiers.pollObservationError | 0xd5290356540>{
                        observation: <*errors.errorString | 0xd52904125d0>{
                            s: "PVC e2e-csi-disk-8ql64/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"",
                        },
                        interruption: <*fmt.wrapError | 0xd5290356520>{
                            msg: "failed to get PVC e2e-csi-disk-8ql64/e2e-csi-pvc: context deadline exceeded",
                            err: <context.deadlineExceededError>{},
                        },
                    },
                },
            },
        ],
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-27 20:12	PR #6081	2115174942373842944

[events] e2e-csi-disk-<id> PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=4, age=2m24s)

full failure pattern:

[events] e2e-csi-disk-<id> PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=4, age=2m24s)

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-6lzc2b"
Unexpected error:
    <*errors.joinError | 0x263fb059e4c8>: 
    VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0s: PVC e2e-csi-disk-sfxr9/e2e-csi-pvc phase is "Pending", expected Bound; pod phase is "Pending"
    [storageclass] name=managed-csi provisioner=disk.csi.azure.com
    [pvc] e2e-csi-disk-sfxr9/e2e-csi-pvc phase=Pending volumeName= storageClass=managed-csi
    [pod] e2e-csi-disk-sfxr9/e2e-csi-pod phase=Pending reason= message=
    [pod] condition type=PodScheduled status=False reason=SchedulerError message=running PreBind plugin "VolumeBinding": binding volumes: context deadline exceeded
    [events] e2e-csi-disk-sfxr9 Pod/e2e-csi-pod: running PreBind plugin "VolumeBinding": binding volumes: context deadline exceeded (reason=FailedScheduling, count=0, age=0s)
    [events] e2e-csi-disk-sfxr9 PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)
    [events] e2e-csi-disk-sfxr9 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=3, age=9m57s)
    [events] e2e-csi-disk-sfxr9 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim "e2e-csi-disk-sfxr9/e2e-csi-pvc" (reason=Provisioning, count=10, age=2m54s)
    [events] e2e-csi-disk-sfxr9 PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node n2t7h9a5i9i7j1l-np-6lzc2b-lzl2m-btqtb (reason=ProvisioningFailed, count=6, age=9m29s)
    [events] e2e-csi-disk-sfxr9 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=3m19s)
    [events] e2e-csi-disk-sfxr9 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=4, age=2m24s)
    {
        errs: [
            <*fmt.wrapError | 0x263fb0784660>{
                msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0s: PVC e2e-csi-disk-sfxr9/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-sfxr9/e2e-csi-pvc phase=Pending volumeName= storageClass=managed-csi\n[pod] e2e-csi-disk-sfxr9/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodScheduled status=False reason=SchedulerError message=running PreBind plugin \"VolumeBinding\": binding volumes: context deadline exceeded\n[events] e2e-csi-disk-sfxr9 Pod/e2e-csi-pod: running PreBind plugin \"VolumeBinding\": binding volumes: context deadline exceeded (reason=FailedScheduling, count=0, age=0s)\n[events] e2e-csi-disk-sfxr9 PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)\n[events] e2e-csi-disk-sfxr9 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=3, age=9m57s)\n[events] e2e-csi-disk-sfxr9 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-sfxr9/e2e-csi-pvc\" (reason=Provisioning, count=10, age=2m54s)\n[events] e2e-csi-disk-sfxr9 PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node n2t7h9a5i9i7j1l-np-6lzc2b-lzl2m-btqtb (reason=ProvisioningFailed, count=6, age=9m29s)\n[events] e2e-csi-disk-sfxr9 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=3m19s)\n[events] e2e-csi-disk-sfxr9 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=4, age=2m24s)",
                err: <*fmt.wrapError | 0x263fb09fc440>{
                    msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0s: PVC e2e-csi-disk-sfxr9/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-sfxr9/e2e-csi-pvc phase=Pending volumeName= storageClass=managed-csi\n[pod] e2e-csi-disk-sfxr9/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodScheduled status=False reason=SchedulerError message=running PreBind plugin \"VolumeBinding\": binding volumes: context deadline exceeded\n[events] e2e-csi-disk-sfxr9 Pod/e2e-csi-pod: running PreBind plugin \"VolumeBinding\": binding volumes: context deadline exceeded (reason=FailedScheduling, count=0, age=0s)\n[events] e2e-csi-disk-sfxr9 PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)\n[events] e2e-csi-disk-sfxr9 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=3, age=9m57s)\n[events] e2e-csi-disk-sfxr9 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-sfxr9/e2e-csi-pvc\" (reason=Provisioning, count=10, age=2m54s)\n[events] e2e-csi-disk-sfxr9 PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node n2t7h9a5i9i7j1l-np-6lzc2b-lzl2m-btqtb (reason=ProvisioningFailed, count=6, age=9m29s)\n[events] e2e-csi-disk-sfxr9 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=3m19s)\n[events] e2e-csi-disk-sfxr9 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=4, age=2m24s)",
                    err: <*errors.errorString | 0x263fb0ff0b50>{
                        s: "PVC e2e-csi-disk-sfxr9/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"",
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-6lzc2b"
Unexpected error:
    <*errors.joinError | 0x263fb059e4c8>: 
    VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0s: PVC e2e-csi-disk-sfxr9/e2e-csi-pvc phase is "Pending", expected Bound; pod phase is "Pending"
    [storageclass] name=managed-csi provisioner=disk.csi.azure.com
    [pvc] e2e-csi-disk-sfxr9/e2e-csi-pvc phase=Pending volumeName= storageClass=managed-csi
    [pod] e2e-csi-disk-sfxr9/e2e-csi-pod phase=Pending reason= message=
    [pod] condition type=PodScheduled status=False reason=SchedulerError message=running PreBind plugin "VolumeBinding": binding volumes: context deadline exceeded
    [events] e2e-csi-disk-sfxr9 Pod/e2e-csi-pod: running PreBind plugin "VolumeBinding": binding volumes: context deadline exceeded (reason=FailedScheduling, count=0, age=0s)
    [events] e2e-csi-disk-sfxr9 PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)
    [events] e2e-csi-disk-sfxr9 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=3, age=9m57s)
    [events] e2e-csi-disk-sfxr9 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim "e2e-csi-disk-sfxr9/e2e-csi-pvc" (reason=Provisioning, count=10, age=2m54s)
    [events] e2e-csi-disk-sfxr9 PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node n2t7h9a5i9i7j1l-np-6lzc2b-lzl2m-btqtb (reason=ProvisioningFailed, count=6, age=9m29s)
    [events] e2e-csi-disk-sfxr9 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=3m19s)
    [events] e2e-csi-disk-sfxr9 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=4, age=2m24s)
    {
        errs: [
            <*fmt.wrapError | 0x263fb0784660>{
                msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0s: PVC e2e-csi-disk-sfxr9/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-sfxr9/e2e-csi-pvc phase=Pending volumeName= storageClass=managed-csi\n[pod] e2e-csi-disk-sfxr9/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodScheduled status=False reason=SchedulerError message=running PreBind plugin \"VolumeBinding\": binding volumes: context deadline exceeded\n[events] e2e-csi-disk-sfxr9 Pod/e2e-csi-pod: running PreBind plugin \"VolumeBinding\": binding volumes: context deadline exceeded (reason=FailedScheduling, count=0, age=0s)\n[events] e2e-csi-disk-sfxr9 PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)\n[events] e2e-csi-disk-sfxr9 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=3, age=9m57s)\n[events] e2e-csi-disk-sfxr9 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-sfxr9/e2e-csi-pvc\" (reason=Provisioning, count=10, age=2m54s)\n[events] e2e-csi-disk-sfxr9 PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node n2t7h9a5i9i7j1l-np-6lzc2b-lzl2m-btqtb (reason=ProvisioningFailed, count=6, age=9m29s)\n[events] e2e-csi-disk-sfxr9 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=3m19s)\n[events] e2e-csi-disk-sfxr9 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=4, age=2m24s)",
                err: <*fmt.wrapError | 0x263fb09fc440>{
                    msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0s: PVC e2e-csi-disk-sfxr9/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-sfxr9/e2e-csi-pvc phase=Pending volumeName= storageClass=managed-csi\n[pod] e2e-csi-disk-sfxr9/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodScheduled status=False reason=SchedulerError message=running PreBind plugin \"VolumeBinding\": binding volumes: context deadline exceeded\n[events] e2e-csi-disk-sfxr9 Pod/e2e-csi-pod: running PreBind plugin \"VolumeBinding\": binding volumes: context deadline exceeded (reason=FailedScheduling, count=0, age=0s)\n[events] e2e-csi-disk-sfxr9 PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)\n[events] e2e-csi-disk-sfxr9 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=3, age=9m57s)\n[events] e2e-csi-disk-sfxr9 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-sfxr9/e2e-csi-pvc\" (reason=Provisioning, count=10, age=2m54s)\n[events] e2e-csi-disk-sfxr9 PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node n2t7h9a5i9i7j1l-np-6lzc2b-lzl2m-btqtb (reason=ProvisioningFailed, count=6, age=9m29s)\n[events] e2e-csi-disk-sfxr9 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=3m19s)\n[events] e2e-csi-disk-sfxr9 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=4, age=2m24s)",
                    err: <*errors.errorString | 0x263fb0ff0b50>{
                        s: "PVC e2e-csi-disk-sfxr9/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"",
                    },
                },
            },
        ],
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-27 17:02	PR #6081	2115127036694499328

error running Shell Step, failed to execute shell command: {"level":"INFO","msg":"starting cluster reconciliation","component":"aks-cluster-create","subscriptionID":"<redacted-id>...

full failure pattern:

error running Shell Step, failed to execute shell command: {"level":"INFO","msg":"starting cluster reconciliation","component":"aks-cluster-create","subscriptionID":"<redacted-id>","resourceGroup":"hcp-underlay-ci01-j5601792-mgmt-1","clusterName":"ci01-j560179

Signal:

provision

none

Full failure examples (1)

time=2026-10-27T15:30:56.670Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-10-27T16:00:56.129Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 err="stamp 1: error running Shell Step, failed to execute shell command: {\"time\":\"2026-10-27T15:30:56.681953143Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5601792-mgmt-1\",\"clusterName\":\"ci01-j5601792-mgmt-1\",\"region\":\"centralus\"}\n{\"time\":\"2026-10-27T15:30:58.731109745Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5601792-mgmt-1\",\"clusterName\":\"ci01-j5601792-mgmt-1\",\"region\":\"centralus\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-10-27T15:30:58.873423526Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5601792-mgmt-1\",\"clusterName\":\"ci01-j5601792-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n signal: killed"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster-create (stamp=1)	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-27 14:50	PR #7670	2115093743005601792

failed to roll out Helm release: resource Deployment/maestro/maestro not ready. status: InProgress, message: Available: 0/2 context deadline exceeded

full failure pattern:

failed to roll out Helm release: resource Deployment/maestro/maestro not ready. status: InProgress, message: Available: 0/2 context deadline exceeded

Signal:

provision

none

Full failure examples (1)

time=2026-10-28T00:24:38.589Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n"
time=2026-10-28T00:24:39.214Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy values="map[azure:map[clientId:80c01b85-e07b-47ba-b2c6-d25569ff69d4 cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] backend:map[namespace:aro-hcp serviceAccount:backend] broker:map[host:arohcp-ci01-maestro-j8270720.westus3-1.ts.eventgrid.azure.net port:8883] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] credsKeyVault:map[name:aro-hcp-dev-svc-kv secret:maestro-server-j8270720] database:map[azureDb:false containerizedDbImage:docker.io/library/postgres:14.2 containerizedDbPvcCapacity:512Mi debug:false host:maestro-db maxOpenConnections:50 name:maestro password:TheBlurstOfTimes port:5432 user:maestro-server] deployment:map[replicas:2 resources:map[limits:map[memory:unlimited] requests:map[cpu:200m memory:512Mi]] zoneCount:3] fleet:map[namespace:fleet serviceAccount:fleet] image:map[digest:sha256:aba8ed3c2d6e8b0f6a5e17b2f65dba996c5cf673b73a35cc550521873b08c231 registry:arohcpsvcdev.azurecr.io repository:redhat-user-workloads/maestro-rhtap-tenant/maestro/maestro] istio:map[restrictIngress:true] maestro:map[glog_v:2 grpcBindPort:8090 healthCheckBindPort:8083 httpBindPort:8000 httpReadTimeout:60s httpWriteTimeout:60s labelMetricsInclusionDuration:168h serviceAccount:maestro] monitoringApiGroup:monitoring.coreos.com pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:redhat-user-workloads/maestro-rhtap-tenant/maestro/maestro:pull workloadIdentityClientId:d99261bb-4b6f-46d6-96d4-bae9c86ef284 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX]]"
time=2026-10-28T00:24:39.214Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-10-28T00:24:39.557Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy namespace=maestro
time=2026-10-28T00:24:39.898Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy dryRun=false
time=2026-10-28T00:24:39.898Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-10-28T00:24:40.025Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-10-28T00:34:46.993Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy err="resource Deployment/maestro/maestro not ready. status: InProgress, message: Available: 0/2\ncontext deadline exceeded"
time=2026-10-28T00:34:46.994Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy err="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource Deployment/maestro/maestro not ready. status: InProgress, message: Available: 0/2\ncontext deadline exceeded"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Maestro.Server/service/deploy	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-27 23:23	PR #5244	2115222887018270720

missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=10944, acm-agent=0

full failure pattern:

missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=10944, acm-agent=0

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 602.696s.
Expected success, but got an error:
    <*errors.errorString | 0x283ddd596200>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=10944, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=10944, acm-agent=0",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 602.696s.
Expected success, but got an error:
    <*errors.errorString | 0x283ddd596200>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=10944, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=10944, acm-agent=0",
    }

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-27 10:19	PR #7662	2115025503944445952

missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=8412, acm-agent=0

full failure pattern:

missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=8412, acm-agent=0

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 600.002s.
Expected success, but got an error:
    <*errors.errorString | 0x211f21252720>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=8412, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=8412, acm-agent=0",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 600.002s.
Expected success, but got an error:
    <*errors.errorString | 0x211f21252720>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=8412, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=8412, acm-agent=0",
    }

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-27 06:56	PR #7662	2114974410921218048

alert [svc] KubePodNotReady fired

full failure pattern:

alert [svc] KubePodNotReady fired

Signal:

alert

none

Full failure examples (3)

alert fired 21 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-28T00:28:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="aks-istio-system", pod="azure-service-mesh-istio-cni-addon-node-ck87t", region="westus3", severity="warning"
  Description: Pod aks-istio-system/azure-service-mesh-istio-cni-addon-node-ck87t has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-28T00:28:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="aks-secrets-store-csi-driver-n68qf", region="westus3", severity="warning"
  Description: Pod kube-system/aks-secrets-store-csi-driver-n68qf has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-28T00:28:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="ama-metrics-node-bx6p7", region="westus3", severity="warning"
  Description: Pod kube-system/ama-metrics-node-bx6p7 has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-28T00:28:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="ama-metrics-node-6fggw", region="westus3", severity="warning"
  Description: Pod kube-system/ama-metrics-node-6fggw has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-10-28T00:28:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="eraser-aks-system-42508117-vmss000002-249kw", region="westus3", severity="warning"
  Description: Pod kube-system/eraser-aks-system-42508117-vmss000002-249kw has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-10-28T00:28:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="aks-istio-system", pod="azure-service-mesh-istio-cni-addon-node-f2bs8", region="westus3", severity="warning"
  Description: Pod aks-istio-system/azure-service-mesh-istio-cni-addon-node-f2bs8 has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Fired (not resolved)
  Started:     2026-10-28T00:28:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="aks-secrets-store-csi-driver-pwgk9", region="westus3", severity="warning"
  Description: Pod kube-system/aks-secrets-store-csi-driver-pwgk9 has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Fired (not resolved)
  Started:     2026-10-28T00:28:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="ama-metrics-node-sb9gv", region="westus3", severity="warning"
  Description: Pod kube-system/ama-metrics-node-sb9gv has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Fired (not resolved)
  Started:     2026-10-28T00:28:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="eraser-aks-system-42508117-vmss000001-z2292", region="westus3", severity="warning"
  Description: Pod kube-system/eraser-aks-system-42508117-vmss000001-z2292 has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Fired (not resolved)
  Started:     2026-10-28T00:28:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="aks-secrets-store-csi-driver-sgh48", region="westus3", severity="warning"
  Description: Pod kube-system/aks-secrets-store-csi-driver-sgh48 has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Fired (not resolved)
  Started:     2026-10-28T00:28:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="aks-istio-system", pod="azure-service-mesh-istio-cni-addon-node-jkt84", region="westus3", severity="warning"
  Description: Pod aks-istio-system/azure-service-mesh-istio-cni-addon-node-jkt84 has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Fired (not resolved)
  Started:     2026-10-28T00:28:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="eraser-aks-system-42508117-vmss000000-b6bzq", region="westus3", severity="warning"
  Description: Pod kube-system/eraser-aks-system-42508117-vmss000000-b6bzq has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Fired (not resolved)
  Started:     2026-10-28T00:28:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="hubble-relay-58594b5dd7-p7rcj", region="westus3", severity="warning"
  Description: Pod kube-system/hubble-relay-58594b5dd7-p7rcj has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Fired (not resolved)
  Started:     2026-10-28T00:30:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-8696d46cbc-gcq7v", region="westus3", severity="warning"
  Description: Pod sessiongate/sessiongate-8696d46cbc-gcq7v has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Fired (not resolved)
  Started:     2026-10-28T00:30:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-8696d46cbc-7tl64", region="westus3", severity="warning"
  Description: Pod sessiongate/sessiongate-8696d46cbc-7tl64 has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Fired (not resolved)
  Started:     2026-10-28T00:30:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="maestro", pod="maestro-6bc6657c6b-nc5dx", region="westus3", severity="warning"
  Description: Pod maestro/maestro-6bc6657c6b-nc5dx has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Fired (not resolved)
  Started:     2026-10-28T00:30:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-567b46758d-6wzh8", region="westus3", severity="warning"
  Description: Pod clusters-service/clusters-service-567b46758d-6wzh8 has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Fired (not resolved)
  Started:     2026-10-28T00:30:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-567b46758d-rsmsf", region="westus3", severity="warning"
  Description: Pod clusters-service/clusters-service-567b46758d-rsmsf has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Fired (not resolved)
  Started:     2026-10-28T00:30:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="ocm-cs-db-7594d5ccfc-vd2b7", region="westus3", severity="warning"
  Description: Pod clusters-service/ocm-cs-db-7594d5ccfc-vd2b7 has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Fired (not resolved)
  Started:     2026-10-28T00:30:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-567b46758d-hzqfx", region="westus3", severity="warning"
  Description: Pod clusters-service/clusters-service-567b46758d-hzqfx has been in a non-ready state for longer than 5 minutes.
Firing 21:
  State:       Fired (not resolved)
  Started:     2026-10-28T00:30:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="maestro", pod="maestro-6bc6657c6b-stp2q", region="westus3", severity="warning"
  Description: Pod maestro/maestro-6bc6657c6b-stp2q has been in a non-ready state for longer than 5 minutes.

alert fired 8 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-27T15:12:43Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9877504-mgmt-2", component="kubernetes-infrastructure", namespace="multicluster-engine", pod="ocm-controller-859484fc49-qhntv", region="westus3", severity="warning"
  Description: Pod multicluster-engine/ocm-controller-859484fc49-qhntv has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-27T15:12:43Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9877504-mgmt-2", component="kubernetes-infrastructure", namespace="multicluster-engine", pod="cluster-manager-c5d6c587f-fd8rf", region="westus3", severity="warning"
  Description: Pod multicluster-engine/cluster-manager-c5d6c587f-fd8rf has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-27T15:12:43Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9877504-mgmt-2", component="kubernetes-infrastructure", namespace="multicluster-engine", pod="multicluster-engine-operator-79c768d748-lg8vd", region="westus3", severity="warning"
  Description: Pod multicluster-engine/multicluster-engine-operator-79c768d748-lg8vd has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-27T15:12:43Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9877504-mgmt-2", component="kubernetes-infrastructure", namespace="kube-applier", pod="kube-applier-8475cd7c99-dfzv8", region="westus3", severity="warning"
  Description: Pod kube-applier/kube-applier-8475cd7c99-dfzv8 has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-10-27T15:12:43Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9877504-mgmt-2", component="kubernetes-infrastructure", namespace="multicluster-engine", pod="ocm-webhook-68676899fd-mklv5", region="westus3", severity="warning"
  Description: Pod multicluster-engine/ocm-webhook-68676899fd-mklv5 has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-10-27T15:12:43Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9877504-mgmt-2", component="kubernetes-infrastructure", namespace="mgmt-agent", pod="mgmt-agent-6dc8498b6f-ntd7x", region="westus3", severity="warning"
  Description: Pod mgmt-agent/mgmt-agent-6dc8498b6f-ntd7x has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Fired (not resolved)
  Started:     2026-10-27T15:12:43Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9877504-mgmt-2", component="kubernetes-infrastructure", namespace="multicluster-engine", pod="managedcluster-import-controller-v2-6cb46794f7-bzb25", region="westus3", severity="warning"
  Description: Pod multicluster-engine/managedcluster-import-controller-v2-6cb46794f7-bzb25 has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Fired (not resolved)
  Started:     2026-10-27T15:12:43Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9877504-mgmt-2", component="kubernetes-infrastructure", namespace="secret-sync-controller", pod="secrets-store-sync-controller-manager-7f4b9bc5df-9hg5c", region="westus3", severity="warning"
  Description: Pod secret-sync-controller/secrets-store-sync-controller-manager-7f4b9bc5df-9hg5c has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-27T14:08:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8622848-mgmt-2", component="kubernetes-infrastructure", namespace="open-cluster-management-agent-addon", pod="klusterlet-addon-workmgr-58f4f6fb5b-wt5ns", region="canadacentral", severity="warning"
  Description: Pod open-cluster-management-agent-addon/klusterlet-addon-workmgr-58f4f6fb5b-wt5ns has been in a non-ready state for longer than 5 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert KubePodNotReady does not fire	6

Affected runs (6)

Date (UTC)	Associated PR	Prow job
2026-10-27 23:23	PR #5244	2115222887018270720
2026-10-27 14:16	PR #6823	2115085218409877504
2026-10-27 13:15	PR #7615	2115069760348622848
2026-10-27 13:14	PR #6954	2115069745341403136
2026-10-27 12:49	PR #6823	2115063427364818944
2026-10-27 06:14	PR #7615	2114964013585534976

alert [svc] BackendControllerQueueDepthHigh fired

full failure pattern:

alert [svc] BackendControllerQueueDepthHigh fired

Signal:

alert

none

Full failure examples (3)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-27T20:20:10Z
  Ended:       2026-10-27T20:29:09Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j7981312-svc", component="backend", name="csstatedump", region="westus3", severity="warning"
  Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-27T15:28:06Z
  Ended:       2026-10-27T15:45:04Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4184192-svc", component="backend", name="csstatedump", region="centralus", severity="warning"
  Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 2:
  State:       Resolved
  Started:     2026-10-27T15:28:06Z
  Ended:       2026-10-27T15:45:04Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4184192-svc", component="backend", name="clusterresources", region="centralus", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-27T12:39:32Z
  Ended:       2026-10-27T12:54:28Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j5195264-svc", component="backend", name="csstatedump", region="westus3", severity="warning"
  Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire	5

Affected runs (5)

Date (UTC)	Associated PR	Prow job
2026-10-27 18:59	PR #7436	2115159601987981312
2026-10-27 14:23	PR #7776	2115086995314184192
2026-10-27 11:17	PR #7769	2115040287725195264
2026-10-27 03:30	PR #7767	2114922665838907392
2026-10-27 01:13	PR #7615	2114888160881152000

alert [hcp] KubePodNotReady fired

full failure pattern:

alert [hcp] KubePodNotReady fired

Signal:

alert

none

Full failure examples (3)

alert fired 2 time(s) (1 unknown, 1 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-27T23:40:09Z
  Ended:       2026-10-27T23:57:09Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1933056-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tneda066vj6kdgdjqh3ub42gu0lckg9-ephemeral-disk", pod="etcd-0", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tneda066vj6kdgdjqh3ub42gu0lckg9-ephemeral-disk/etcd-0 has been in a non-ready state for longer than 5 minutes.

alert fired 13 time(s) (12 unknown, 1 known)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-27T15:44:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1476864-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tn75avtj8u4futjt2om8gigffgddu35-ingress-cert-le", pod="openshift-apiserver-5fbf969589-rwzww", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tn75avtj8u4futjt2om8gigffgddu35-ingress-cert-le/openshift-apiserver-5fbf969589-rwzww has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-27T15:44:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1476864-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tn74o7qhu6go8i6sdevboooi4cvm5o8-o9s7l1c0u6a0c2b", pod="openshift-apiserver-7c75f5b8c-869mc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tn74o7qhu6go8i6sdevboooi4cvm5o8-o9s7l1c0u6a0c2b/openshift-apiserver-7c75f5b8c-869mc has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-27T15:45:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1476864-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tn75q9b8j8s3323r8hbovnl3fa3ah93-a8h6t7r0k6i7a1w", pod="openshift-apiserver-6cf589c77c-9cn7v", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tn75q9b8j8s3323r8hbovnl3fa3ah93-a8h6t7r0k6i7a1w/openshift-apiserver-6cf589c77c-9cn7v has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-27T15:45:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1476864-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tn74rifog60es9rvt2t2mfrpvs36h9e-t2y6g6x3g9p9s5t", pod="openshift-apiserver-9c9cfdf89-62q4b", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tn74rifog60es9rvt2t2mfrpvs36h9e-t2y6g6x3g9p9s5t/openshift-apiserver-9c9cfdf89-62q4b has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-10-27T15:45:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1476864-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tn757i0j3orr665p0dcqg487jcvp9a9-q1n4k4z8u2y9x7u", pod="openshift-apiserver-7bbccfbcd8-nj4zh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tn757i0j3orr665p0dcqg487jcvp9a9-q1n4k4z8u2y9x7u/openshift-apiserver-7bbccfbcd8-nj4zh has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-10-27T15:45:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1476864-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tn75637s7ka00dgh3b0o5nl473kscrl-h2n8s4a0l1b6e9x", pod="openshift-apiserver-7b79cf8d6d-klztg", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tn75637s7ka00dgh3b0o5nl473kscrl-h2n8s4a0l1b6e9x/openshift-apiserver-7b79cf8d6d-klztg has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Fired (not resolved)
  Started:     2026-10-27T15:45:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1476864-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tn7522f0rjmsl54mp9vum53bfeis22t-x1e5u3y9j2l5a2u", pod="openshift-apiserver-7d75b894c6-n4cmx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tn7522f0rjmsl54mp9vum53bfeis22t-x1e5u3y9j2l5a2u/openshift-apiserver-7d75b894c6-n4cmx has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Fired (not resolved)
  Started:     2026-10-27T15:46:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1476864-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tn75ndq1oh2entq0cv9ulrug5jog853-ingress-cert-le", pod="openshift-apiserver-764b69ffcb-7ngbg", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tn75ndq1oh2entq0cv9ulrug5jog853-ingress-cert-le/openshift-apiserver-764b69ffcb-7ngbg has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Fired (not resolved)
  Started:     2026-10-27T15:46:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1476864-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tn7cebrepv6hitmognobm7c9cl2clnd-o5c0v9b6h0c5z0a", pod="cluster-policy-controller-698bfd9c65-v6xdj", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tn7cebrepv6hitmognobm7c9cl2clnd-o5c0v9b6h0c5z0a/cluster-policy-controller-698bfd9c65-v6xdj has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Fired (not resolved)
  Started:     2026-10-27T15:46:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1476864-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tn75h85al80vo23t362mjtc3qgd04dp-c1b6v6c2x7c8p8f", pod="openshift-apiserver-57bf4dc46f-h7mj5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tn75h85al80vo23t362mjtc3qgd04dp-c1b6v6c2x7c8p8f/openshift-apiserver-57bf4dc46f-h7mj5 has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Fired (not resolved)
  Started:     2026-10-27T15:46:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1476864-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tn759i4ns4omt60u9uecngkrn5mj618-u8q4i0h7g0i1r6k", pod="openshift-apiserver-59b86f6c67-76jwk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tn759i4ns4omt60u9uecngkrn5mj618-u8q4i0h7g0i1r6k/openshift-apiserver-59b86f6c67-76jwk has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Fired (not resolved)
  Started:     2026-10-27T15:47:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1476864-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tn7615u8n39pf897sksqb5f7beb9d2u-b0o3o9t7e5r0u9g", pod="openshift-apiserver-77c77b89b8-kr9pl", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tn7615u8n39pf897sksqb5f7beb9d2u-b0o3o9t7e5r0u9g/openshift-apiserver-77c77b89b8-kr9pl has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-27T12:07:21Z
  Ended:       2026-10-27T12:13:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1657088-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tn3rdsscm4njq5vp9j636vrehjrq243-d8n2y7y8n3i9x3v", pod="cluster-policy-controller-fd98b88bf-g2g4g", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tn3rdsscm4njq5vp9j636vrehjrq243-d8n2y7y8n3i9x3v/cluster-policy-controller-fd98b88bf-g2g4g has been in a non-ready state for longer than 5 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubePodNotReady does not fire	5

Affected runs (5)

Date (UTC)	Associated PR	Prow job
2026-10-27 22:20	PR #6924	2115207073061933056
2026-10-27 14:26	PR #7730	2115087800381476864
2026-10-27 10:43	PR #7441	2115031737661657088
2026-10-27 10:31	PR #7598	2115028667607289856
2026-10-27 09:55	PR #7730	2115019495591383040

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift

Signal:

e2e

INT, PROD

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:98]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0x1620db330380>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-gfmmkfqxc5x6" to finish creating: GET https://rp.j8626944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb0ecfe0-4f44-41f1-923e-4f3520584861
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb0ecfe0-4f44-41f1-923e-4f3520584861",
      "name": "cb0ecfe0-4f44-41f1-923e-4f3520584861",
      "status": "Failed",
      "startTime": "2026-10-27T21:18:30.659898808Z",
      "endTime": "2026-10-27T21:47:37.431864682Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8626944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb0ecfe0-4f44-41f1-923e-4f3520584861
            correlation ID: 63848e13-133e-4fce-9179-580109e2b00e
            client request ID: a25acb69-a9bf-4e2d-8af8-06447ee114b4
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:98]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0x1620db330380>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-gfmmkfqxc5x6" to finish creating: GET https://rp.j8626944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb0ecfe0-4f44-41f1-923e-4f3520584861
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb0ecfe0-4f44-41f1-923e-4f3520584861",
      "name": "cb0ecfe0-4f44-41f1-923e-4f3520584861",
      "status": "Failed",
      "startTime": "2026-10-27T21:18:30.659898808Z",
      "endTime": "2026-10-27T21:47:37.431864682Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8626944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb0ecfe0-4f44-41f1-923e-4f3520584861
            correlation ID: 63848e13-133e-4fce-9179-580109e2b00e
            client request ID: a25acb69-a9bf-4e2d-8af8-06447ee114b4
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0x10bb5e13c7a0>: 
    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-6gzvpqk2566b" to finish creating: GET https://rp.j4445952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f17d3b43-5248-42e8-9f44-dbdc6056affd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f17d3b43-5248-42e8-9f44-dbdc6056affd",
      "name": "f17d3b43-5248-42e8-9f44-dbdc6056affd",
      "status": "Failed",
      "startTime": "2026-10-27T11:25:28.537630373Z",
      "endTime": "2026-10-27T11:54:32.228087726Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j4445952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f17d3b43-5248-42e8-9f44-dbdc6056affd
            correlation ID: bb86b9bc-b93e-4515-9de7-3f81a42f7835
            client request ID: b2ae866e-4769-4b2c-931c-8d15a52ca9ea
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0x10bb5e13c7a0>: 
    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-6gzvpqk2566b" to finish creating: GET https://rp.j4445952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f17d3b43-5248-42e8-9f44-dbdc6056affd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f17d3b43-5248-42e8-9f44-dbdc6056affd",
      "name": "f17d3b43-5248-42e8-9f44-dbdc6056affd",
      "status": "Failed",
      "startTime": "2026-10-27T11:25:28.537630373Z",
      "endTime": "2026-10-27T11:54:32.228087726Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j4445952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f17d3b43-5248-42e8-9f44-dbdc6056affd
            correlation ID: bb86b9bc-b93e-4515-9de7-3f81a42f7835
            client request ID: b2ae866e-4769-4b2c-931c-8d15a52ca9ea
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/metrics_amw_hcp_ksm.go:87]: failed to create HCP cluster "ksm-hcp-metrics"
Unexpected error:
    <*fmt.wrapError | 0x250d4d0585c0>: 
    failed to create HCP cluster ksm-hcp-metrics: failed waiting for cluster="ksm-hcp-metrics" in resourcegroup="hcp-metrics-8dj5fcfzk684" to finish creating: GET https://rp.j1152000.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d5ecd395-85d4-4167-8c0b-d86ab4676770
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d5ecd395-85d4-4167-8c0b-d86ab4676770",
      "name": "d5ecd395-85d4-4167-8c0b-d86ab4676770",
      "status": "Failed",
      "startTime": "2026-10-27T01:51:23.29970675Z",
      "endTime": "2026-10-27T02:20:31.588657474Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1152000.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d5ecd395-85d4-4167-8c0b-d86ab4676770
            correlation ID: b2ed5fd1-aced-406c-9da8-d4a42aa70231
            client request ID: 2be74574-9856-48bd-8976-fff7fa4252d8
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/metrics_amw_hcp_ksm.go:87]: failed to create HCP cluster "ksm-hcp-metrics"
Unexpected error:
    <*fmt.wrapError | 0x250d4d0585c0>: 
    failed to create HCP cluster ksm-hcp-metrics: failed waiting for cluster="ksm-hcp-metrics" in resourcegroup="hcp-metrics-8dj5fcfzk684" to finish creating: GET https://rp.j1152000.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d5ecd395-85d4-4167-8c0b-d86ab4676770
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d5ecd395-85d4-4167-8c0b-d86ab4676770",
      "name": "d5ecd395-85d4-4167-8c0b-d86ab4676770",
      "status": "Failed",
      "startTime": "2026-10-27T01:51:23.29970675Z",
      "endTime": "2026-10-27T02:20:31.588657474Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1152000.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d5ecd395-85d4-4167-8c0b-d86ab4676770
            correlation ID: b2ed5fd1-aced-406c-9da8-d4a42aa70231
            client request ID: 2be74574-9856-48bd-8976-fff7fa4252d8
occurred

Contributing tests (3)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update node pool labels and taints	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering want metrics from HCP to be present in Azure Monitor Workspace	1

Affected runs (3)

Date (UTC)	Associated PR	Prow job
2026-10-27 20:13	PR #7705	2115175056668626944
2026-10-27 10:19	PR #7662	2115025503944445952
2026-10-27 01:13	PR #7615	2114888160881152000

alert [hcp] KubeDeploymentReplicasMismatch fired

full failure pattern:

alert [hcp] KubeDeploymentReplicasMismatch fired

Signal:

alert

none

Full failure examples (2)

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-27T21:53:20Z
  Ended:       2026-10-27T22:05:18Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8626944-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.113:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8626944/providers/microsoft.monitor/accounts/hcps-j8626944", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnce9sp5mvj0fek8lfnjcm2ej2a4s3g-ea-cluster", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-r9m2r", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tnce9sp5mvj0fek8lfnjcm2ej2a4s3g-ea-cluster/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-27T21:53:20Z
  Ended:       2026-10-27T22:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8626944-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.113:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8626944/providers/microsoft.monitor/accounts/hcps-j8626944", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnce9sp5mvj0fek8lfnjcm2ej2a4s3g-ea-cluster", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-r9m2r", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tnce9sp5mvj0fek8lfnjcm2ej2a4s3g-ea-cluster/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-27T02:25:40Z
  Ended:       2026-10-27T02:37:39Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1152000-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.174:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1152000/providers/microsoft.monitor/accounts/hcps-j1152000", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tmrb70fk8q8hg0qepkmrjljph9ejlcl-ksm-hcp-metrics", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-zp65c", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tmrb70fk8q8hg0qepkmrjljph9ejlcl-ksm-hcp-metrics/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-27T02:26:36Z
  Ended:       2026-10-27T02:37:39Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1152000-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.174:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1152000/providers/microsoft.monitor/accounts/hcps-j1152000", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tmrb70fk8q8hg0qepkmrjljph9ejlcl-ksm-hcp-metrics", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-zp65c", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tmrb70fk8q8hg0qepkmrjljph9ejlcl-ksm-hcp-metrics/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-27 20:13	PR #7705	2115175056668626944
2026-10-27 01:13	PR #7615	2114888160881152000

alert [hcp] KubePodCrashLooping fired

full failure pattern:

alert [hcp] KubePodCrashLooping fired

Signal:

alert

none

Full failure examples (2)

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-27T21:41:15Z
  Ended:       2026-10-27T22:09:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j8626944-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.113:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8626944/providers/microsoft.monitor/accounts/hcps-j8626944", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnce9sp5mvj0fek8lfnjcm2ej2a4s3g-ea-cluster", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="f3fe228f-1ce0-4bce-a91a-29cdfd52c071"
  Description: Pod ocm-arohcpci01-2tnce9sp5mvj0fek8lfnjcm2ej2a4s3g-ea-cluster/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-27T21:41:15Z
  Ended:       2026-10-27T22:09:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j8626944-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.113:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8626944/providers/microsoft.monitor/accounts/hcps-j8626944", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnce9sp5mvj0fek8lfnjcm2ej2a4s3g-ea-cluster", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="f3fe228f-1ce0-4bce-a91a-29cdfd52c071"
  Description: Pod ocm-arohcpci01-2tnce9sp5mvj0fek8lfnjcm2ej2a4s3g-ea-cluster/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-27T02:13:36Z
  Ended:       2026-10-27T02:41:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1152000-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.174:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1152000/providers/microsoft.monitor/accounts/hcps-j1152000", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tmrb70fk8q8hg0qepkmrjljph9ejlcl-ksm-hcp-metrics", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="0cbd6fe7-cf8c-4402-9cca-95a6f5bddfe0"
  Description: Pod ocm-arohcpci01-2tmrb70fk8q8hg0qepkmrjljph9ejlcl-ksm-hcp-metrics/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-27T02:14:36Z
  Ended:       2026-10-27T02:41:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1152000-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.174:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1152000/providers/microsoft.monitor/accounts/hcps-j1152000", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tmrb70fk8q8hg0qepkmrjljph9ejlcl-ksm-hcp-metrics", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="0cbd6fe7-cf8c-4402-9cca-95a6f5bddfe0"
  Description: Pod ocm-arohcpci01-2tmrb70fk8q8hg0qepkmrjljph9ejlcl-ksm-hcp-metrics/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubePodCrashLooping does not fire	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-27 20:13	PR #7705	2115175056668626944
2026-10-27 01:13	PR #7615	2114888160881152000

alert [hcp] KubeStatefulSetReplicasMismatch fired

full failure pattern:

alert [hcp] KubeStatefulSetReplicasMismatch fired

Signal:

alert

none

Full failure examples (2)

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-27T21:38:22Z
  Ended:       2026-10-27T22:04:19Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j8626944-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.113:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8626944/providers/microsoft.monitor/accounts/hcps-j8626944", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnce9sp5mvj0fek8lfnjcm2ej2a4s3g-ea-cluster", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-r9m2r", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2tnce9sp5mvj0fek8lfnjcm2ej2a4s3g-ea-cluster/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-27T21:39:21Z
  Ended:       2026-10-27T22:04:19Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j8626944-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.113:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8626944/providers/microsoft.monitor/accounts/hcps-j8626944", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnce9sp5mvj0fek8lfnjcm2ej2a4s3g-ea-cluster", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-r9m2r", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2tnce9sp5mvj0fek8lfnjcm2ej2a4s3g-ea-cluster/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-27T02:11:40Z
  Ended:       2026-10-27T02:36:38Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j1152000-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.174:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1152000/providers/microsoft.monitor/accounts/hcps-j1152000", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tmrb70fk8q8hg0qepkmrjljph9ejlcl-ksm-hcp-metrics", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-zp65c", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2tmrb70fk8q8hg0qepkmrjljph9ejlcl-ksm-hcp-metrics/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-27T02:11:40Z
  Ended:       2026-10-27T02:37:39Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j1152000-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.174:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1152000/providers/microsoft.monitor/accounts/hcps-j1152000", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tmrb70fk8q8hg0qepkmrjljph9ejlcl-ksm-hcp-metrics", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-zp65c", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2tmrb70fk8q8hg0qepkmrjljph9ejlcl-ksm-hcp-metrics/etcd has not matched the expected number of replicas for longer than 15 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubeStatefulSetReplicasMismatch does not fire	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-27 20:13	PR #7705	2115175056668626944
2026-10-27 01:13	PR #7615	2114888160881152000

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift

Signal:

e2e

PROD

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:96]: failed to create node pool "nodepool-128GiB" with 128GiB osDisk
Unexpected error:
    <*fmt.wrapError | 0x3d4aea46a060>: 
    failed to create NodePool nodepool-128GiB: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-85bttqbvwzdg" to finish creating: GET https://rp.j7635968.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7c4e2323-4cc6-4875-b033-1cc27f1dc9c9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7c4e2323-4cc6-4875-b033-1cc27f1dc9c9",
      "name": "7c4e2323-4cc6-4875-b033-1cc27f1dc9c9",
      "status": "Failed",
      "startTime": "2026-10-27T17:47:01.433391882Z",
      "endTime": "2026-10-27T18:06:02.631484435Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine b3p4u5e0h1k8y7f-nodepool-128gib-6n7h6-rw99j: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-85bttqbvwzdg--managed/providers/Microsoft.Compute/virtualMachines/b3p4u5e0h1k8y7f-nodepool-128gib-6n7h6-rw99j to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine b3p4u5e0h1k8y7f-nodepool-128gib-6n7h6-rw99j: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-85bttqbvwzdg--managed/providers/Microsoft.Compute/virtualMachines/b3p4u5e0h1k8y7f-nodepool-128gib-6n7h6-rw99j to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7635968.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7c4e2323-4cc6-4875-b033-1cc27f1dc9c9
            correlation ID: b503cd11-01c7-4b49-bd8f-b11bdfa64e3b
            client request ID: 8e981bdf-718f-4374-bc9d-9fe858083611
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:96]: failed to create node pool "nodepool-128GiB" with 128GiB osDisk
Unexpected error:
    <*fmt.wrapError | 0x3d4aea46a060>: 
    failed to create NodePool nodepool-128GiB: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-85bttqbvwzdg" to finish creating: GET https://rp.j7635968.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7c4e2323-4cc6-4875-b033-1cc27f1dc9c9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7c4e2323-4cc6-4875-b033-1cc27f1dc9c9",
      "name": "7c4e2323-4cc6-4875-b033-1cc27f1dc9c9",
      "status": "Failed",
      "startTime": "2026-10-27T17:47:01.433391882Z",
      "endTime": "2026-10-27T18:06:02.631484435Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine b3p4u5e0h1k8y7f-nodepool-128gib-6n7h6-rw99j: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-85bttqbvwzdg--managed/providers/Microsoft.Compute/virtualMachines/b3p4u5e0h1k8y7f-nodepool-128gib-6n7h6-rw99j to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine b3p4u5e0h1k8y7f-nodepool-128gib-6n7h6-rw99j: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-85bttqbvwzdg--managed/providers/Microsoft.Compute/virtualMachines/b3p4u5e0h1k8y7f-nodepool-128gib-6n7h6-rw99j to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7635968.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7c4e2323-4cc6-4875-b033-1cc27f1dc9c9
            correlation ID: b503cd11-01c7-4b49-bd8f-b11bdfa64e3b
            client request ID: 8e981bdf-718f-4374-bc9d-9fe858083611
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and custom node pool osDisk size	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-27 16:44	PR #7615	2115122591927635968

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet; provider Microsoft.RedHatOpenShift

Signal:

e2e

PROD

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x19f0aa4bea20>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-wj8x9hp7bqx5" to finish creating: GET https://rp.j8023424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6a73d0d2-8fd5-4af1-adbb-f385ed243d20
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6a73d0d2-8fd5-4af1-adbb-f385ed243d20",
      "name": "6a73d0d2-8fd5-4af1-adbb-f385ed243d20",
      "status": "Failed",
      "startTime": "2026-10-27T08:09:38.679552409Z",
      "endTime": "2026-10-27T08:28:40.674204457Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8023424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6a73d0d2-8fd5-4af1-adbb-f385ed243d20
            correlation ID: 51afa422-8d59-4334-91a3-bc90ade1f885
            client request ID: ebb6ee5a-68e9-481e-8fab-49a3ebe79568
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x19f0aa4bea20>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-wj8x9hp7bqx5" to finish creating: GET https://rp.j8023424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6a73d0d2-8fd5-4af1-adbb-f385ed243d20
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6a73d0d2-8fd5-4af1-adbb-f385ed243d20",
      "name": "6a73d0d2-8fd5-4af1-adbb-f385ed243d20",
      "status": "Failed",
      "startTime": "2026-10-27T08:09:38.679552409Z",
      "endTime": "2026-10-27T08:28:40.674204457Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8023424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6a73d0d2-8fd5-4af1-adbb-f385ed243d20
            correlation ID: 51afa422-8d59-4334-91a3-bc90ade1f885
            client request ID: ebb6ee5a-68e9-481e-8fab-49a3ebe79568
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-27 06:59	PR #7768	2114975256258023424

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x26ee15ea6540>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-lz9smvq6kbwg" to finish creating: GET https://rp.j1933056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ea7a0229-5912-411c-9d71-ed6121f1fa93
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ea7a0229-5912-411c-9d71-ed6121f1fa93",
      "name": "ea7a0229-5912-411c-9d71-ed6121f1fa93",
      "status": "Failed",
      "startTime": "2026-10-27T23:28:59.117849892Z",
      "endTime": "2026-10-27T23:48:07.430527402Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine l6k1q7q7h9e3k0z-np-1-thnn4-9rrd8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-lz9smvq6kbwg--managed/providers/Microsoft.Compute/virtualMachines/l6k1q7q7h9e3k0z-np-1-thnn4-9rrd8 to exist\nMachine l6k1q7q7h9e3k0z-np-1-thnn4-w82sd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-lz9smvq6kbwg--managed/providers/Microsoft.Compute/virtualMachines/l6k1q7q7h9e3k0z-np-1-thnn4-w82sd to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine l6k1q7q7h9e3k0z-np-1-thnn4-9rrd8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-lz9smvq6kbwg--managed/providers/Microsoft.Compute/virtualMachines/l6k1q7q7h9e3k0z-np-1-thnn4-9rrd8 to exist\nMachine l6k1q7q7h9e3k0z-np-1-thnn4-w82sd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-lz9smvq6kbwg--managed/providers/Microsoft.Compute/virtualMachines/l6k1q7q7h9e3k0z-np-1-thnn4-w82sd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1933056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ea7a0229-5912-411c-9d71-ed6121f1fa93
            correlation ID: e016f547-ae67-4080-be2a-aa90f089b488
            client request ID: c52a8689-3c38-4f1f-a275-e48620136ff8
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x26ee15ea6540>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-lz9smvq6kbwg" to finish creating: GET https://rp.j1933056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ea7a0229-5912-411c-9d71-ed6121f1fa93
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ea7a0229-5912-411c-9d71-ed6121f1fa93",
      "name": "ea7a0229-5912-411c-9d71-ed6121f1fa93",
      "status": "Failed",
      "startTime": "2026-10-27T23:28:59.117849892Z",
      "endTime": "2026-10-27T23:48:07.430527402Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine l6k1q7q7h9e3k0z-np-1-thnn4-9rrd8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-lz9smvq6kbwg--managed/providers/Microsoft.Compute/virtualMachines/l6k1q7q7h9e3k0z-np-1-thnn4-9rrd8 to exist\nMachine l6k1q7q7h9e3k0z-np-1-thnn4-w82sd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-lz9smvq6kbwg--managed/providers/Microsoft.Compute/virtualMachines/l6k1q7q7h9e3k0z-np-1-thnn4-w82sd to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine l6k1q7q7h9e3k0z-np-1-thnn4-9rrd8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-lz9smvq6kbwg--managed/providers/Microsoft.Compute/virtualMachines/l6k1q7q7h9e3k0z-np-1-thnn4-9rrd8 to exist\nMachine l6k1q7q7h9e3k0z-np-1-thnn4-w82sd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-lz9smvq6kbwg--managed/providers/Microsoft.Compute/virtualMachines/l6k1q7q7h9e3k0z-np-1-thnn4-w82sd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1933056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ea7a0229-5912-411c-9d71-ed6121f1fa93
            correlation ID: e016f547-ae67-4080-be2a-aa90f089b488
            client request ID: c52a8689-3c38-4f1f-a275-e48620136ff8
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-27 22:20	PR #6924	2115207073061933056

alert [svc] KubeDeploymentReplicasMismatch fired

full failure pattern:

alert [svc] KubeDeploymentReplicasMismatch fired

Signal:

alert

none

Full failure examples (1)

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-27T14:32:19Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8622848-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.158:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8622848/providers/microsoft.monitor/accounts/services-j8622848", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-xxzkn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-27T14:33:18Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8622848-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.158:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8622848/providers/microsoft.monitor/accounts/services-j8622848", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-xxzkn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert KubeDeploymentReplicasMismatch does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-27 13:15	PR #7615	2115069760348622848

alert [svc] KubeDeploymentRolloutStuck fired

full failure pattern:

alert [svc] KubeDeploymentRolloutStuck fired

Signal:

alert

none

Full failure examples (1)

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-27T14:42:17Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j8622848-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.158:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8622848/providers/microsoft.monitor/accounts/services-j8622848", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-xxzkn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-27T14:43:21Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j8622848-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.158:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8622848/providers/microsoft.monitor/accounts/services-j8622848", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-xxzkn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert KubeDeploymentRolloutStuck does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-27 13:15	PR #7615	2115069760348622848

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: NotReady: * NodeHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:222]: failed to create nodepool "np-1" for authorized CIDRs cluster
Unexpected error:
    <*fmt.wrapError | 0x19ee49ef83c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-wwr8wdxmg7md" to finish creating: GET https://rp.j9721728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e7d2379c-44d1-408b-924b-bb3e4a41c31e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e7d2379c-44d1-408b-924b-bb3e4a41c31e",
      "name": "e7d2379c-44d1-408b-924b-bb3e4a41c31e",
      "status": "Failed",
      "startTime": "2026-10-27T16:09:15.328141673Z",
      "endTime": "2026-10-27T16:28:23.766899649Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine o0j6i6a1b6t0q5h-np-1-9klrv-7r7qm: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine o0j6i6a1b6t0q5h-np-1-9klrv-7r7qm: NotReady: * NodeHealthy:\n  * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9721728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e7d2379c-44d1-408b-924b-bb3e4a41c31e
            correlation ID: 963eb402-1e02-4815-b759-a4a687cc24af
            client request ID: 393f46a9-acc4-4c32-9f2a-527f2574f38e
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:222]: failed to create nodepool "np-1" for authorized CIDRs cluster
Unexpected error:
    <*fmt.wrapError | 0x19ee49ef83c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-wwr8wdxmg7md" to finish creating: GET https://rp.j9721728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e7d2379c-44d1-408b-924b-bb3e4a41c31e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e7d2379c-44d1-408b-924b-bb3e4a41c31e",
      "name": "e7d2379c-44d1-408b-924b-bb3e4a41c31e",
      "status": "Failed",
      "startTime": "2026-10-27T16:09:15.328141673Z",
      "endTime": "2026-10-27T16:28:23.766899649Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine o0j6i6a1b6t0q5h-np-1-9klrv-7r7qm: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine o0j6i6a1b6t0q5h-np-1-9klrv-7r7qm: NotReady: * NodeHealthy:\n  * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9721728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e7d2379c-44d1-408b-924b-bb3e4a41c31e
            correlation ID: 963eb402-1e02-4815-b759-a4a687cc24af
            client request ID: 393f46a9-acc4-4c32-9f2a-527f2574f38e
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-27 15:10	PR #7700	2115098874459721728

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want >= <count> (autoscaling min); node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x2c3a9a99f660>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-hqzrkb5qdbkx" to finish creating: GET https://rp.j1666560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ff22fd81-847d-4bcb-b22f-4250d9071095
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ff22fd81-847d-4bcb-b22f-4250d9071095",
      "name": "ff22fd81-847d-4bcb-b22f-4250d9071095",
      "status": "Failed",
      "startTime": "2026-10-27T20:11:38.810985949Z",
      "endTime": "2026-10-27T20:30:42.147810175Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine m7z0h7l7n3n4n3s-autoscale-noaz-hfvwg-htn8f: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-hqzrkb5qdbkx--managed/providers/Microsoft.Compute/virtualMachines/m7z0h7l7n3n4n3s-autoscale-noaz-hfvwg-htn8f to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine m7z0h7l7n3n4n3s-autoscale-noaz-hfvwg-htn8f: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-hqzrkb5qdbkx--managed/providers/Microsoft.Compute/virtualMachines/m7z0h7l7n3n4n3s-autoscale-noaz-hfvwg-htn8f to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1666560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ff22fd81-847d-4bcb-b22f-4250d9071095
            correlation ID: 20b2a10f-dd2b-4a62-8cff-21fa59649da0
            client request ID: a877eec2-04c5-44bd-9434-61383daeb4f5
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x2c3a9a99f660>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-hqzrkb5qdbkx" to finish creating: GET https://rp.j1666560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ff22fd81-847d-4bcb-b22f-4250d9071095
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ff22fd81-847d-4bcb-b22f-4250d9071095",
      "name": "ff22fd81-847d-4bcb-b22f-4250d9071095",
      "status": "Failed",
      "startTime": "2026-10-27T20:11:38.810985949Z",
      "endTime": "2026-10-27T20:30:42.147810175Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine m7z0h7l7n3n4n3s-autoscale-noaz-hfvwg-htn8f: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-hqzrkb5qdbkx--managed/providers/Microsoft.Compute/virtualMachines/m7z0h7l7n3n4n3s-autoscale-noaz-hfvwg-htn8f to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine m7z0h7l7n3n4n3s-autoscale-noaz-hfvwg-htn8f: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-hqzrkb5qdbkx--managed/providers/Microsoft.Compute/virtualMachines/m7z0h7l7n3n4n3s-autoscale-noaz-hfvwg-htn8f to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1666560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ff22fd81-847d-4bcb-b22f-4250d9071095
            correlation ID: 20b2a10f-dd2b-4a62-8cff-21fa59649da0
            client request ID: a877eec2-04c5-44bd-9434-61383daeb4f5
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-27 17:40	PR #7781	2115149099081666560

at least one schedule should have a LastBackupTime

full failure pattern:

at least one schedule should have a LastBackupTime

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:166]: Timed out after 660.004s.
at least one schedule should have a LastBackupTime
Expected
    <bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:166]: Timed out after 660.004s.
at least one schedule should have a LastBackupTime
Expected
    <bool>: false
to be true

fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:166]: Timed out after 660.008s.
at least one schedule should have a LastBackupTime
Expected
    <bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:166]: Timed out after 660.008s.
at least one schedule should have a LastBackupTime
Expected
    <bool>: false
to be true

fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:166]: Timed out after 660.005s.
at least one schedule should have a LastBackupTime
Expected
    <bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:166]: Timed out after 660.005s.
at least one schedule should have a LastBackupTime
Expected
    <bool>: false
to be true

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	SRE can pause schedules to stop backup execution for an HCP cluster	22

Affected runs (22)

Date (UTC)	Associated PR	Prow job
2026-10-27 09:55	PR #6635	2115019604114804736
2026-10-27 09:55	PR #6823	2115019570476486656
2026-10-27 09:10	PR #7514	2115008102154637312
2026-10-27 09:09	PR #7770	2115008051160289280
2026-10-27 08:17	PR #7740	2114994824758169600
2026-10-27 08:10	PR #6714	2114993082914050048
2026-10-27 07:48	PR #6688	2114987666448584704
2026-10-27 07:29	PR #7598	2114982798447087616
2026-10-27 07:28	PR #7041	2114982484201443328
2026-10-27 07:10	PR #7761	2114977910874968064
2026-10-27 07:07	PR #7770	2114977151651418112
2026-10-27 07:01	PR #7769	2114975739370541056
2026-10-27 06:59	PR #7768	2114975256258023424
2026-10-27 06:56	PR #7662	2114974410921218048
2026-10-27 06:14	PR #7615	2114964013585534976
2026-10-27 05:30	PR #7740	2114952950173405184
2026-10-27 03:30	PR #7767	2114922665838907392
2026-10-27 03:15	PR #6364	2114918973651816448
2026-10-27 03:14	PR #7615	2114918518389477376
2026-10-27 01:13	PR #7615	2114888160881152000
2026-10-27 01:07	PR #7739	2114886591380983808
2026-10-27 00:03	PR #7177	2114870474402959360

ERROR CODE: DeploymentFailed; detail code AzureKeyVaultKmsValidateIdentityPermissionCustomerError; detail message The identity does not have keys encrypt/decrypt permission on key...

full failure pattern:

ERROR CODE: DeploymentFailed; detail code AzureKeyVaultKmsValidateIdentityPermissionCustomerError; detail message The identity does not have keys encrypt/decrypt permission on key vault "<url>"; provider Microsoft.ContainerService

Signal:

provision

none

Full failure examples (3)

time=2026-10-27T21:46:01.135Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-27T21:46:03.401Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-27T21:46:05.062Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=ec85056f79198d17f9a4a06a9d62ec213e8ff2f13643b300963fd031a60589b0 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j0909440-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fec85056f79198d17f9a4a06a9d62ec213e8ff2f13643b300963fd031a60589b0
time=2026-10-27T21:48:05.587Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster step=cluster error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0909440-svc/providers/Microsoft.Resources/deployments/ec85056f79198d17f9a4a06a9d62ec213e8ff2f13643b300963fd031a60589b0/operationStatuses/08584084693213550197\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0909440-svc/providers/Microsoft.Resources/deployments/cluster-r5scxlmdedogk\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0909440-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j0909440-svc\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"AzureKeyVaultKmsValidateIdentityPermissionCustomerError\\\",\\r\\n                \\\"message\\\": \\\"The identity does not have keys encrypt/decrypt permission on key vault \\\\\\\"https://ah-ci01-se-j0909440-1.vault.azure.net\\\\\\\"\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
time=2026-10-27T21:50:05.589Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-27T21:50:06.576Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-27T21:50:08.087Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=cd575f368230e7d22208e5f91e10168040367ee73eee75ae5c003a42111d906b portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j0909440-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fcd575f368230e7d22208e5f91e10168040367ee73eee75ae5c003a42111d906b
time=2026-10-27T21:51:38.615Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0909440-svc/providers/Microsoft.Resources/deployments/cd575f368230e7d22208e5f91e10168040367ee73eee75ae5c003a42111d906b/operationStatuses/08584084690782406556\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"Failed to obtain the resource body, because the resource is currently in a 'Failed' provisioning state. If this resource has not been modified by a prior step in this deployment, you may need to address this before retrying the deployment.\\\"\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

time=2026-10-27T16:22:47.857Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-27T16:22:49.418Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-27T16:22:50.992Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=7ad6b585c2a0db4b1394b6a9516e5015c1679a296a1e0d17bcbd7a3fba0e88c5 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j5036160-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F7ad6b585c2a0db4b1394b6a9516e5015c1679a296a1e0d17bcbd7a3fba0e88c5
time=2026-10-27T16:25:21.585Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster step=cluster error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5036160-svc/providers/Microsoft.Resources/deployments/7ad6b585c2a0db4b1394b6a9516e5015c1679a296a1e0d17bcbd7a3fba0e88c5/operationStatuses/08584084887153570073\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5036160-svc/providers/Microsoft.Resources/deployments/cluster-iq4rpd4u335k2\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5036160-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j5036160-svc\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"AzureKeyVaultKmsValidateIdentityPermissionCustomerError\\\",\\r\\n                \\\"message\\\": \\\"The identity does not have keys encrypt/decrypt permission on key vault \\\\\\\"https://ah-ci01-se-j5036160-1.vault.azure.net\\\\\\\"\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
time=2026-10-27T16:27:21.586Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-27T16:27:22.568Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-27T16:27:27.384Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=3dc95d5c58fec68b29e80b8f5b89cd82220ecff7029d45288f7b18d7b85bc595 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j5036160-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F3dc95d5c58fec68b29e80b8f5b89cd82220ecff7029d45288f7b18d7b85bc595
time=2026-10-27T16:28:57.922Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5036160-svc/providers/Microsoft.Resources/deployments/3dc95d5c58fec68b29e80b8f5b89cd82220ecff7029d45288f7b18d7b85bc595/operationStatuses/08584084884394674722\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"Failed to obtain the resource body, because the resource is currently in a 'Failed' provisioning state. If this resource has not been modified by a prior step in this deployment, you may need to address this before retrying the deployment.\\\"\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

time=2026-10-27T11:21:11.139Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-27T11:21:12.548Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-27T11:21:13.677Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=7510a52b8b6967f369df752e1ecd7511738ad8092c8cb66a264114e05ac9f88f portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j2510848-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F7510a52b8b6967f369df752e1ecd7511738ad8092c8cb66a264114e05ac9f88f
time=2026-10-27T11:23:14.094Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster step=cluster error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2510848-svc/providers/Microsoft.Resources/deployments/7510a52b8b6967f369df752e1ecd7511738ad8092c8cb66a264114e05ac9f88f/operationStatuses/08584085068123757360\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2510848-svc/providers/Microsoft.Resources/deployments/cluster-7vqst4vw3vptw\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2510848-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j2510848-svc\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"AzureKeyVaultKmsValidateIdentityPermissionCustomerError\\\",\\r\\n                \\\"message\\\": \\\"The identity does not have keys encrypt/decrypt permission on key vault \\\\\\\"https://ah-ci01-se-j2510848-1.vault.azure.net\\\\\\\"\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
time=2026-10-27T11:25:14.100Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-27T11:25:14.853Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-27T11:25:16.187Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=d377865c9ece3412c41698f7290eb35643fec6eb6c74141ac09360de5e692c4b portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j2510848-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fd377865c9ece3412c41698f7290eb35643fec6eb6c74141ac09360de5e692c4b
time=2026-10-27T11:26:46.433Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2510848-svc/providers/Microsoft.Resources/deployments/d377865c9ece3412c41698f7290eb35643fec6eb6c74141ac09360de5e692c4b/operationStatuses/08584085065700090638\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"Failed to obtain the resource body, because the resource is currently in a 'Failed' provisioning state. If this resource has not been modified by a prior step in this deployment, you may need to address this before retrying the deployment.\\\"\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster	4

Affected runs (4)

Date (UTC)	Associated PR	Prow job
2026-10-27 21:25	PR #7670	2115193188430909440
2026-10-27 16:05	PR #7615	2115112713075036160
2026-10-27 10:54	PR #7041	2115034279682510848
2026-10-27 10:22	PR #6688	2115026232805429248

DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): co...

full failure pattern:

DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:241]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-rvmpld"
Unexpected error:
    <*fmt.wrapError | 0x1aa0726d2620>: 
    DNS for route host agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x1aa071c17780>{
            msg: "DNS for agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:241]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-rvmpld"
Unexpected error:
    <*fmt.wrapError | 0x1aa0726d2620>: 
    DNS for route host agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x1aa071c17780>{
            msg: "DNS for agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:121]: failed to verify simple web app runs on v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0x1cc8b3a2b3a0>: 
    DNS for route host agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x1cc8b3559460>{
            msg: "DNS for agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:121]: failed to verify simple web app runs on v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0x1cc8b3a2b3a0>: 
    DNS for route host agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x1cc8b3559460>{
            msg: "DNS for agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:116]: failed to verify simple web app runs on cluster "cluster-hs-pre-lh6qk9"
Unexpected error:
    <*fmt.wrapError | 0x3fdd98ed4800>: 
    DNS for route host agnhost-e2e-sample-app-bvlxv.apps.aro.p6n9f5i0h7d1x2n.1mdy.j9327744.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-bvlxv.apps.aro.p6n9f5i0h7d1x2n.1mdy.j9327744.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-bvlxv.apps.aro.p6n9f5i0h7d1x2n.1mdy.j9327744.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-bvlxv.apps.aro.p6n9f5i0h7d1x2n.1mdy.j9327744.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-bvlxv.apps.aro.p6n9f5i0h7d1x2n.1mdy.j9327744.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-bvlxv.apps.aro.p6n9f5i0h7d1x2n.1mdy.j9327744.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x3fdd9882d200>{
            msg: "DNS for agnhost-e2e-sample-app-bvlxv.apps.aro.p6n9f5i0h7d1x2n.1mdy.j9327744.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-bvlxv.apps.aro.p6n9f5i0h7d1x2n.1mdy.j9327744.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:116]: failed to verify simple web app runs on cluster "cluster-hs-pre-lh6qk9"
Unexpected error:
    <*fmt.wrapError | 0x3fdd98ed4800>: 
    DNS for route host agnhost-e2e-sample-app-bvlxv.apps.aro.p6n9f5i0h7d1x2n.1mdy.j9327744.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-bvlxv.apps.aro.p6n9f5i0h7d1x2n.1mdy.j9327744.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-bvlxv.apps.aro.p6n9f5i0h7d1x2n.1mdy.j9327744.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-bvlxv.apps.aro.p6n9f5i0h7d1x2n.1mdy.j9327744.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-bvlxv.apps.aro.p6n9f5i0h7d1x2n.1mdy.j9327744.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-bvlxv.apps.aro.p6n9f5i0h7d1x2n.1mdy.j9327744.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x3fdd9882d200>{
            msg: "DNS for agnhost-e2e-sample-app-bvlxv.apps.aro.p6n9f5i0h7d1x2n.1mdy.j9327744.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-bvlxv.apps.aro.p6n9f5i0h7d1x2n.1mdy.j9327744.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred

Contributing tests (4)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster using v20260901preview API and verify cluster health	1

Affected runs (4)

Date (UTC)	Associated PR	Prow job
2026-10-27 20:12	PR #7769	2115174872618373120
2026-10-27 19:36	PR #6580	2115165748904071168
2026-10-27 11:17	PR #7770	2115040147719327744
2026-10-27 08:17	PR #7740	2114994824758169600

ERROR CODE: InvalidResource; detail message DataPlaneIdentitiesPermissionsValidation: InternalError: Unable to verify data plane identities permissions.; provider Microsoft.RedHat...

full failure pattern:

ERROR CODE: InvalidResource; detail message DataPlaneIdentitiesPermissionsValidation: InternalError: Unable to verify data plane identities permissions.; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-23-hd6rvlcsz446/cluster-candidate-4-23-z5pr5h should provision
Unexpected error:
    <*fmt.wrapError | 0xd0b2342e360>: 
    failed to create HCP cluster cluster-candidate-4-23-z5pr5h: failed waiting for cluster="cluster-candidate-4-23-z5pr5h" in resourcegroup="rg-candidate-4-23-hd6rvlcsz446" to finish creating: GET https://rp.j9327744.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7afde425-4c5d-47c7-a424-08163e41623a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidResource
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7afde425-4c5d-47c7-a424-08163e41623a",
      "name": "7afde425-4c5d-47c7-a424-08163e41623a",
      "status": "Failed",
      "startTime": "2026-10-27T12:16:28.588568581Z",
      "endTime": "2026-10-27T12:27:01.432725234Z",
      "error": {
        "code": "InvalidResource",
        "message": "DataPlaneIdentitiesPermissionsValidation: InternalError: Unable to verify data plane identities permissions."
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9327744.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7afde425-4c5d-47c7-a424-08163e41623a
            correlation ID: 3d3fc710-9f38-4fa0-b097-667c1de10d30
            client request ID: 7b1eba2b-65de-404e-b3ec-50e9f1177df4
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-23-hd6rvlcsz446/cluster-candidate-4-23-z5pr5h should provision
Unexpected error:
    <*fmt.wrapError | 0xd0b2342e360>: 
    failed to create HCP cluster cluster-candidate-4-23-z5pr5h: failed waiting for cluster="cluster-candidate-4-23-z5pr5h" in resourcegroup="rg-candidate-4-23-hd6rvlcsz446" to finish creating: GET https://rp.j9327744.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7afde425-4c5d-47c7-a424-08163e41623a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidResource
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7afde425-4c5d-47c7-a424-08163e41623a",
      "name": "7afde425-4c5d-47c7-a424-08163e41623a",
      "status": "Failed",
      "startTime": "2026-10-27T12:16:28.588568581Z",
      "endTime": "2026-10-27T12:27:01.432725234Z",
      "error": {
        "code": "InvalidResource",
        "message": "DataPlaneIdentitiesPermissionsValidation: InternalError: Unable to verify data plane identities permissions."
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9327744.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7afde425-4c5d-47c7-a424-08163e41623a
            correlation ID: 3d3fc710-9f38-4fa0-b097-667c1de10d30
            client request ID: 7b1eba2b-65de-404e-b3ec-50e9f1177df4
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-22-gcsfgvdh5qhb/cluster-candidate-4-22-2w4gz9 should provision
Unexpected error:
    <*fmt.wrapError | 0x30d8f6930800>: 
    failed to create HCP cluster cluster-candidate-4-22-2w4gz9: failed waiting for cluster="cluster-candidate-4-22-2w4gz9" in resourcegroup="rg-candidate-4-22-gcsfgvdh5qhb" to finish creating: GET https://rp.j9327744.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/74f9eae7-150f-4f86-81e4-f4ada23068e6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidResource
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/74f9eae7-150f-4f86-81e4-f4ada23068e6",
      "name": "74f9eae7-150f-4f86-81e4-f4ada23068e6",
      "status": "Failed",
      "startTime": "2026-10-27T12:17:08.196779757Z",
      "endTime": "2026-10-27T12:28:21.232176744Z",
      "error": {
        "code": "InvalidResource",
        "message": "DataPlaneIdentitiesPermissionsValidation: InternalError: Unable to verify data plane identities permissions."
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9327744.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/74f9eae7-150f-4f86-81e4-f4ada23068e6
            correlation ID: d480a26c-43a2-49b4-8de4-02fc3c46d885
            client request ID: 8e7d5833-9216-45e5-8c64-c9b98a48c0a2
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-22-gcsfgvdh5qhb/cluster-candidate-4-22-2w4gz9 should provision
Unexpected error:
    <*fmt.wrapError | 0x30d8f6930800>: 
    failed to create HCP cluster cluster-candidate-4-22-2w4gz9: failed waiting for cluster="cluster-candidate-4-22-2w4gz9" in resourcegroup="rg-candidate-4-22-gcsfgvdh5qhb" to finish creating: GET https://rp.j9327744.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/74f9eae7-150f-4f86-81e4-f4ada23068e6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidResource
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/74f9eae7-150f-4f86-81e4-f4ada23068e6",
      "name": "74f9eae7-150f-4f86-81e4-f4ada23068e6",
      "status": "Failed",
      "startTime": "2026-10-27T12:17:08.196779757Z",
      "endTime": "2026-10-27T12:28:21.232176744Z",
      "error": {
        "code": "InvalidResource",
        "message": "DataPlaneIdentitiesPermissionsValidation: InternalError: Unable to verify data plane identities permissions."
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9327744.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/74f9eae7-150f-4f86-81e4-f4ada23068e6
            correlation ID: d480a26c-43a2-49b4-8de4-02fc3c46d885
            client request ID: 8e7d5833-9216-45e5-8c64-c9b98a48c0a2
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-jhb5b2" with version 4.22.19 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x32758ffae460>: 
    failed to create HCP cluster cluster-zstream-4-22-jhb5b2: failed waiting for cluster="cluster-zstream-4-22-jhb5b2" in resourcegroup="rg-zstream-upgrade-4-22-lhz7fdj9k86z" to finish creating: GET https://rp.j7472896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fd748747-ebe4-4429-9e81-de4bcbbdaddc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidResource
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fd748747-ebe4-4429-9e81-de4bcbbdaddc",
      "name": "fd748747-ebe4-4429-9e81-de4bcbbdaddc",
      "status": "Failed",
      "startTime": "2026-10-27T12:19:17.939641835Z",
      "endTime": "2026-10-27T12:30:15.55604778Z",
      "error": {
        "code": "InvalidResource",
        "message": "DataPlaneIdentitiesPermissionsValidation: InternalError: Unable to verify data plane identities permissions."
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7472896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fd748747-ebe4-4429-9e81-de4bcbbdaddc
            correlation ID: 046c9d63-6366-4a78-bd4e-177dad3c7de6
            client request ID: 40011f58-fbc8-44b5-a868-12a5ed482aad
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-jhb5b2" with version 4.22.19 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x32758ffae460>: 
    failed to create HCP cluster cluster-zstream-4-22-jhb5b2: failed waiting for cluster="cluster-zstream-4-22-jhb5b2" in resourcegroup="rg-zstream-upgrade-4-22-lhz7fdj9k86z" to finish creating: GET https://rp.j7472896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fd748747-ebe4-4429-9e81-de4bcbbdaddc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidResource
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fd748747-ebe4-4429-9e81-de4bcbbdaddc",
      "name": "fd748747-ebe4-4429-9e81-de4bcbbdaddc",
      "status": "Failed",
      "startTime": "2026-10-27T12:19:17.939641835Z",
      "endTime": "2026-10-27T12:30:15.55604778Z",
      "error": {
        "code": "InvalidResource",
        "message": "DataPlaneIdentitiesPermissionsValidation: InternalError: Unable to verify data plane identities permissions."
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7472896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fd748747-ebe4-4429-9e81-de4bcbbdaddc
            correlation ID: 046c9d63-6366-4a78-bd4e-177dad3c7de6
            client request ID: 40011f58-fbc8-44b5-a868-12a5ed482aad
occurred

Contributing tests (3)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-27 11:17	PR #7770	2115040147719327744
2026-10-27 11:16	PR #7740	2115040060427472896

failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>

full failure pattern:

failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>

Signal:

e2e

STG

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0xb71d4fca080>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-bqbx2znvzb54" to finish creating: GET https://rp.j1933056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0ed006bd-b8aa-4770-9062-3fd090d4fea9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0ed006bd-b8aa-4770-9062-3fd090d4fea9",
      "name": "0ed006bd-b8aa-4770-9062-3fd090d4fea9",
      "status": "Failed",
      "startTime": "2026-10-27T23:25:38.076776565Z",
      "endTime": "2026-10-27T23:44:40.197052822Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-xjlrw-mjpj4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-bqbx2znvzb54--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-xjlrw-mjpj4 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine ingress-cert-le-np-1-xjlrw-7jnb9: Failed: interfaces failed to create or update. err: failed to get existing resource ingress-cert-le-bqbx2znvzb54--managed/ingress-cert-le-np-1-xjlrw-7jnb9-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-bqbx2znvzb54--managed/providers/Microsoft.Network/networkInterfaces/ingress-cert-le-np-1-xjlrw-7jnb9-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '8d1ff6d0-aadd-4331-9171-9443ac5f4337' exceeded. Please try again after '10' seconds after additional tokens are available. Refer ... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1933056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0ed006bd-b8aa-4770-9062-3fd090d4fea9
            correlation ID: d0096963-5f3a-4ca8-942f-d516adbeb2fc
            client request ID: 336f75bd-67db-4bb4-b035-81a9154977bb
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0xb71d4fca080>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-bqbx2znvzb54" to finish creating: GET https://rp.j1933056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0ed006bd-b8aa-4770-9062-3fd090d4fea9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0ed006bd-b8aa-4770-9062-3fd090d4fea9",
      "name": "0ed006bd-b8aa-4770-9062-3fd090d4fea9",
      "status": "Failed",
      "startTime": "2026-10-27T23:25:38.076776565Z",
      "endTime": "2026-10-27T23:44:40.197052822Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-xjlrw-mjpj4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-bqbx2znvzb54--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-xjlrw-mjpj4 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine ingress-cert-le-np-1-xjlrw-7jnb9: Failed: interfaces failed to create or update. err: failed to get existing resource ingress-cert-le-bqbx2znvzb54--managed/ingress-cert-le-np-1-xjlrw-7jnb9-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-bqbx2znvzb54--managed/providers/Microsoft.Network/networkInterfaces/ingress-cert-le-np-1-xjlrw-7jnb9-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '8d1ff6d0-aadd-4331-9171-9443ac5f4337' exceeded. Please try again after '10' seconds after additional tokens are available. Refer ... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1933056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0ed006bd-b8aa-4770-9062-3fd090d4fea9
            correlation ID: d0096963-5f3a-4ca8-942f-d516adbeb2fc
            client request ID: 336f75bd-67db-4bb4-b035-81a9154977bb

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0x3591356d60>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-6b928vvvc9lv" to finish creating: GET https://rp.j1933056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e68fe219-929c-499f-9d94-52a89aa54331
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e68fe219-929c-499f-9d94-52a89aa54331",
      "name": "e68fe219-929c-499f-9d94-52a89aa54331",
      "status": "Failed",
      "startTime": "2026-10-27T23:27:39.482004964Z",
      "endTime": "2026-10-27T23:46:40.23395218Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-qtp5t-vlfjh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-6b928vvvc9lv--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-qtp5t-vlfjh to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-qtp5t-vlfjh: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-6b928vvvc9lv--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-qtp5t-vlfjh to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1933056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e68fe219-929c-499f-9d94-52a89aa54331
            correlation ID: 0fae4e0c-a402-45c6-8599-10c6e41f505d
            client request ID: c1da63b5-9f53-4e68-8989-fc0f3ef1f3cb
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0x3591356d60>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-6b928vvvc9lv" to finish creating: GET https://rp.j1933056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e68fe219-929c-499f-9d94-52a89aa54331
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e68fe219-929c-499f-9d94-52a89aa54331",
      "name": "e68fe219-929c-499f-9d94-52a89aa54331",
      "status": "Failed",
      "startTime": "2026-10-27T23:27:39.482004964Z",
      "endTime": "2026-10-27T23:46:40.23395218Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-qtp5t-vlfjh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-6b928vvvc9lv--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-qtp5t-vlfjh to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-qtp5t-vlfjh: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-6b928vvvc9lv--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-qtp5t-vlfjh to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1933056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e68fe219-929c-499f-9d94-52a89aa54331
            correlation ID: 0fae4e0c-a402-45c6-8599-10c6e41f505d
            client request ID: c1da63b5-9f53-4e68-8989-fc0f3ef1f3cb

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0x1218a86ea3a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-trmq28plbffk" to finish creating: GET https://rp.j7635968.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4dccef60-3d8d-4da6-abc7-6f50fc2d7fea
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4dccef60-3d8d-4da6-abc7-6f50fc2d7fea",
      "name": "4dccef60-3d8d-4da6-abc7-6f50fc2d7fea",
      "status": "Failed",
      "startTime": "2026-10-27T17:48:09.944151958Z",
      "endTime": "2026-10-27T18:07:12.889340309Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-vq2bz-jbwhf: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine ingress-cert-le-np-1-vq2bz-jbwhf: Failed: interfaces failed to create or update. err: failed to get existing resource ingress-cert-le-trmq28plbffk--managed/ingress-cert-le-np-1-vq2bz-jbwhf-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-trmq28plbffk--managed/providers/Microsoft.Network/networkInterfaces/ingress-cert-le-np-1-vq2bz-jbwhf-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'b26c2343-0863-4a55-bb7e-dfb2544999c1' exceeded. Please try again after '10' seconds after additional tokens are available. Refer ... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7635968.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4dccef60-3d8d-4da6-abc7-6f50fc2d7fea
            correlation ID: 08a9a0eb-cef3-4f31-9ee9-fe86da6b3d5c
            client request ID: 4911c042-45cb-4cb5-8e27-b7f9d98a083a
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0x1218a86ea3a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-trmq28plbffk" to finish creating: GET https://rp.j7635968.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4dccef60-3d8d-4da6-abc7-6f50fc2d7fea
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4dccef60-3d8d-4da6-abc7-6f50fc2d7fea",
      "name": "4dccef60-3d8d-4da6-abc7-6f50fc2d7fea",
      "status": "Failed",
      "startTime": "2026-10-27T17:48:09.944151958Z",
      "endTime": "2026-10-27T18:07:12.889340309Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-vq2bz-jbwhf: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine ingress-cert-le-np-1-vq2bz-jbwhf: Failed: interfaces failed to create or update. err: failed to get existing resource ingress-cert-le-trmq28plbffk--managed/ingress-cert-le-np-1-vq2bz-jbwhf-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-trmq28plbffk--managed/providers/Microsoft.Network/networkInterfaces/ingress-cert-le-np-1-vq2bz-jbwhf-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'b26c2343-0863-4a55-bb7e-dfb2544999c1' exceeded. Please try again after '10' seconds after additional tokens are available. Refer ... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7635968.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4dccef60-3d8d-4da6-abc7-6f50fc2d7fea
            correlation ID: 08a9a0eb-cef3-4f31-9ee9-fe86da6b3d5c
            client request ID: 4911c042-45cb-4cb5-8e27-b7f9d98a083a

Contributing tests (2)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 5.1	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 5.0	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-27 22:20	PR #6924	2115207073061933056
2026-10-27 16:44	PR #7615	2115122591927635968

ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<u...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0x5f50682dba0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-zj558pxt722v" to finish creating: GET https://rp.j4340096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a82259f2-e169-4ef9-b443-fe55248e1a5a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a82259f2-e169-4ef9-b443-fe55248e1a5a",
      "name": "a82259f2-e169-4ef9-b443-fe55248e1a5a",
      "status": "Failed",
      "startTime": "2026-10-28T00:07:43.50327074Z",
      "endTime": "2026-10-28T00:26:52.986365896Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20260901-np-1-lrdwb-g7cqn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-zj558pxt722v--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-lrdwb-g7cqn to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20260901-np-1-lrdwb-ck62b: Failed: interfaces failed to create or update. err: failed to get existing resource v20260901preview-zj558pxt722v--managed/v20260901-np-1-lrdwb-ck62b-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-zj558pxt722v--managed/providers/Microsoft.Network/networkInterfaces/v20260901-np-1-lrdwb-ck62b-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'fd35ac5f-6493-40cb-9131-d669a70114c3' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.m... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j4340096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a82259f2-e169-4ef9-b443-fe55248e1a5a
            correlation ID: 3ba28a91-2919-42f2-8746-e40059a09487
            client request ID: ed3e3d87-8d0e-41b4-b0be-5121fe0a6ea7
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0x5f50682dba0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-zj558pxt722v" to finish creating: GET https://rp.j4340096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a82259f2-e169-4ef9-b443-fe55248e1a5a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a82259f2-e169-4ef9-b443-fe55248e1a5a",
      "name": "a82259f2-e169-4ef9-b443-fe55248e1a5a",
      "status": "Failed",
      "startTime": "2026-10-28T00:07:43.50327074Z",
      "endTime": "2026-10-28T00:26:52.986365896Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20260901-np-1-lrdwb-g7cqn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-zj558pxt722v--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-lrdwb-g7cqn to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20260901-np-1-lrdwb-ck62b: Failed: interfaces failed to create or update. err: failed to get existing resource v20260901preview-zj558pxt722v--managed/v20260901-np-1-lrdwb-ck62b-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-zj558pxt722v--managed/providers/Microsoft.Network/networkInterfaces/v20260901-np-1-lrdwb-ck62b-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'fd35ac5f-6493-40cb-9131-d669a70114c3' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.m... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j4340096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a82259f2-e169-4ef9-b443-fe55248e1a5a
            correlation ID: 3ba28a91-2919-42f2-8746-e40059a09487
            client request ID: ed3e3d87-8d0e-41b4-b0be-5121fe0a6ea7
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0x267931bca260>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-hpqngqxblkt5" to finish creating: GET https://rp.j3936384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4ce6c32c-2374-4848-87d4-d06da5841ac0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4ce6c32c-2374-4848-87d4-d06da5841ac0",
      "name": "4ce6c32c-2374-4848-87d4-d06da5841ac0",
      "status": "Failed",
      "startTime": "2026-10-27T20:35:56.651068091Z",
      "endTime": "2026-10-27T20:55:03.998387798Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20260901-np-1-npwtg-fbvfl: Failed: virtualmachine failed to create or update. err: failed to get existing resource v20260901preview-hpqngqxblkt5--managed/v20260901-np-1-npwtg-fbvfl (service: virtualmachine): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-hpqngqxblkt5--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-npwtg-fbvfl\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '0d25d885-2cd8-4972-b610-4d85881c1ec4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3936384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4ce6c32c-2374-4848-87d4-d06da5841ac0
            correlation ID: d0468374-ad1f-4a29-ba7f-fc2cf91bd94a
            client request ID: 907abad4-e4b1-4719-9ea0-4da30fbe13ee
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0x267931bca260>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-hpqngqxblkt5" to finish creating: GET https://rp.j3936384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4ce6c32c-2374-4848-87d4-d06da5841ac0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4ce6c32c-2374-4848-87d4-d06da5841ac0",
      "name": "4ce6c32c-2374-4848-87d4-d06da5841ac0",
      "status": "Failed",
      "startTime": "2026-10-27T20:35:56.651068091Z",
      "endTime": "2026-10-27T20:55:03.998387798Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20260901-np-1-npwtg-fbvfl: Failed: virtualmachine failed to create or update. err: failed to get existing resource v20260901preview-hpqngqxblkt5--managed/v20260901-np-1-npwtg-fbvfl (service: virtualmachine): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-hpqngqxblkt5--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-npwtg-fbvfl\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '0d25d885-2cd8-4972-b610-4d85881c1ec4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3936384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4ce6c32c-2374-4848-87d4-d06da5841ac0
            correlation ID: d0468374-ad1f-4a29-ba7f-fc2cf91bd94a
            client request ID: 907abad4-e4b1-4719-9ea0-4da30fbe13ee
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster using v20260901preview API and verify cluster health	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-27 22:56	PR #7769	2115216026894340096
2026-10-27 19:28	PR #5244	2115163835793936384

alert [svc] KubePodCrashLooping fired

full failure pattern:

alert [svc] KubePodCrashLooping fired

Signal:

alert

none

Full failure examples (2)

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-27T11:22:09Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j4445952-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.199:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4445952/providers/microsoft.monitor/accounts/services-j4445952", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-qj8wt", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="4196d6e7-f3b2-4df6-a959-ed3e1297c6ea"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-qj8wt (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-27T11:24:08Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j4445952-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.27:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4445952/providers/microsoft.monitor/accounts/services-j4445952", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-vpw6d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e6f828e4-a265-4c17-9265-1793badfcdf7"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-vpw6d (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-27T11:24:08Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j4445952-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.199:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4445952/providers/microsoft.monitor/accounts/services-j4445952", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-qj8wt", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="4196d6e7-f3b2-4df6-a959-ed3e1297c6ea"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-qj8wt (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-27T11:24:08Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j4445952-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.27:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4445952/providers/microsoft.monitor/accounts/services-j4445952", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-vpw6d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e6f828e4-a265-4c17-9265-1793badfcdf7"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-vpw6d (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-27T08:03:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1218048-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.120:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1218048/providers/microsoft.monitor/accounts/services-j1218048", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-487l5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="1f3e87ec-25dc-43bc-b62e-8a9a539fc230"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-487l5 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-27T08:06:18Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1218048-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.41:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1218048/providers/microsoft.monitor/accounts/services-j1218048", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-qrn9k", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="b0c2ca63-da52-4110-a38b-c0b2caede123"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-qrn9k (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-27T08:06:18Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1218048-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.120:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1218048/providers/microsoft.monitor/accounts/services-j1218048", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-487l5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="1f3e87ec-25dc-43bc-b62e-8a9a539fc230"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-487l5 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-27T08:06:18Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1218048-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.41:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1218048/providers/microsoft.monitor/accounts/services-j1218048", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-qrn9k", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="b0c2ca63-da52-4110-a38b-c0b2caede123"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-qrn9k (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert KubePodCrashLooping does not fire	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-27 10:19	PR #7662	2115025503944445952
2026-10-27 06:56	PR #7662	2114974410921218048

VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node <node> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "<node>": nodes "<node>" ...

full failure pattern:

VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node <node> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "<node>": nodes "<node>" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not

Signal:

e2e

INT

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-0-j22hl8"
Unexpected error:
    <*errors.joinError | 0x237cb4aa4138>: 
    VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99": nodes "s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
    {
        errs: [
            <*fmt.wrapError | 0x237cb4952040>{
                msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\": nodes \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                err: <*fmt.wrapError | 0x237cb4952020>{
                    msg: "failed to label node s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\": nodes \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                    err: <*fmt.wrapError | 0x237cb4952000>{
                        msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\": nodes \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                        err: <*errors.StatusError | 0x237cb49b60a0>{
                            ErrStatus: {
                                TypeMeta: {Kind: "", APIVersion: ""},
                                ListMeta: {
                                    SelfLink: "",
                                    ResourceVersion: "",
                                    Continue: "",
                                    RemainingItemCount: nil,
                                    ShardInfo: nil,
                                },
                                Status: "Failure",
                                Message: "nodes \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                                Reason: "Forbidden",
                                Details: {
                                    Name: "s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99",
                                    Group: "",
                                    Kind: "nodes",
                                    UID: "",
                                    Causes: nil,
                                    RetryAfterSeconds: 0,
                                },
                                Code: 403,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-0-j22hl8"
Unexpected error:
    <*errors.joinError | 0x237cb4aa4138>: 
    VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99": nodes "s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
    {
        errs: [
            <*fmt.wrapError | 0x237cb4952040>{
                msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\": nodes \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                err: <*fmt.wrapError | 0x237cb4952020>{
                    msg: "failed to label node s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\": nodes \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                    err: <*fmt.wrapError | 0x237cb4952000>{
                        msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\": nodes \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                        err: <*errors.StatusError | 0x237cb49b60a0>{
                            ErrStatus: {
                                TypeMeta: {Kind: "", APIVersion: ""},
                                ListMeta: {
                                    SelfLink: "",
                                    ResourceVersion: "",
                                    Continue: "",
                                    RemainingItemCount: nil,
                                    ShardInfo: nil,
                                },
                                Status: "Failure",
                                Message: "nodes \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                                Reason: "Forbidden",
                                Details: {
                                    Name: "s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99",
                                    Group: "",
                                    Kind: "nodes",
                                    UID: "",
                                    Causes: nil,
                                    RetryAfterSeconds: 0,
                                },
                                Code: 403,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-1-mf9p48"
Unexpected error:
    <*errors.joinError | 0x3fde6df0348>: 
    VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node n5m6j4y9e5r3n1x-np-mf9p48-5rm8z-btdr5 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "n5m6j4y9e5r3n1x-np-mf9p48-5rm8z-btdr5": nodes "n5m6j4y9e5r3n1x-np-mf9p48-5rm8z-btdr5" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
    {
        errs: [
            <*fmt.wrapError | 0x3fde61ae7e0>{
                msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node n5m6j4y9e5r3n1x-np-mf9p48-5rm8z-btdr5 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"n5m6j4y9e5r3n1x-np-mf9p48-5rm8z-btdr5\": nodes \"n5m6j4y9e5r3n1x-np-mf9p48-5rm8z-btdr5\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                err: <*fmt.wrapError | 0x3fde61ae7c0>{
                    msg: "failed to label node n5m6j4y9e5r3n1x-np-mf9p48-5rm8z-btdr5 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"n5m6j4y9e5r3n1x-np-mf9p48-5rm8z-btdr5\": nodes \"n5m6j4y9e5r3n1x-np-mf9p48-5rm8z-btdr5\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                    err: <*fmt.wrapError | 0x3fde61ae7a0>{
                        msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"n5m6j4y9e5r3n1x-np-mf9p48-5rm8z-btdr5\": nodes \"n5m6j4y9e5r3n1x-np-mf9p48-5rm8z-btdr5\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                        err: <*errors.StatusError | 0x3fde6bae3c0>{
                            ErrStatus: {
                                TypeMeta: {Kind: "", APIVersion: ""},
                                ListMeta: {
                                    SelfLink: "",
                                    ResourceVersion: "",
                                    Continue: "",
                                    RemainingItemCount: nil,
                                    ShardInfo: nil,
                                },
                                Status: "Failure",
                                Message: "nodes \"n5m6j4y9e5r3n1x-np-mf9p48-5rm8z-btdr5\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                                Reason: "Forbidden",
                                Details: {
                                    Name: "n5m6j4y9e5r3n1x-np-mf9p48-5rm8z-btdr5",
                                    Group: "",
                                    Kind: "nodes",
                                    UID: "",
                                    Causes: nil,
                                    RetryAfterSeconds: 0,
                                },
                                Code: 403,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-1-mf9p48"
Unexpected error:
    <*errors.joinError | 0x3fde6df0348>: 
    VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node n5m6j4y9e5r3n1x-np-mf9p48-5rm8z-btdr5 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "n5m6j4y9e5r3n1x-np-mf9p48-5rm8z-btdr5": nodes "n5m6j4y9e5r3n1x-np-mf9p48-5rm8z-btdr5" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
    {
        errs: [
            <*fmt.wrapError | 0x3fde61ae7e0>{
                msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node n5m6j4y9e5r3n1x-np-mf9p48-5rm8z-btdr5 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"n5m6j4y9e5r3n1x-np-mf9p48-5rm8z-btdr5\": nodes \"n5m6j4y9e5r3n1x-np-mf9p48-5rm8z-btdr5\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                err: <*fmt.wrapError | 0x3fde61ae7c0>{
                    msg: "failed to label node n5m6j4y9e5r3n1x-np-mf9p48-5rm8z-btdr5 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"n5m6j4y9e5r3n1x-np-mf9p48-5rm8z-btdr5\": nodes \"n5m6j4y9e5r3n1x-np-mf9p48-5rm8z-btdr5\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                    err: <*fmt.wrapError | 0x3fde61ae7a0>{
                        msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"n5m6j4y9e5r3n1x-np-mf9p48-5rm8z-btdr5\": nodes \"n5m6j4y9e5r3n1x-np-mf9p48-5rm8z-btdr5\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                        err: <*errors.StatusError | 0x3fde6bae3c0>{
                            ErrStatus: {
                                TypeMeta: {Kind: "", APIVersion: ""},
                                ListMeta: {
                                    SelfLink: "",
                                    ResourceVersion: "",
                                    Continue: "",
                                    RemainingItemCount: nil,
                                    ShardInfo: nil,
                                },
                                Status: "Failure",
                                Message: "nodes \"n5m6j4y9e5r3n1x-np-mf9p48-5rm8z-btdr5\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                                Reason: "Forbidden",
                                Details: {
                                    Name: "n5m6j4y9e5r3n1x-np-mf9p48-5rm8z-btdr5",
                                    Group: "",
                                    Kind: "nodes",
                                    UID: "",
                                    Causes: nil,
                                    RetryAfterSeconds: 0,
                                },
                                Code: 403,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred

Contributing tests (2)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.1	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-27 14:43	PR #6718	2115092150784888832
2026-10-27 11:04	PR #7771	2115036855358459904

alert [svc] ClusterCredentialMetricMissing fired

full failure pattern:

alert [svc] ClusterCredentialMetricMissing fired

Signal:

alert

none

Full failure examples (2)

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-27T21:37:46Z
  Severity:    Sev3
  Labels:      alertname="ClusterCredentialMetricMissing", cluster="ci01-j3842944-svc", severity="warning"
  Description: Credential refresher expiration metric is missing for service cluster ci01-j3842944-svc.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-27T18:18:42Z
  Severity:    Sev3
  Labels:      alertname="ClusterCredentialMetricMissing", cluster="ci01-j4499328-svc", severity="warning"
  Description: Credential refresher expiration metric is missing for service cluster ci01-j4499328-svc.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert ClusterCredentialMetricMissing does not fire	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-27 20:12	PR #6081	2115174942373842944
2026-10-27 17:02	PR #6081	2115127036694499328

alert [svc] userJourneyBackupMetricsAbsent fired

full failure pattern:

alert [svc] userJourneyBackupMetricsAbsent fired

Signal:

alert

none

Full failure examples (2)

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-28T00:19:29Z
  Ended:       2026-10-28T00:26:31Z
  Severity:    Sev3
  Labels:      alertname="userJourneyBackupMetricsAbsent", cluster="ci01-j8270720-mgmt-2", component="slo", severity="3", slo="backup-metrics-absent"
  Description: Management cluster ci01-j8270720-mgmt-2 is in the underlay-cluster inventory but has had no successful Velero server metrics scrape for 15m. Check Velero, its PodMonitor, and the management-cluster metrics pipeline. A healthy empty cluster with up=1 is not a failure merely because it has no backup attempts.
Firing 2:
  State:       Resolved
  Started:     2026-10-28T00:20:31Z
  Ended:       2026-10-28T00:26:31Z
  Severity:    Sev3
  Labels:      alertname="userJourneyBackupMetricsAbsent", cluster="ci01-j8270720-mgmt-1", component="slo", severity="3", slo="backup-metrics-absent"
  Description: Management cluster ci01-j8270720-mgmt-1 is in the underlay-cluster inventory but has had no successful Velero server metrics scrape for 15m. Check Velero, its PodMonitor, and the management-cluster metrics pipeline. A healthy empty cluster with up=1 is not a failure merely because it has no backup attempts.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-27T15:59:47Z
  Severity:    Sev3
  Labels:      alertname="userJourneyBackupMetricsAbsent", cluster="ci01-j5601792-mgmt-2", component="slo", severity="3", slo="backup-metrics-absent"
  Description: Management cluster ci01-j5601792-mgmt-2 is in the underlay-cluster inventory but has had no successful Velero server metrics scrape for 15m. Check Velero, its PodMonitor, and the management-cluster metrics pipeline. A healthy empty cluster with up=1 is not a failure merely because it has no backup attempts.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert userJourneyBackupMetricsAbsent does not fire	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-27 23:23	PR #5244	2115222887018270720
2026-10-27 14:50	PR #7670	2115093743005601792

ERROR CODE: InvalidResource; detail message ControlPlaneIdentitiesPermissionsClusterValidation: InternalError: Unable to verify control plane identities permissions.; provider Mic...

full failure pattern:

ERROR CODE: InvalidResource; detail message ControlPlaneIdentitiesPermissionsClusterValidation: InternalError: Unable to verify control plane identities permissions.; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0x1523283f0200>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-dzpqr982szjp" to finish creating: GET https://rp.j7472896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5c606ccb-fd8e-489a-a868-6835e0345028
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidResource
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5c606ccb-fd8e-489a-a868-6835e0345028",
      "name": "5c606ccb-fd8e-489a-a868-6835e0345028",
      "status": "Failed",
      "startTime": "2026-10-27T12:19:12.67761363Z",
      "endTime": "2026-10-27T12:30:55.93851152Z",
      "error": {
        "code": "InvalidResource",
        "message": "ControlPlaneIdentitiesPermissionsClusterValidation: InternalError: Unable to verify control plane identities permissions."
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7472896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5c606ccb-fd8e-489a-a868-6835e0345028
            correlation ID: d35a35bb-1a40-4d4f-9155-942bc6b994d5
            client request ID: 483339d8-63cb-4cfb-a4bc-1a9aa9a36174
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0x1523283f0200>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-dzpqr982szjp" to finish creating: GET https://rp.j7472896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5c606ccb-fd8e-489a-a868-6835e0345028
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidResource
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5c606ccb-fd8e-489a-a868-6835e0345028",
      "name": "5c606ccb-fd8e-489a-a868-6835e0345028",
      "status": "Failed",
      "startTime": "2026-10-27T12:19:12.67761363Z",
      "endTime": "2026-10-27T12:30:55.93851152Z",
      "error": {
        "code": "InvalidResource",
        "message": "ControlPlaneIdentitiesPermissionsClusterValidation: InternalError: Unable to verify control plane identities permissions."
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7472896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5c606ccb-fd8e-489a-a868-6835e0345028
            correlation ID: d35a35bb-1a40-4d4f-9155-942bc6b994d5
            client request ID: 483339d8-63cb-4cfb-a4bc-1a9aa9a36174
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0x45c8638e340>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-5vfww4h6t295" to finish creating: GET https://rp.j7472896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ac0cdc9-8355-4081-91f3-1ae9204e1899
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidResource
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ac0cdc9-8355-4081-91f3-1ae9204e1899",
      "name": "1ac0cdc9-8355-4081-91f3-1ae9204e1899",
      "status": "Failed",
      "startTime": "2026-10-27T12:19:13.19675558Z",
      "endTime": "2026-10-27T12:30:26.100595452Z",
      "error": {
        "code": "InvalidResource",
        "message": "ControlPlaneIdentitiesPermissionsClusterValidation: InternalError: Unable to verify control plane identities permissions."
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7472896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ac0cdc9-8355-4081-91f3-1ae9204e1899
            correlation ID: 724394e1-dd61-4ca3-aed0-37913a4e01f3
            client request ID: f5bcc160-8ac9-4bb8-b98f-d16472eb1f7c
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0x45c8638e340>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-5vfww4h6t295" to finish creating: GET https://rp.j7472896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ac0cdc9-8355-4081-91f3-1ae9204e1899
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidResource
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ac0cdc9-8355-4081-91f3-1ae9204e1899",
      "name": "1ac0cdc9-8355-4081-91f3-1ae9204e1899",
      "status": "Failed",
      "startTime": "2026-10-27T12:19:13.19675558Z",
      "endTime": "2026-10-27T12:30:26.100595452Z",
      "error": {
        "code": "InvalidResource",
        "message": "ControlPlaneIdentitiesPermissionsClusterValidation: InternalError: Unable to verify control plane identities permissions."
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7472896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ac0cdc9-8355-4081-91f3-1ae9204e1899
            correlation ID: 724394e1-dd61-4ca3-aed0-37913a4e01f3
            client request ID: f5bcc160-8ac9-4bb8-b98f-d16472eb1f7c
occurred

Contributing tests (2)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with back-level version 4.20	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 4.20	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-27 11:16	PR #7740	2115040060427472896

VerifyAllAPIServicesAvailable failed: apiservice/v1beta1

full failure pattern:

VerifyAllAPIServicesAvailable failed: apiservice/v1beta1

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:183]: cluster viability check failed
Expected success, but got an error:
    <*errors.joinError | 0xbc827a50558>: 
    VerifyAllAPIServicesAvailable failed: apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.129.0.20:10250/apis/metrics.k8s.io/v1beta1: Get "https://10.129.0.20:10250/apis/metrics.k8s.io/v1beta1": tls: failed to verify certificate: x509: certificate signed by unknown authority
    {
        errs: [
            <*fmt.wrapError | 0xbc827a00340>{
                msg: "VerifyAllAPIServicesAvailable failed: apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.129.0.20:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.129.0.20:10250/apis/metrics.k8s.io/v1beta1\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
                err: <*errors.joinError | 0xbc827b00060>{
                    errs: [
                        <*errors.errorString | 0xbc826b52230>{
                            s: "apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.129.0.20:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.129.0.20:10250/apis/metrics.k8s.io/v1beta1\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
                        },
                    ],
                },
            },
        ],
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:183]: cluster viability check failed
Expected success, but got an error:
    <*errors.joinError | 0xbc827a50558>: 
    VerifyAllAPIServicesAvailable failed: apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.129.0.20:10250/apis/metrics.k8s.io/v1beta1: Get "https://10.129.0.20:10250/apis/metrics.k8s.io/v1beta1": tls: failed to verify certificate: x509: certificate signed by unknown authority
    {
        errs: [
            <*fmt.wrapError | 0xbc827a00340>{
                msg: "VerifyAllAPIServicesAvailable failed: apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.129.0.20:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.129.0.20:10250/apis/metrics.k8s.io/v1beta1\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
                err: <*errors.joinError | 0xbc827b00060>{
                    errs: [
                        <*errors.errorString | 0xbc826b52230>{
                            s: "apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.129.0.20:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.129.0.20:10250/apis/metrics.k8s.io/v1beta1\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
                        },
                    ],
                },
            },
        ],
    }

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 4.23	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-27 20:38	PR #7786	2115181315694268416

ERROR CODE: DeploymentFailed; detail code InternalServerError; detail message An error occurred.

full failure pattern:

ERROR CODE: DeploymentFailed; detail code InternalServerError; detail message An error occurred.

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:108]: failed to create customer resources for z-stream cluster "cluster-zstream-4-23-4q7sfm"
Unexpected error:
    <*fmt.wrapError | 0x1abe3e2ce320>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-zstream-4-23-4q7sfm-ffc6p2" in resourcegroup="rg-zstream-upgrade-4-23-r8wjcwsdhsgm" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-zstream-upgrade-4-23-r8wjcwsdhsgm/providers/Microsoft.Resources/deployments/customer-infra-cluster-zstream-4-23-4q7sfm-ffc6p2/operationStatuses/08584085034054152549
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"InternalServerError\",\r\n        \"message\": \"An error occurred.\",\r\n        \"details\": []\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-zstream-upgrade-4-23-r8wjcwsdhsgm/providers/Microsoft.Resources/deployments/customer-infra-cluster-zstream-4-23-4q7sfm-ffc6p2/operationStatuses/08584085034054152549
            correlation ID: adb21106-8945-4c8c-8718-f7a2e847f630
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:108]: failed to create customer resources for z-stream cluster "cluster-zstream-4-23-4q7sfm"
Unexpected error:
    <*fmt.wrapError | 0x1abe3e2ce320>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-zstream-4-23-4q7sfm-ffc6p2" in resourcegroup="rg-zstream-upgrade-4-23-r8wjcwsdhsgm" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-zstream-upgrade-4-23-r8wjcwsdhsgm/providers/Microsoft.Resources/deployments/customer-infra-cluster-zstream-4-23-4q7sfm-ffc6p2/operationStatuses/08584085034054152549
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"InternalServerError\",\r\n        \"message\": \"An error occurred.\",\r\n        \"details\": []\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-zstream-upgrade-4-23-r8wjcwsdhsgm/providers/Microsoft.Resources/deployments/customer-infra-cluster-zstream-4-23-4q7sfm-ffc6p2/operationStatuses/08584085034054152549
            correlation ID: adb21106-8945-4c8c-8718-f7a2e847f630
            client request ID: unavailable
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.23	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-27 11:16	PR #7768	2115039807775182848

ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<u...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '10' ... message truncated; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-m4xxs2" for cluster "cluster-candidate-4-22-m4xxs2"
Unexpected error:
    <*fmt.wrapError | 0xfe103ee9c0>: 
    failed to create NodePool np-m4xxs2: failed waiting for nodepool="np-m4xxs2" for cluster "cluster-candidate-4-22-m4xxs2" in resourcegroup="rg-candidate-4-22-9hlfn44n2sw7" to finish creating: GET https://rp.j4902016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/fd0c6ffb-fa75-4ec9-bacd-47340ee2ccc0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/fd0c6ffb-fa75-4ec9-bacd-47340ee2ccc0",
      "name": "fd0c6ffb-fa75-4ec9-bacd-47340ee2ccc0",
      "status": "Failed",
      "startTime": "2026-10-27T15:26:37.894672503Z",
      "endTime": "2026-10-27T15:45:41.452550561Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine m5b2t7u3p0b4b5s-np-m4xxs2-7dxm2-5dghn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-9hlfn44n2sw7-candidate-m4xxs2--managed/providers/Microsoft.Compute/virtualMachines/m5b2t7u3p0b4b5s-np-m4xxs2-7dxm2-5dghn to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine m5b2t7u3p0b4b5s-np-m4xxs2-7dxm2-5dghn: Failed: interfaces failed to create or update. err: failed to get existing resource rg-candidate-4-22-9hlfn44n2sw7-candidate-m4xxs2--managed/m5b2t7u3p0b4b5s-np-m4xxs2-7dxm2-5dghn-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-9hlfn44n2sw7-candidate-m4xxs2--managed/providers/Microsoft.Network/networkInterfaces/m5b2t7u3p0b4b5s-np-m4xxs2-7dxm2-5dghn-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '39eefc4f-5e24-491d-9f09-bcc0ad573bcf' exceeded. Please try again after '10' ... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j4902016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/fd0c6ffb-fa75-4ec9-bacd-47340ee2ccc0
            correlation ID: c84286ce-64a0-40f9-bed8-eadb6c3c019f
            client request ID: 39555e0e-ee7f-4ff0-9056-9ab0bae5c62d
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-m4xxs2" for cluster "cluster-candidate-4-22-m4xxs2"
Unexpected error:
    <*fmt.wrapError | 0xfe103ee9c0>: 
    failed to create NodePool np-m4xxs2: failed waiting for nodepool="np-m4xxs2" for cluster "cluster-candidate-4-22-m4xxs2" in resourcegroup="rg-candidate-4-22-9hlfn44n2sw7" to finish creating: GET https://rp.j4902016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/fd0c6ffb-fa75-4ec9-bacd-47340ee2ccc0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/fd0c6ffb-fa75-4ec9-bacd-47340ee2ccc0",
      "name": "fd0c6ffb-fa75-4ec9-bacd-47340ee2ccc0",
      "status": "Failed",
      "startTime": "2026-10-27T15:26:37.894672503Z",
      "endTime": "2026-10-27T15:45:41.452550561Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine m5b2t7u3p0b4b5s-np-m4xxs2-7dxm2-5dghn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-9hlfn44n2sw7-candidate-m4xxs2--managed/providers/Microsoft.Compute/virtualMachines/m5b2t7u3p0b4b5s-np-m4xxs2-7dxm2-5dghn to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine m5b2t7u3p0b4b5s-np-m4xxs2-7dxm2-5dghn: Failed: interfaces failed to create or update. err: failed to get existing resource rg-candidate-4-22-9hlfn44n2sw7-candidate-m4xxs2--managed/m5b2t7u3p0b4b5s-np-m4xxs2-7dxm2-5dghn-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-9hlfn44n2sw7-candidate-m4xxs2--managed/providers/Microsoft.Network/networkInterfaces/m5b2t7u3p0b4b5s-np-m4xxs2-7dxm2-5dghn-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '39eefc4f-5e24-491d-9f09-bcc0ad573bcf' exceeded. Please try again after '10' ... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j4902016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/fd0c6ffb-fa75-4ec9-bacd-47340ee2ccc0
            correlation ID: c84286ce-64a0-40f9-bed8-eadb6c3c019f
            client request ID: 39555e0e-ee7f-4ff0-9056-9ab0bae5c62d
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-27 14:16	PR #7769	2115085319714902016

failed to roll out Helm release: failed post-install: resource Job/multicluster-engine/finalize-mce-config not ready. status: InProgress, message: Job in progress context deadline...

full failure pattern:

failed to roll out Helm release: failed post-install: resource Job/multicluster-engine/finalize-mce-config not ready. status: InProgress, message: Job in progress context deadline exceeded

Signal:

provision

none

Full failure examples (1)

time=2026-10-27T17:52:24.476Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config stamp=2 description="Step deploy-mce-config\n Kind: Helm\n"
time=2026-10-27T17:52:24.848Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config stamp=2 values="map[finalizeMceJobImage:mcr.microsoft.com/aks/command/runtime@sha256:37df117f1179d78fd372fd35878d53ff30dfb350fb571fb197f3a2bee48bd157 global:map[imageOverrides:map[governance_policy_propagator:governance-policy-propagator-acm-216@sha256:deb737b19932d520553c902ee5e0738ebeaad4592e8aa9649c41fec6fc00b62e] registryOverride:arohcpsvcdev.azurecr.io/acm-d-cache] localCluster:map[addonConfig:map[applicationManagerEnabled:false certPolicyControllerEnabled:false policyControllerEnabled:true searchCollectorEnabled:false] kubeApiUrl:https://kubernetes.default.svc] policy:map[cluster-lifecycle:map[hubconfig:map[replicaCount:2]] grc:map[hubconfig:map[replicaCount:2]]]]"
time=2026-10-27T17:52:24.848Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config stamp=2
time=2026-10-27T17:52:24.880Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config stamp=2 namespace=multicluster-engine
time=2026-10-27T17:52:24.933Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config stamp=2 dryRun=false
time=2026-10-27T17:52:24.933Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config stamp=2
time=2026-10-27T17:52:24.990Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config stamp=2
time=2026-10-27T18:02:49.740Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config stamp=2 err="failed post-install: resource Job/multicluster-engine/finalize-mce-config not ready. status: InProgress, message: Job in progress\ncontext deadline exceeded"
time=2026-10-27T18:02:49.741Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config stamp=2 err="stamp 2: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed post-install: resource Job/multicluster-engine/finalize-mce-config not ready. status: InProgress, message: Job in progress\ncontext deadline exceeded"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.ACM/management/deploy-mce-config (stamp=2)	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-27 17:04	PR #7753	2115127534017318912

failed to roll out Helm release: resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1 context de...

full failure pattern:

failed to roll out Helm release: resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1 context deadline exceeded

Signal:

provision

none

Full failure examples (1)

time=2026-10-27T15:06:00.351Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 description="Step deploy\n Kind: Helm\n"
time=2026-10-27T15:06:00.927Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 values="map[controllerName:secrets-store-sync-controller-manager image:arohcpsvcdev.azurecr.io/secrets-store-sync/controller@sha256:72abeb657cfbdd6a5891fc102a16429875d70c5932ad5994524d0246db914013 imagePullPolicy:IfNotPresent metricsPort:8085 providerContainer:[map[args:[--endpoint=unix:///provider/azure.sock --construct-pem-chain=true --healthz-port=8989 --healthz-path=/healthz --healthz-timeout=5s] image:mcr.microsoft.com/oss/v2/azure/secrets-store/provider-azure@sha256:55a0ad895a8f10193bb070883194449f5252c057353887af1ba0767760a0d2f0 imagePullPolicy:IfNotPresent livenessProbe:map[failureThreshold:3 httpGet:map[path:/healthz port:8989] initialDelaySeconds:5 periodSeconds:30 timeoutSeconds:10] name:provider-azure-installer ports:[map[containerPort:8898 name:metrics protocol:TCP]] resources:map[limits:map[memory:unlimited] requests:map[cpu:50m memory:100Mi]] securityContext:map[allowPrivilegeEscalation:false capabilities:map[drop:[ALL]] readOnlyRootFilesystem:true runAsUser:0] volumeMounts:[map[mountPath:/provider name:providervol]]]] tokenRequestAudience:[map[audience:<nil>]] validatingAdmissionPolicies:map[allowedSecretTypes:[Opaque kubernetes.io/basic-auth bootstrap.kubernetes.io/token kubernetes.io/dockerconfigjson kubernetes.io/dockercfg kubernetes.io/ssh-auth kubernetes.io/tls] applyPolicies:true deniedSecretTypes:[kubernetes.io/service-account-token]]]"
time=2026-10-27T15:06:00.927Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2
time=2026-10-27T15:06:01.327Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 namespace=secret-sync-controller
time=2026-10-27T15:06:01.636Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 dryRun=false
time=2026-10-27T15:06:01.636Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2
time=2026-10-27T15:06:01.765Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2
time=2026-10-27T15:11:07.858Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 err="resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded"
time=2026-10-27T15:11:07.859Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 err="stamp 2: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.SecretSyncController/management/deploy (stamp=2)	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-27 14:16	PR #6823	2115085218409877504