CIHealth Failure Patterns

Environment: DEV

Failure patterns: 53 · Failures matched: 90 · Total runs: 83 · Runs affected: 50

Failure pattern

Failed at

Trend

Also in

ERROR CODE: InvalidResource; detail message EtcdEncryptionKMSKeyAccessValidation: KeyVaultNetworkAccessDenied: The control plane cannot reach etcd encryption key "etcd-data-kms-en...

full failure pattern:

ERROR CODE: InvalidResource; detail message EtcdEncryptionKMSKeyAccessValidation: KeyVaultNetworkAccessDenied: The control plane cannot reach etcd encryption key "etcd-data-kms-encryption-key" in Key Vault "cust-kv-<id>": Key Vault responded with HTTP 403 (ForbiddenByConnection): Public network access is disabled and request is not from a trusted service nor via an approved private link. Clusters created with etcd.dataEncryption.customerManaged.kms.visibility "Private" reach the Key Vault from the virtual network that contains the cluster's VNet integration subnet. Ensure the Key Vault has an approved private endpoint, and that a privatelink.vaultcore.azure.net private DNS zone with a record for the Key Vault is linked to that virtual network.; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xda1b3d9a500>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-rlbt6k884zhh" to finish creating: GET https://rp.j9033216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/90fc1494-ffd6-415a-ac40-0adc1fc596dc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidResource
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/90fc1494-ffd6-415a-ac40-0adc1fc596dc",
      "name": "90fc1494-ffd6-415a-ac40-0adc1fc596dc",
      "status": "Failed",
      "startTime": "2026-10-21T21:21:31.114383608Z",
      "endTime": "2026-10-21T21:37:30.726056138Z",
      "error": {
        "code": "InvalidResource",
        "message": "EtcdEncryptionKMSKeyAccessValidation: KeyVaultNetworkAccessDenied: The control plane cannot reach etcd encryption key \"etcd-data-kms-encryption-key\" in Key Vault \"cust-kv-pa2z63bxwn3cs\": Key Vault responded with HTTP 403 (ForbiddenByConnection): Public network access is disabled and request is not from a trusted service nor via an approved private link. Clusters created with etcd.dataEncryption.customerManaged.kms.visibility \"Private\" reach the Key Vault from the virtual network that contains the cluster's VNet integration subnet. Ensure the Key Vault has an approved private endpoint, and that a privatelink.vaultcore.azure.net private DNS zone with a record for the Key Vault is linked to that virtual network."
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9033216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/90fc1494-ffd6-415a-ac40-0adc1fc596dc
            correlation ID: df67305e-b659-4d48-9094-7c36da447124
            client request ID: 406a6784-9511-4782-b611-e92af52ad4ab
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xda1b3d9a500>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-rlbt6k884zhh" to finish creating: GET https://rp.j9033216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/90fc1494-ffd6-415a-ac40-0adc1fc596dc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidResource
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/90fc1494-ffd6-415a-ac40-0adc1fc596dc",
      "name": "90fc1494-ffd6-415a-ac40-0adc1fc596dc",
      "status": "Failed",
      "startTime": "2026-10-21T21:21:31.114383608Z",
      "endTime": "2026-10-21T21:37:30.726056138Z",
      "error": {
        "code": "InvalidResource",
        "message": "EtcdEncryptionKMSKeyAccessValidation: KeyVaultNetworkAccessDenied: The control plane cannot reach etcd encryption key \"etcd-data-kms-encryption-key\" in Key Vault \"cust-kv-pa2z63bxwn3cs\": Key Vault responded with HTTP 403 (ForbiddenByConnection): Public network access is disabled and request is not from a trusted service nor via an approved private link. Clusters created with etcd.dataEncryption.customerManaged.kms.visibility \"Private\" reach the Key Vault from the virtual network that contains the cluster's VNet integration subnet. Ensure the Key Vault has an approved private endpoint, and that a privatelink.vaultcore.azure.net private DNS zone with a record for the Key Vault is linked to that virtual network."
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9033216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/90fc1494-ffd6-415a-ac40-0adc1fc596dc
            correlation ID: df67305e-b659-4d48-9094-7c36da447124
            client request ID: 406a6784-9511-4782-b611-e92af52ad4ab
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0x2ff3aa19a440>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-mjjsfb5sc9cl" to finish creating: GET https://rp.j9033216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0b471f6-0b3c-4f58-9820-fa7265add151
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidResource
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0b471f6-0b3c-4f58-9820-fa7265add151",
      "name": "f0b471f6-0b3c-4f58-9820-fa7265add151",
      "status": "Failed",
      "startTime": "2026-10-21T21:21:31.052814653Z",
      "endTime": "2026-10-21T21:37:10.593291759Z",
      "error": {
        "code": "InvalidResource",
        "message": "EtcdEncryptionKMSKeyAccessValidation: KeyVaultNetworkAccessDenied: The control plane cannot reach etcd encryption key \"etcd-data-kms-encryption-key\" in Key Vault \"cust-kv-om5carnxwh2f6\": Key Vault responded with HTTP 403 (ForbiddenByConnection): Public network access is disabled and request is not from a trusted service nor via an approved private link. Clusters created with etcd.dataEncryption.customerManaged.kms.visibility \"Private\" reach the Key Vault from the virtual network that contains the cluster's VNet integration subnet. Ensure the Key Vault has an approved private endpoint, and that a privatelink.vaultcore.azure.net private DNS zone with a record for the Key Vault is linked to that virtual network."
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9033216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0b471f6-0b3c-4f58-9820-fa7265add151
            correlation ID: c7b4c1ab-73f1-409f-8dbc-b609459a629b
            client request ID: 2e59ce2b-1874-419f-a049-3854aa659093
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0x2ff3aa19a440>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-mjjsfb5sc9cl" to finish creating: GET https://rp.j9033216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0b471f6-0b3c-4f58-9820-fa7265add151
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidResource
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0b471f6-0b3c-4f58-9820-fa7265add151",
      "name": "f0b471f6-0b3c-4f58-9820-fa7265add151",
      "status": "Failed",
      "startTime": "2026-10-21T21:21:31.052814653Z",
      "endTime": "2026-10-21T21:37:10.593291759Z",
      "error": {
        "code": "InvalidResource",
        "message": "EtcdEncryptionKMSKeyAccessValidation: KeyVaultNetworkAccessDenied: The control plane cannot reach etcd encryption key \"etcd-data-kms-encryption-key\" in Key Vault \"cust-kv-om5carnxwh2f6\": Key Vault responded with HTTP 403 (ForbiddenByConnection): Public network access is disabled and request is not from a trusted service nor via an approved private link. Clusters created with etcd.dataEncryption.customerManaged.kms.visibility \"Private\" reach the Key Vault from the virtual network that contains the cluster's VNet integration subnet. Ensure the Key Vault has an approved private endpoint, and that a privatelink.vaultcore.azure.net private DNS zone with a record for the Key Vault is linked to that virtual network."
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9033216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0b471f6-0b3c-4f58-9820-fa7265add151
            correlation ID: c7b4c1ab-73f1-409f-8dbc-b609459a629b
            client request ID: 2e59ce2b-1874-419f-a049-3854aa659093
occurred

Contributing tests (2)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-21 20:26	PR #7394	2113003926449033216

ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed

full failure pattern:

ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:509]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x36772c952c50>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:509]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x36772c952c50>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 5.1	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-21 12:58	PR #7635	2112891349849083904

ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<u...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds a... message truncated; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0xe9fe1d4cc20>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-drk5zr95nhk4" to finish creating: GET https://rp.j6828160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e4b75858-d08d-4fff-b2a6-1eaa8bb64568
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e4b75858-d08d-4fff-b2a6-1eaa8bb64568",
      "name": "e4b75858-d08d-4fff-b2a6-1eaa8bb64568",
      "status": "Failed",
      "startTime": "2026-10-21T19:03:42.372924526Z",
      "endTime": "2026-10-21T19:22:50.94711189Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine u2g8b3t8g2z9g0c-np-1-98sr2-8zrxf: Failed: availabilitysets failed to create or update. err: failed to get existing resource fips-enabled-drk5zr95nhk4--managed/2tjbpg3o68qrhrlgableu4g1r57ub3r2_u2g8b3t8g2z9g0c-np-1-as (service: availabilitysets): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-drk5zr95nhk4--managed/providers/Microsoft.Compute/availabilitySets/2tjbpg3o68qrhrlgableu4g1r57ub3r2_u2g8b3t8g2z9g0c-np-1-as\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'cd39c606-1f6a-4062-a5b9-497cd04c39fc' exceeded. Please try again after '1' seconds a... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6828160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e4b75858-d08d-4fff-b2a6-1eaa8bb64568
            correlation ID: 89bccd47-951a-48a6-a441-8b8045c21068
            client request ID: 9c8a9209-4abc-4bd6-b4b0-28c6073afcb6
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0xe9fe1d4cc20>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-drk5zr95nhk4" to finish creating: GET https://rp.j6828160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e4b75858-d08d-4fff-b2a6-1eaa8bb64568
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e4b75858-d08d-4fff-b2a6-1eaa8bb64568",
      "name": "e4b75858-d08d-4fff-b2a6-1eaa8bb64568",
      "status": "Failed",
      "startTime": "2026-10-21T19:03:42.372924526Z",
      "endTime": "2026-10-21T19:22:50.94711189Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine u2g8b3t8g2z9g0c-np-1-98sr2-8zrxf: Failed: availabilitysets failed to create or update. err: failed to get existing resource fips-enabled-drk5zr95nhk4--managed/2tjbpg3o68qrhrlgableu4g1r57ub3r2_u2g8b3t8g2z9g0c-np-1-as (service: availabilitysets): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-drk5zr95nhk4--managed/providers/Microsoft.Compute/availabilitySets/2tjbpg3o68qrhrlgableu4g1r57ub3r2_u2g8b3t8g2z9g0c-np-1-as\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'cd39c606-1f6a-4062-a5b9-497cd04c39fc' exceeded. Please try again after '1' seconds a... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6828160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e4b75858-d08d-4fff-b2a6-1eaa8bb64568
            correlation ID: 89bccd47-951a-48a6-a441-8b8045c21068
            client request ID: 9c8a9209-4abc-4bd6-b4b0-28c6073afcb6
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-21 17:42	PR #7428	2112962887956828160

ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<u...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are... message truncated; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:112]: failed to create node pool "np-tk5gq6" for cluster "cluster-hs-pre-tk5gq6"
Unexpected error:
    <*fmt.wrapError | 0x28a51cc17480>: 
    failed to create NodePool np-tk5gq6: failed waiting for nodepool="np-tk5gq6" for cluster "cluster-hs-pre-tk5gq6" in resourcegroup="rg-hs-pre--c25l8fnlxbfp" to finish creating: GET https://rp.j2080256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f420461f-bece-47f6-898e-248327371b0b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f420461f-bece-47f6-898e-248327371b0b",
      "name": "f420461f-bece-47f6-898e-248327371b0b",
      "status": "Failed",
      "startTime": "2026-10-21T06:16:15.392203282Z",
      "endTime": "2026-10-21T06:35:22.529847577Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine b1f8d3l9s7p6k7c-np-tk5gq6-8t786-lnwcz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--c25l8fnlxbfp-tk5gq6--managed/providers/Microsoft.Compute/virtualMachines/b1f8d3l9s7p6k7c-np-tk5gq6-8t786-lnwcz to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine b1f8d3l9s7p6k7c-np-tk5gq6-8t786-lnwcz: Failed: interfaces failed to create or update. err: failed to get existing resource rg-hs-pre--c25l8fnlxbfp-tk5gq6--managed/b1f8d3l9s7p6k7c-np-tk5gq6-8t786-lnwcz-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--c25l8fnlxbfp-tk5gq6--managed/providers/Microsoft.Network/networkInterfaces/b1f8d3l9s7p6k7c-np-tk5gq6-8t786-lnwcz-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '5a34d2fd-f7db-460a-a272-334006a8a3b8' exceeded. Please try again after '1' seconds after additional tokens are... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2080256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f420461f-bece-47f6-898e-248327371b0b
            correlation ID: babfafd3-932a-4646-be0e-e8e3feb4e4fd
            client request ID: 4b55642a-8b13-4351-9120-9e71849c75ec
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:112]: failed to create node pool "np-tk5gq6" for cluster "cluster-hs-pre-tk5gq6"
Unexpected error:
    <*fmt.wrapError | 0x28a51cc17480>: 
    failed to create NodePool np-tk5gq6: failed waiting for nodepool="np-tk5gq6" for cluster "cluster-hs-pre-tk5gq6" in resourcegroup="rg-hs-pre--c25l8fnlxbfp" to finish creating: GET https://rp.j2080256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f420461f-bece-47f6-898e-248327371b0b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f420461f-bece-47f6-898e-248327371b0b",
      "name": "f420461f-bece-47f6-898e-248327371b0b",
      "status": "Failed",
      "startTime": "2026-10-21T06:16:15.392203282Z",
      "endTime": "2026-10-21T06:35:22.529847577Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine b1f8d3l9s7p6k7c-np-tk5gq6-8t786-lnwcz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--c25l8fnlxbfp-tk5gq6--managed/providers/Microsoft.Compute/virtualMachines/b1f8d3l9s7p6k7c-np-tk5gq6-8t786-lnwcz to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine b1f8d3l9s7p6k7c-np-tk5gq6-8t786-lnwcz: Failed: interfaces failed to create or update. err: failed to get existing resource rg-hs-pre--c25l8fnlxbfp-tk5gq6--managed/b1f8d3l9s7p6k7c-np-tk5gq6-8t786-lnwcz-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--c25l8fnlxbfp-tk5gq6--managed/providers/Microsoft.Network/networkInterfaces/b1f8d3l9s7p6k7c-np-tk5gq6-8t786-lnwcz-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '5a34d2fd-f7db-460a-a272-334006a8a3b8' exceeded. Please try again after '1' seconds after additional tokens are... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2080256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f420461f-bece-47f6-898e-248327371b0b
            correlation ID: babfafd3-932a-4646-be0e-e8e3feb4e4fd
            client request ID: 4b55642a-8b13-4351-9120-9e71849c75ec
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-21 05:01	PR #7097	2112771199242080256

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want >= <count> (autoscaling min); node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-<id> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-<id> to exist; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x12fed7d17db8>: 
    VerifyNodesReady failed: there are not ready nodes: agg-cluster-agg-np-a-7m9bd-99rcc (status=False, reason=KubeletNotReady, message=[PLEG is not healthy: pleg has yet to be successful, container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?, CSINode is not yet initialized], since=2026-10-21T06:37:27Z)
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-sskzwsj6mcw6" to finish creating: GET https://rp.j2080256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5549a46b-1e5e-4888-bbe8-6d3e268cb178
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5549a46b-1e5e-4888-bbe8-6d3e268cb178",
      "name": "5549a46b-1e5e-4888-bbe8-6d3e268cb178",
      "status": "Failed",
      "startTime": "2026-10-21T06:17:49.830044932Z",
      "endTime": "2026-10-21T06:36:52.904336002Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-7m9bd-99rcc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-sskzwsj6mcw6--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-7m9bd-99rcc to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-7m9bd-99rcc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-sskzwsj6mcw6--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-7m9bd-99rcc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2080256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5549a46b-1e5e-4888-bbe8-6d3e268cb178
            correlation ID: d7122e31-231d-46b1-9fc6-c124c9ad9057
            client request ID: b05e45db-d8fe-4015-bf19-f42edb0a0ae5
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x12fed7d17db8>: 
    VerifyNodesReady failed: there are not ready nodes: agg-cluster-agg-np-a-7m9bd-99rcc (status=False, reason=KubeletNotReady, message=[PLEG is not healthy: pleg has yet to be successful, container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?, CSINode is not yet initialized], since=2026-10-21T06:37:27Z)
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-sskzwsj6mcw6" to finish creating: GET https://rp.j2080256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5549a46b-1e5e-4888-bbe8-6d3e268cb178
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5549a46b-1e5e-4888-bbe8-6d3e268cb178",
      "name": "5549a46b-1e5e-4888-bbe8-6d3e268cb178",
      "status": "Failed",
      "startTime": "2026-10-21T06:17:49.830044932Z",
      "endTime": "2026-10-21T06:36:52.904336002Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-7m9bd-99rcc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-sskzwsj6mcw6--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-7m9bd-99rcc to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-7m9bd-99rcc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-sskzwsj6mcw6--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-7m9bd-99rcc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2080256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5549a46b-1e5e-4888-bbe8-6d3e268cb178
            correlation ID: d7122e31-231d-46b1-9fc6-c124c9ad9057
            client request ID: b05e45db-d8fe-4015-bf19-f42edb0a0ae5
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-21 05:01	PR #7097	2112771199242080256

ERROR CODE: DeploymentFailed; detail code NotFound; detail message Resource '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.ManagedIdentity/user...

full failure pattern:

ERROR CODE: DeploymentFailed; detail code NotFound; detail message Resource '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/logs-mdsd' was not found.; provider Microsoft.ManagedIdentity

Signal:

provision

none

Full failure examples (1)

time=2026-10-21T16:44:29.860Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 description="Step cluster\n  Kind: ARM\n  Template: templates/mgmt-cluster.bicep\n  Parameters: configurations/mgmt-cluster.tmpl.bicepparam"
time=2026-10-21T16:44:30.303Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2
time=2026-10-21T16:44:30.918Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 deployment=64152644a53dcaa071ce9ff6422266013cd78714911c11a5dae06a36893119aa portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j6080768-mgmt-2%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F64152644a53dcaa071ce9ff6422266013cd78714911c11a5dae06a36893119aa
time=2026-10-21T16:46:31.557Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 err="stamp 2: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6080768-mgmt-2/providers/Microsoft.Resources/deployments/64152644a53dcaa071ce9ff6422266013cd78714911c11a5dae06a36893119aa/operationStatuses/08584090058149128475\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6080768-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-post\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"NotFound\\\",\\r\\n            \\\"message\\\": \\\"Resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6080768-mgmt-2/providers/Microsoft.ManagedIdentity/userAssignedIdentities/logs-mdsd' was not found.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      },\n      {\n        \"code\": \"NotFound\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"NotFound\\\",\\r\\n    \\\"message\\\": \\\"Resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6080768-mgmt-2/providers/Microsoft.ManagedIdentity/userAssignedIdentities/logs-mdsd' was not found.\\\"\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster (stamp=2)	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-21 16:11	PR #7394	2112939820866080768

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...

full failure pattern:

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [resourceTeardown] tearing down namespaces; [descendantResources] remaining resources: externalAuths, serviceProviderClusters, serviceProviderExternalAuths; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
    <*errors.joinError | 0xd2af1258138>: 
    resource group "rg-candidate-4-20-hth2d5sgmqkq": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-20-82k925" in resourcegroup="rg-candidate-4-20-hth2d5sgmqkq" to finish deleting: GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ff5d427f-8d05-44df-b7ea-04a0f7376b37
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ff5d427f-8d05-44df-b7ea-04a0f7376b37",
      "name": "ff5d427f-8d05-44df-b7ea-04a0f7376b37",
      "status": "Failed",
      "startTime": "2026-10-21T14:40:14.70614785Z",
      "endTime": "2026-10-21T15:04:22.445579694Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tj7kg5i5v3qeqbhhu1aj4t26p20km4l still exists (deletion dispatched at 2026-10-21T14:40:16Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 microsoft.redhatopenshift/hcpopenshiftclusters/externalauths/serviceProviderExternalAuths, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ff5d427f-8d05-44df-b7ea-04a0f7376b37
            correlation ID: fc2328c7-eacc-4077-91a9-9b41a9c39453
            client request ID: 7963c683-9f71-48b3-b976-d344efc9d8b4
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
    <*errors.joinError | 0xd2af1258138>: 
    resource group "rg-candidate-4-20-hth2d5sgmqkq": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-20-82k925" in resourcegroup="rg-candidate-4-20-hth2d5sgmqkq" to finish deleting: GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ff5d427f-8d05-44df-b7ea-04a0f7376b37
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ff5d427f-8d05-44df-b7ea-04a0f7376b37",
      "name": "ff5d427f-8d05-44df-b7ea-04a0f7376b37",
      "status": "Failed",
      "startTime": "2026-10-21T14:40:14.70614785Z",
      "endTime": "2026-10-21T15:04:22.445579694Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tj7kg5i5v3qeqbhhu1aj4t26p20km4l still exists (deletion dispatched at 2026-10-21T14:40:16Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 microsoft.redhatopenshift/hcpopenshiftclusters/externalauths/serviceProviderExternalAuths, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ff5d427f-8d05-44df-b7ea-04a0f7376b37
            correlation ID: fc2328c7-eacc-4077-91a9-9b41a9c39453
            client request ID: 7963c683-9f71-48b3-b976-d344efc9d8b4
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-21 12:58	PR #7635	2112891349849083904

ERROR CODE: InvalidResource; detail message EtcdEncryptionKMSKeyAccessValidation: KeyVaultNetworkAccessDenied: The control plane cannot reach etcd encryption key "etcd-data-kms-en...

full failure pattern:

ERROR CODE: InvalidResource; detail message EtcdEncryptionKMSKeyAccessValidation: KeyVaultNetworkAccessDenied: The control plane cannot reach etcd encryption key "etcd-data-kms-encryption-key" in Key Vault "cust-kv-4j7orvssmvpqc": Key Vault responded with HTTP 403 (ForbiddenByConnection): Public network access is disabled and request is not from a trusted service nor via an approved private link. Clusters created with etcd.dataEncryption.customerManaged.kms.visibility "Private" reach the Key Vault from the virtual network that contains the cluster's VNet integration subnet. Ensure the Key Vault has an approved private endpoint, and that a privatelink.vaultcore.azure.net private DNS zone with a record for the Key Vault is linked to that virtual network.; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0x3fddd609a460>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-zlfmgtnzzr42" to finish creating: GET https://rp.j9033216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1d4c1fe4-fcd1-4c1a-9e7c-e514ce812ccd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidResource
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1d4c1fe4-fcd1-4c1a-9e7c-e514ce812ccd",
      "name": "1d4c1fe4-fcd1-4c1a-9e7c-e514ce812ccd",
      "status": "Failed",
      "startTime": "2026-10-21T21:28:36.459657793Z",
      "endTime": "2026-10-21T21:43:30.938568702Z",
      "error": {
        "code": "InvalidResource",
        "message": "EtcdEncryptionKMSKeyAccessValidation: KeyVaultNetworkAccessDenied: The control plane cannot reach etcd encryption key \"etcd-data-kms-encryption-key\" in Key Vault \"cust-kv-4j7orvssmvpqc\": Key Vault responded with HTTP 403 (ForbiddenByConnection): Public network access is disabled and request is not from a trusted service nor via an approved private link. Clusters created with etcd.dataEncryption.customerManaged.kms.visibility \"Private\" reach the Key Vault from the virtual network that contains the cluster's VNet integration subnet. Ensure the Key Vault has an approved private endpoint, and that a privatelink.vaultcore.azure.net private DNS zone with a record for the Key Vault is linked to that virtual network."
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9033216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1d4c1fe4-fcd1-4c1a-9e7c-e514ce812ccd
            correlation ID: f0c1fd73-2e2e-42e9-ba5b-c72a1b7faac0
            client request ID: 536018dd-5a4e-47e8-879c-f794371f0ffe
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0x3fddd609a460>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-zlfmgtnzzr42" to finish creating: GET https://rp.j9033216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1d4c1fe4-fcd1-4c1a-9e7c-e514ce812ccd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidResource
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1d4c1fe4-fcd1-4c1a-9e7c-e514ce812ccd",
      "name": "1d4c1fe4-fcd1-4c1a-9e7c-e514ce812ccd",
      "status": "Failed",
      "startTime": "2026-10-21T21:28:36.459657793Z",
      "endTime": "2026-10-21T21:43:30.938568702Z",
      "error": {
        "code": "InvalidResource",
        "message": "EtcdEncryptionKMSKeyAccessValidation: KeyVaultNetworkAccessDenied: The control plane cannot reach etcd encryption key \"etcd-data-kms-encryption-key\" in Key Vault \"cust-kv-4j7orvssmvpqc\": Key Vault responded with HTTP 403 (ForbiddenByConnection): Public network access is disabled and request is not from a trusted service nor via an approved private link. Clusters created with etcd.dataEncryption.customerManaged.kms.visibility \"Private\" reach the Key Vault from the virtual network that contains the cluster's VNet integration subnet. Ensure the Key Vault has an approved private endpoint, and that a privatelink.vaultcore.azure.net private DNS zone with a record for the Key Vault is linked to that virtual network."
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9033216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1d4c1fe4-fcd1-4c1a-9e7c-e514ce812ccd
            correlation ID: f0c1fd73-2e2e-42e9-ba5b-c72a1b7faac0
            client request ID: 536018dd-5a4e-47e8-879c-f794371f0ffe
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-21 20:26	PR #7394	2113003926449033216

VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.002s: failed to list pods in namespace pullsecret-image-test: client rate limiter Wait returned an error: rat...

full failure pattern:

VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.002s: failed to list pods in namespace pullsecret-image-test: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:297]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0x24cdc40f4060>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.002s: failed to list pods in namespace pullsecret-image-test: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline
    {
        msg: "VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.002s: failed to list pods in namespace pullsecret-image-test: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline",
        err: <*fmt.wrapError | 0x24cdc4962880>{
            msg: "failed to list pods in namespace pullsecret-image-test: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline",
            err: <*fmt.wrapError | 0x24cdc4962840>{
                msg: "client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline",
                err: <*errors.errorString | 0x24cdc686cd30>{
                    s: "rate: Wait(n=1) would exceed context deadline",
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:297]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0x24cdc40f4060>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.002s: failed to list pods in namespace pullsecret-image-test: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline
    {
        msg: "VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.002s: failed to list pods in namespace pullsecret-image-test: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline",
        err: <*fmt.wrapError | 0x24cdc4962880>{
            msg: "failed to list pods in namespace pullsecret-image-test: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline",
            err: <*fmt.wrapError | 0x24cdc4962840>{
                msg: "client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline",
                err: <*errors.errorString | 0x24cdc686cd30>{
                    s: "rate: Wait(n=1) would exceed context deadline",
                },
            },
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-21 22:59	PR #7662	2113042472400588800

VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.002s: failed to list pods in namespace pullsecret-image-test: context deadline exceeded

full failure pattern:

VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.002s: failed to list pods in namespace pullsecret-image-test: context deadline exceeded

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:297]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0x266d4ce3a1e0>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.002s: failed to list pods in namespace pullsecret-image-test: context deadline exceeded
    {
        msg: "VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.002s: failed to list pods in namespace pullsecret-image-test: context deadline exceeded",
        err: <*fmt.wrapError | 0x266d4ce3a180>{
            msg: "failed to list pods in namespace pullsecret-image-test: context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:297]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0x266d4ce3a1e0>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.002s: failed to list pods in namespace pullsecret-image-test: context deadline exceeded
    {
        msg: "VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.002s: failed to list pods in namespace pullsecret-image-test: context deadline exceeded",
        err: <*fmt.wrapError | 0x266d4ce3a180>{
            msg: "failed to list pods in namespace pullsecret-image-test: context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-21 22:24	PR #7675	2113033770973007872

VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.008s: failed to list pods in namespace pullsecret-image-test: context deadline exceeded

full failure pattern:

VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.008s: failed to list pods in namespace pullsecret-image-test: context deadline exceeded

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:297]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0x550b080a240>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.008s: failed to list pods in namespace pullsecret-image-test: context deadline exceeded
    {
        msg: "VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.008s: failed to list pods in namespace pullsecret-image-test: context deadline exceeded",
        err: <*fmt.wrapError | 0x550b080a180>{
            msg: "failed to list pods in namespace pullsecret-image-test: context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:297]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0x550b080a240>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.008s: failed to list pods in namespace pullsecret-image-test: context deadline exceeded
    {
        msg: "VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.008s: failed to list pods in namespace pullsecret-image-test: context deadline exceeded",
        err: <*fmt.wrapError | 0x550b080a180>{
            msg: "failed to list pods in namespace pullsecret-image-test: context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-21 22:55	PR #7646	2113041621648936960

VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.024s: expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: image pull error...

full failure pattern:

VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.024s: expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: image pull errors:

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:297]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0x32bf9ef05a0>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.024s: expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: image pull errors:
      - pod pull-secret-test container pull-test: ImagePullBackOff - Back-off pulling image "registry.redhat.io/ubi9/ubi-minimal:latest": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://registry.redhat.io/ubi9/ubi-minimal:latest: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable; artifact err: get manifest: build image source: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable
    {
        msg: "VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.024s: expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: image pull errors:\n  - pod pull-secret-test container pull-test: ImagePullBackOff - Back-off pulling image \"registry.redhat.io/ubi9/ubi-minimal:latest\": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://registry.redhat.io/ubi9/ubi-minimal:latest: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable; artifact err: get manifest: build image source: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable",
        err: <*errors.errorString | 0x32bf9447a70>{
            s: "expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: image pull errors:\n  - pod pull-secret-test container pull-test: ImagePullBackOff - Back-off pulling image \"registry.redhat.io/ubi9/ubi-minimal:latest\": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://registry.redhat.io/ubi9/ubi-minimal:latest: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable; artifact err: get manifest: build image source: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable",
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:297]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0x32bf9ef05a0>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.024s: expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: image pull errors:
      - pod pull-secret-test container pull-test: ImagePullBackOff - Back-off pulling image "registry.redhat.io/ubi9/ubi-minimal:latest": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://registry.redhat.io/ubi9/ubi-minimal:latest: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable; artifact err: get manifest: build image source: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable
    {
        msg: "VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.024s: expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: image pull errors:\n  - pod pull-secret-test container pull-test: ImagePullBackOff - Back-off pulling image \"registry.redhat.io/ubi9/ubi-minimal:latest\": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://registry.redhat.io/ubi9/ubi-minimal:latest: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable; artifact err: get manifest: build image source: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable",
        err: <*errors.errorString | 0x32bf9447a70>{
            s: "expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: image pull errors:\n  - pod pull-secret-test container pull-test: ImagePullBackOff - Back-off pulling image \"registry.redhat.io/ubi9/ubi-minimal:latest\": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://registry.redhat.io/ubi9/ubi-minimal:latest: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable; artifact err: get manifest: build image source: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable",
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-21 23:03	PR #7177	2113043630527614976

alert [hcp] userJourneyKubeApiserverAvailability1h5m fired

full failure pattern:

alert [hcp] userJourneyKubeApiserverAvailability1h5m fired

Signal:

alert

none

Full failure examples (1)

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-21T15:38:57Z
  Severity:    Sev3
  Labels:      _id="e3c51a01-e77e-4b85-8495-f258f44ab7a0", alertname="userJourneyKubeApiserverAvailability1h5m", cluster="ci01-j9083904-mgmt-2", component="slo", long_window="1h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9083904/providers/microsoft.monitor/accounts/hcps-j9083904", name="s6i4e5m7h6r7f2g", namespace="ocm-arohcpci01-2tj7kg5i5v3qeqbhhu1aj4t26p20km4l", region="centralus", resource_id="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-candidate-4-20-hth2d5sgmqkq/providers/microsoft.redhatopenshift/hcpopenshiftclusters/cluster-candidate-4-20-82k925", severity="3", short_window="5m", slo="kas-availability"
  Description: Resource ID: /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-candidate-4-20-hth2d5sgmqkq/providers/microsoft.redhatopenshift/hcpopenshiftclusters/cluster-candidate-4-20-82k925
Management Cluster: ci01-j9083904-mgmt-2
Namespace: ocm-arohcpci01-2tj7kg5i5v3qeqbhhu1aj4t26p20km4l

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-21T15:42:04Z
  Severity:    Sev3
  Labels:      _id="001fac7e-54b1-459b-8e12-67588890841e", alertname="userJourneyKubeApiserverAvailability1h5m", cluster="ci01-j9083904-mgmt-1", component="slo", long_window="1h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9083904/providers/microsoft.monitor/accounts/hcps-j9083904", name="sre-hcp-cluster", namespace="ocm-arohcpci01-2tj7od9dtldg9atesqujpab56gdtktc3", region="centralus", resource_id="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/admin-api-breakglass-9q9nnh4t7882/providers/microsoft.redhatopenshift/hcpopenshiftclusters/sre-hcp-cluster", severity="3", short_window="5m", slo="kas-availability"
  Description: Resource ID: /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/admin-api-breakglass-9q9nnh4t7882/providers/microsoft.redhatopenshift/hcpopenshiftclusters/sre-hcp-cluster
Management Cluster: ci01-j9083904-mgmt-1
Namespace: ocm-arohcpci01-2tj7od9dtldg9atesqujpab56gdtktc3

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert userJourneyKubeApiserverAvailability1h5m does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-21 12:58	PR #7635	2112891349849083904

failed to list ClusterOperators: context deadline exceeded

full failure pattern:

failed to list ClusterOperators: context deadline exceeded

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:255]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0x5f255b9c0d8>: 
    context deadline exceeded
    failed to list ClusterOperators: context deadline exceeded
    {
        errs: [
            <context.deadlineExceededError>{},
            <*fmt.wrapError | 0x5f2544743e0>{
                msg: "failed to list ClusterOperators: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:255]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0x5f255b9c0d8>: 
    context deadline exceeded
    failed to list ClusterOperators: context deadline exceeded
    {
        errs: [
            <context.deadlineExceededError>{},
            <*fmt.wrapError | 0x5f2544743e0>{
                msg: "failed to list ClusterOperators: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-21 11:34	PR #7658	2112870236943290368

missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per source, missing: [maestro]; observed counts: maestro=0, hypershift=11048, acm-agent=...

full failure pattern:

missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per source, missing: [maestro]; observed counts: maestro=0, hypershift=11048, acm-agent=207

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 600.001s.
Expected success, but got an error:
    <*errors.errorString | 0x2df52e691b0>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per source, missing: [maestro]; observed counts: maestro=0, hypershift=11048, acm-agent=207
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per source, missing: [maestro]; observed counts: maestro=0, hypershift=11048, acm-agent=207",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 600.001s.
Expected success, but got an error:
    <*errors.errorString | 0x2df52e691b0>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per source, missing: [maestro]; observed counts: maestro=0, hypershift=11048, acm-agent=207
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per source, missing: [maestro]; observed counts: maestro=0, hypershift=11048, acm-agent=207",
    }

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-21 16:58	PR #7665	2112951578825592832

alert [hcp] KubePodNotReady fired

full failure pattern:

alert [hcp] KubePodNotReady fired

Signal:

alert

none

Full failure examples (3)

alert fired 40 time(s) (17 unknown, 23 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-21T19:17:35Z
  Ended:       2026-10-21T19:27:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1024128-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tjbvidupgp9nlik7eg10fsh59vogi9q-k9q7k5x7d5s9g6p", pod="etcd-0", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tjbvidupgp9nlik7eg10fsh59vogi9q-k9q7k5x7d5s9g6p/etcd-0 has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-21T19:18:35Z
  Ended:       2026-10-21T19:29:35Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1024128-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tjbvo8p5rl6fb29feg2u1dlsmppl5br-cilium-cl", pod="etcd-2", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tjbvo8p5rl6fb29feg2u1dlsmppl5br-cilium-cl/etcd-2 has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-21T19:19:34Z
  Ended:       2026-10-21T19:29:35Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1024128-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tjbvrsgfcv134idooo6qgdvr42t8n6n-sre-hcp-cluster", pod="etcd-2", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tjbvrsgfcv134idooo6qgdvr42t8n6n-sre-hcp-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-10-21T19:20:33Z
  Ended:       2026-10-21T19:28:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1024128-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tjbvv9tadp00611k0lgs7t2avvomnlg-l4m3x6u1z1a0c9u", pod="etcd-0", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tjbvv9tadp00611k0lgs7t2avvomnlg-l4m3x6u1z1a0c9u/etcd-0 has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-10-21T19:20:33Z
  Ended:       2026-10-21T19:29:35Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1024128-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tjc0bg3vri3dhs1ag8ialdt811kjgsr-n4j7n9z1g6c5x9u", pod="etcd-0", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tjc0bg3vri3dhs1ag8ialdt811kjgsr-n4j7n9z1g6c5x9u/etcd-0 has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-10-21T19:20:33Z
  Ended:       2026-10-21T19:28:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1024128-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tjc1se0ft7nkghlt40iu0k36buchj61-t8n1u1u8g5n4s0q", pod="etcd-0", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tjc1se0ft7nkghlt40iu0k36buchj61-t8n1u1u8g5n4s0q/etcd-0 has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-10-21T19:20:33Z
  Ended:       2026-10-21T19:29:35Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1024128-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tjc0bbun58ddhmh76bq5nrgimbkrbfu-j5e9k6c2t1w8y4y", pod="etcd-0", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tjc0bbun58ddhmh76bq5nrgimbkrbfu-j5e9k6c2t1w8y4y/etcd-0 has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-10-21T19:20:33Z
  Ended:       2026-10-21T19:27:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1024128-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tjbvutvgtgr8umvqql88ajad4lgh83d-ea-cluster", pod="etcd-2", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tjbvutvgtgr8umvqql88ajad4lgh83d-ea-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-10-21T19:20:33Z
  Ended:       2026-10-21T19:28:35Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1024128-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tjbvu2jghqqammopbihdm0dua266fi5-private-ingress", pod="etcd-0", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tjbvu2jghqqammopbihdm0dua266fi5-private-ingress/etcd-0 has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-10-21T19:21:35Z
  Ended:       2026-10-21T19:29:35Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1024128-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tjbvo5nqsrtu7s8gb82nob2v0i374vg-z7e6i1u0k4w3u9y", pod="etcd-0", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tjbvo5nqsrtu7s8gb82nob2v0i374vg-z7e6i1u0k4w3u9y/etcd-0 has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-10-21T19:21:35Z
  Ended:       2026-10-21T19:29:35Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1024128-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tjc0q7gfdcimk889lv3jap60dbckk4v-n9e5q5l8r9b9g6l", pod="etcd-0", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tjc0q7gfdcimk889lv3jap60dbckk4v-n9e5q5l8r9b9g6l/etcd-0 has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-10-21T19:21:35Z
  Ended:       2026-10-21T19:27:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1024128-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tjc0smbcqrb2gmp6er6bfn3u5iaj34t-agg-cluster", pod="etcd-1", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tjc0smbcqrb2gmp6er6bfn3u5iaj34t-agg-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-10-21T19:21:35Z
  Ended:       2026-10-21T19:29:35Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1024128-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tjbvv6evjhccd20n1g3tbu8ipmll113-h5a2r1t7x9p4q1s", pod="etcd-1", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tjbvv6evjhccd20n1g3tbu8ipmll113-h5a2r1t7x9p4q1s/etcd-1 has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-10-21T19:22:33Z
  Ended:       2026-10-21T19:29:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1024128-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tjbvo6loe3k9ebp5ulerq9htpom697h-v20261001", pod="etcd-0", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tjbvo6loe3k9ebp5ulerq9htpom697h-v20261001/etcd-0 has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-10-21T19:22:33Z
  Ended:       2026-10-21T19:29:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1024128-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tjc00j4931i0aiu6lhdns95hm39tnvb-l7y7k5n2f0z7s4f", pod="etcd-1", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tjc00j4931i0aiu6lhdns95hm39tnvb-l7y7k5n2f0z7s4f/etcd-1 has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-10-21T19:22:33Z
  Ended:       2026-10-21T19:29:35Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1024128-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tjc0rts3ebhkjfde65hvjnl880lkqc0-c0h6l9s0y7a2e1w", pod="etcd-1", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tjc0rts3ebhkjfde65hvjnl880lkqc0-c0h6l9s0y7a2e1w/etcd-1 has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Resolved
  Started:     2026-10-21T19:22:34Z
  Ended:       2026-10-21T19:29:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1024128-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tjc01gjo3kpogvpf3p4k8245voidlnd-t1v4y7h5e1t6w4t", pod="etcd-0", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tjc01gjo3kpogvpf3p4k8245voidlnd-t1v4y7h5e1t6w4t/etcd-0 has been in a non-ready state for longer than 5 minutes.

alert fired 23 time(s) (13 unknown, 10 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-21T17:29:47Z
  Ended:       2026-10-21T17:38:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0147200-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tja5ssdu0crilsa2dh6uqu5d769g4h6-e8b3q7j1e4i8v5t", pod="azure-file-csi-driver-controller-67c8cd5c96-2nmct", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2tja5ssdu0crilsa2dh6uqu5d769g4h6-e8b3q7j1e4i8v5t/azure-file-csi-driver-controller-67c8cd5c96-2nmct has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-21T17:29:47Z
  Ended:       2026-10-21T17:38:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0147200-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tja5ssdu0crilsa2dh6uqu5d769g4h6-e8b3q7j1e4i8v5t", pod="azure-file-csi-driver-controller-67c8cd5c96-vkxg4", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2tja5ssdu0crilsa2dh6uqu5d769g4h6-e8b3q7j1e4i8v5t/azure-file-csi-driver-controller-67c8cd5c96-vkxg4 has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-21T17:29:47Z
  Ended:       2026-10-21T17:38:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0147200-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tja5ssdu0crilsa2dh6uqu5d769g4h6-e8b3q7j1e4i8v5t", pod="azure-file-csi-driver-controller-67c8cd5c96-v7scv", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2tja5ssdu0crilsa2dh6uqu5d769g4h6-e8b3q7j1e4i8v5t/azure-file-csi-driver-controller-67c8cd5c96-v7scv has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-10-21T17:29:47Z
  Ended:       2026-10-21T17:38:49Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0147200-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tja5ssdu0crilsa2dh6uqu5d769g4h6-e8b3q7j1e4i8v5t", pod="azure-file-csi-driver-controller-67c8cd5c96-dv9w2", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2tja5ssdu0crilsa2dh6uqu5d769g4h6-e8b3q7j1e4i8v5t/azure-file-csi-driver-controller-67c8cd5c96-dv9w2 has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-10-21T17:29:47Z
  Ended:       2026-10-21T17:38:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0147200-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tja5ssdu0crilsa2dh6uqu5d769g4h6-e8b3q7j1e4i8v5t", pod="azure-file-csi-driver-controller-67c8cd5c96-rn59z", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2tja5ssdu0crilsa2dh6uqu5d769g4h6-e8b3q7j1e4i8v5t/azure-file-csi-driver-controller-67c8cd5c96-rn59z has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-10-21T17:29:47Z
  Ended:       2026-10-21T17:38:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0147200-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tja5ssdu0crilsa2dh6uqu5d769g4h6-e8b3q7j1e4i8v5t", pod="azure-file-csi-driver-controller-67c8cd5c96-r5rmw", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2tja5ssdu0crilsa2dh6uqu5d769g4h6-e8b3q7j1e4i8v5t/azure-file-csi-driver-controller-67c8cd5c96-r5rmw has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-10-21T17:29:47Z
  Ended:       2026-10-21T17:38:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0147200-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tja5ssdu0crilsa2dh6uqu5d769g4h6-e8b3q7j1e4i8v5t", pod="azure-file-csi-driver-controller-67c8cd5c96-k4vpn", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2tja5ssdu0crilsa2dh6uqu5d769g4h6-e8b3q7j1e4i8v5t/azure-file-csi-driver-controller-67c8cd5c96-k4vpn has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-10-21T17:29:47Z
  Ended:       2026-10-21T17:38:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0147200-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tja5ssdu0crilsa2dh6uqu5d769g4h6-e8b3q7j1e4i8v5t", pod="azure-file-csi-driver-controller-67c8cd5c96-v2422", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2tja5ssdu0crilsa2dh6uqu5d769g4h6-e8b3q7j1e4i8v5t/azure-file-csi-driver-controller-67c8cd5c96-v2422 has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-10-21T17:29:47Z
  Ended:       2026-10-21T17:38:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0147200-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tja5ssdu0crilsa2dh6uqu5d769g4h6-e8b3q7j1e4i8v5t", pod="azure-file-csi-driver-controller-67c8cd5c96-xm94d", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2tja5ssdu0crilsa2dh6uqu5d769g4h6-e8b3q7j1e4i8v5t/azure-file-csi-driver-controller-67c8cd5c96-xm94d has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-10-21T17:29:47Z
  Ended:       2026-10-21T17:38:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0147200-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tja5ssdu0crilsa2dh6uqu5d769g4h6-e8b3q7j1e4i8v5t", pod="azure-file-csi-driver-controller-67c8cd5c96-xbkbv", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2tja5ssdu0crilsa2dh6uqu5d769g4h6-e8b3q7j1e4i8v5t/azure-file-csi-driver-controller-67c8cd5c96-xbkbv has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-10-21T17:29:47Z
  Ended:       2026-10-21T17:38:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0147200-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tja5ssdu0crilsa2dh6uqu5d769g4h6-e8b3q7j1e4i8v5t", pod="azure-file-csi-driver-controller-67c8cd5c96-bzxx9", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2tja5ssdu0crilsa2dh6uqu5d769g4h6-e8b3q7j1e4i8v5t/azure-file-csi-driver-controller-67c8cd5c96-bzxx9 has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-10-21T17:29:47Z
  Ended:       2026-10-21T17:38:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0147200-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tja5ssdu0crilsa2dh6uqu5d769g4h6-e8b3q7j1e4i8v5t", pod="azure-file-csi-driver-controller-67c8cd5c96-2szds", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2tja5ssdu0crilsa2dh6uqu5d769g4h6-e8b3q7j1e4i8v5t/azure-file-csi-driver-controller-67c8cd5c96-2szds has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-10-21T17:29:47Z
  Ended:       2026-10-21T17:38:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0147200-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tja5ssdu0crilsa2dh6uqu5d769g4h6-e8b3q7j1e4i8v5t", pod="azure-file-csi-driver-controller-67c8cd5c96-mvtgc", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2tja5ssdu0crilsa2dh6uqu5d769g4h6-e8b3q7j1e4i8v5t/azure-file-csi-driver-controller-67c8cd5c96-mvtgc has been in a non-ready state for longer than 5 minutes.

alert fired 14 time(s) (7 unknown, 7 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-21T14:32:51Z
  Ended:       2026-10-21T14:43:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9083904-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tj7kqm1us0ib8nfkht65c5gkioeorko-k5a0f3u2b1h3i6d", pod="azure-disk-csi-driver-controller-67d8f6fb68-blg8t", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tj7kqm1us0ib8nfkht65c5gkioeorko-k5a0f3u2b1h3i6d/azure-disk-csi-driver-controller-67d8f6fb68-blg8t has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-21T14:32:51Z
  Ended:       2026-10-21T14:43:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9083904-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tj7kqm1us0ib8nfkht65c5gkioeorko-k5a0f3u2b1h3i6d", pod="azure-file-csi-driver-controller-7ccfdf6644-64j9p", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tj7kqm1us0ib8nfkht65c5gkioeorko-k5a0f3u2b1h3i6d/azure-file-csi-driver-controller-7ccfdf6644-64j9p has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-21T15:30:44Z
  Ended:       2026-10-21T15:40:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9083904-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tj8166hsuqubnhvhjdq3a2dlhjeqtpc-m0t1y4l2g3r0w4c", pod="cluster-storage-operator-6587998f7c-8j9g8", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tj8166hsuqubnhvhjdq3a2dlhjeqtpc-m0t1y4l2g3r0w4c/cluster-storage-operator-6587998f7c-8j9g8 has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-10-21T15:30:44Z
  Ended:       2026-10-21T15:40:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9083904-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tj8166hsuqubnhvhjdq3a2dlhjeqtpc-m0t1y4l2g3r0w4c", pod="cluster-network-operator-7ddcb6875d-x8rjm", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tj8166hsuqubnhvhjdq3a2dlhjeqtpc-m0t1y4l2g3r0w4c/cluster-network-operator-7ddcb6875d-x8rjm has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-10-21T15:30:44Z
  Ended:       2026-10-21T15:40:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9083904-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tj8166hsuqubnhvhjdq3a2dlhjeqtpc-m0t1y4l2g3r0w4c", pod="csi-snapshot-controller-operator-86969ccc8c-pcv4n", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tj8166hsuqubnhvhjdq3a2dlhjeqtpc-m0t1y4l2g3r0w4c/csi-snapshot-controller-operator-86969ccc8c-pcv4n has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-10-21T15:30:44Z
  Ended:       2026-10-21T15:40:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9083904-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tj8166hsuqubnhvhjdq3a2dlhjeqtpc-m0t1y4l2g3r0w4c", pod="dns-operator-7d6d66cd69-mmzzh", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tj8166hsuqubnhvhjdq3a2dlhjeqtpc-m0t1y4l2g3r0w4c/dns-operator-7d6d66cd69-mmzzh has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-10-21T15:30:44Z
  Ended:       2026-10-21T15:40:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9083904-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tj8166hsuqubnhvhjdq3a2dlhjeqtpc-m0t1y4l2g3r0w4c", pod="ingress-operator-6c94f6f674-6p4n9", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tj8166hsuqubnhvhjdq3a2dlhjeqtpc-m0t1y4l2g3r0w4c/ingress-operator-6c94f6f674-6p4n9 has been in a non-ready state for longer than 5 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubePodNotReady does not fire	11

Affected runs (11)

Date (UTC)	Associated PR	Prow job
2026-10-21 18:21	PR #7641	2112972661071024128
2026-10-21 16:18	PR #7544	2112941576740147200
2026-10-21 12:58	PR #7635	2112891349849083904
2026-10-21 12:08	PR #7655	2112878574095568896
2026-10-21 11:34	PR #7658	2112870236943290368
2026-10-21 10:03	PR #7598	2112847311775207424
2026-10-21 08:37	PR #7463	2112825723625410560
2026-10-21 05:55	PR #7073	2112784863382736896
2026-10-21 04:34	PR #7133	2112764404264079360
2026-10-21 04:03	PR #7629	2112756690729308160
2026-10-21 01:11	PR #7633	2112713279871651840

alert [svc] BackendControllerQueueDepthHigh fired

full failure pattern:

alert [svc] BackendControllerQueueDepthHigh fired

Signal:

alert

none

Full failure examples (3)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-21T21:50:37Z
  Ended:       2026-10-21T22:00:31Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j9052672-svc", component="backend", name="csstatedump", region="westus3", severity="warning"
  Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-21T21:32:19Z
  Ended:       2026-10-21T21:51:19Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j0788736-svc", component="backend", name="csstatedump", region="centralus", severity="warning"
  Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 2:
  State:       Resolved
  Started:     2026-10-21T21:32:19Z
  Ended:       2026-10-21T21:51:19Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j0788736-svc", component="backend", name="clusterresources", region="centralus", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-21T21:09:58Z
  Ended:       2026-10-21T21:23:00Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4444160-svc", component="backend", name="csstatedump", region="canadacentral", severity="warning"
  Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire	8

Affected runs (8)

Date (UTC)	Associated PR	Prow job
2026-10-21 20:49	PR #7371	2113009860609052672
2026-10-21 20:25	PR #7669	2113003778620788736
2026-10-21 19:55	PR #7592	2112996419844444160
2026-10-21 11:34	PR #7658	2112870236943290368
2026-10-21 10:13	PR #7657	2112849728864522240
2026-10-21 10:03	PR #7598	2112847311775207424
2026-10-21 05:55	PR #7073	2112784863382736896
2026-10-21 01:11	PR #7629	2112713315011530752

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0x3c48d8a2f6e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-b9m4k92gvj4g" to finish creating: GET https://rp.j5137024.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/34253dc7-331a-43c6-b614-64cab812db2d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/34253dc7-331a-43c6-b614-64cab812db2d",
      "name": "34253dc7-331a-43c6-b614-64cab812db2d",
      "status": "Failed",
      "startTime": "2026-10-21T17:06:32.563128324Z",
      "endTime": "2026-10-21T17:25:32.638298339Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5137024.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/34253dc7-331a-43c6-b614-64cab812db2d
            correlation ID: 730b525c-8264-4232-94a3-62d2afb7c9db
            client request ID: f267fbb0-aa2f-4e04-8d7a-e731b332311e
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0x3c48d8a2f6e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-b9m4k92gvj4g" to finish creating: GET https://rp.j5137024.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/34253dc7-331a-43c6-b614-64cab812db2d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/34253dc7-331a-43c6-b614-64cab812db2d",
      "name": "34253dc7-331a-43c6-b614-64cab812db2d",
      "status": "Failed",
      "startTime": "2026-10-21T17:06:32.563128324Z",
      "endTime": "2026-10-21T17:25:32.638298339Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5137024.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/34253dc7-331a-43c6-b614-64cab812db2d
            correlation ID: 730b525c-8264-4232-94a3-62d2afb7c9db
            client request ID: f267fbb0-aa2f-4e04-8d7a-e731b332311e
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x164574f57840>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-fcpwj2cw9k8v" to finish creating: GET https://rp.j1735936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/649ff5b3-4b10-49f2-be45-b62cf479b6c7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/649ff5b3-4b10-49f2-be45-b62cf479b6c7",
      "name": "649ff5b3-4b10-49f2-be45-b62cf479b6c7",
      "status": "Failed",
      "startTime": "2026-10-21T17:16:17.057110724Z",
      "endTime": "2026-10-21T17:35:26.070571768Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1735936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/649ff5b3-4b10-49f2-be45-b62cf479b6c7
            correlation ID: 7a498bd5-b28c-425f-9407-7345c3d78071
            client request ID: b5fdf720-d04b-4b3d-91c7-235c845f2533
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x164574f57840>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-fcpwj2cw9k8v" to finish creating: GET https://rp.j1735936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/649ff5b3-4b10-49f2-be45-b62cf479b6c7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/649ff5b3-4b10-49f2-be45-b62cf479b6c7",
      "name": "649ff5b3-4b10-49f2-be45-b62cf479b6c7",
      "status": "Failed",
      "startTime": "2026-10-21T17:16:17.057110724Z",
      "endTime": "2026-10-21T17:35:26.070571768Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1735936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/649ff5b3-4b10-49f2-be45-b62cf479b6c7
            correlation ID: 7a498bd5-b28c-425f-9407-7345c3d78071
            client request ID: b5fdf720-d04b-4b3d-91c7-235c845f2533
occurred

Contributing tests (2)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-21 15:53	PR #7662	2112935310475137024
2026-10-21 15:48	PR #7065	2112933937901735936

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...

full failure pattern:

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
    <*errors.joinError | 0x2dc0b06220f0>: 
    resource group "des-encrypt-xb5tc64jjbk5": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="des-encrypt" in resourcegroup="des-encrypt-xb5tc64jjbk5" to finish deleting: GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7d6b2c93-7730-4824-8846-3824a20bfdec
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7d6b2c93-7730-4824-8846-3824a20bfdec",
      "name": "7d6b2c93-7730-4824-8846-3824a20bfdec",
      "status": "Failed",
      "startTime": "2026-10-21T14:45:35.149437794Z",
      "endTime": "2026-10-21T15:09:42.233654784Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tj7ubn23jup03cfdsjlt8o475dmidid still exists (deletion dispatched at 2026-10-21T14:45:35Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7d6b2c93-7730-4824-8846-3824a20bfdec
            correlation ID: 60235ac6-f0dd-4c09-b328-c07af59f6052
            client request ID: 76e45f1e-02ec-4d1b-9036-bebaa6509dd0
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
    <*errors.joinError | 0x2dc0b06220f0>: 
    resource group "des-encrypt-xb5tc64jjbk5": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="des-encrypt" in resourcegroup="des-encrypt-xb5tc64jjbk5" to finish deleting: GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7d6b2c93-7730-4824-8846-3824a20bfdec
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7d6b2c93-7730-4824-8846-3824a20bfdec",
      "name": "7d6b2c93-7730-4824-8846-3824a20bfdec",
      "status": "Failed",
      "startTime": "2026-10-21T14:45:35.149437794Z",
      "endTime": "2026-10-21T15:09:42.233654784Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tj7ubn23jup03cfdsjlt8o475dmidid still exists (deletion dispatched at 2026-10-21T14:45:35Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7d6b2c93-7730-4824-8846-3824a20bfdec
            correlation ID: 60235ac6-f0dd-4c09-b328-c07af59f6052
            client request ID: 76e45f1e-02ec-4d1b-9036-bebaa6509dd0
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
    <*errors.joinError | 0x3784d74e8120>: 
    resource group "rg-zstream-upgrade-4-21-vxd5jn5bhkxj": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-5dpxf2" in resourcegroup="rg-zstream-upgrade-4-21-vxd5jn5bhkxj" to finish deleting: GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bc3c7e87-6636-4e88-bae2-c31e0e4df79d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bc3c7e87-6636-4e88-bae2-c31e0e4df79d",
      "name": "bc3c7e87-6636-4e88-bae2-c31e0e4df79d",
      "status": "Failed",
      "startTime": "2026-10-21T14:39:39.382659348Z",
      "endTime": "2026-10-21T15:03:42.226397448Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tj7kh8478418ujofeaedu0kqd3iab5p still exists (deletion dispatched at 2026-10-21T14:39:40Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bc3c7e87-6636-4e88-bae2-c31e0e4df79d
            correlation ID: 91639a6f-eb4e-41bf-9deb-689b316310a7
            client request ID: 7eff7a5b-5235-4d57-89e8-fc1d52195b20
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
    <*errors.joinError | 0x3784d74e8120>: 
    resource group "rg-zstream-upgrade-4-21-vxd5jn5bhkxj": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-5dpxf2" in resourcegroup="rg-zstream-upgrade-4-21-vxd5jn5bhkxj" to finish deleting: GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bc3c7e87-6636-4e88-bae2-c31e0e4df79d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bc3c7e87-6636-4e88-bae2-c31e0e4df79d",
      "name": "bc3c7e87-6636-4e88-bae2-c31e0e4df79d",
      "status": "Failed",
      "startTime": "2026-10-21T14:39:39.382659348Z",
      "endTime": "2026-10-21T15:03:42.226397448Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tj7kh8478418ujofeaedu0kqd3iab5p still exists (deletion dispatched at 2026-10-21T14:39:40Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bc3c7e87-6636-4e88-bae2-c31e0e4df79d
            correlation ID: 91639a6f-eb4e-41bf-9deb-689b316310a7
            client request ID: 7eff7a5b-5235-4d57-89e8-fc1d52195b20
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
    <*errors.joinError | 0x3d91346be150>: 
    resource group "rg-np-version-upgrade-wxkr7l-gs2n5hfrc7cj": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-wxkr7l" in resourcegroup="rg-np-version-upgrade-wxkr7l-gs2n5hfrc7cj" to finish deleting: GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f609346f-a626-48c2-b3f7-2432c0d4f1c1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f609346f-a626-48c2-b3f7-2432c0d4f1c1",
      "name": "f609346f-a626-48c2-b3f7-2432c0d4f1c1",
      "status": "Failed",
      "startTime": "2026-10-21T15:16:30.030111335Z",
      "endTime": "2026-10-21T15:40:31.183335705Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tj8166hsuqubnhvhjdq3a2dlhjeqtpc still exists (deletion dispatched at 2026-10-21T15:16:30Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f609346f-a626-48c2-b3f7-2432c0d4f1c1
            correlation ID: 7043ac86-0234-4e2b-aec8-d9e0460c5a5c
            client request ID: 8cc99b04-9a95-4db0-933a-c9718f38aca5
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
    <*errors.joinError | 0x3d91346be150>: 
    resource group "rg-np-version-upgrade-wxkr7l-gs2n5hfrc7cj": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-wxkr7l" in resourcegroup="rg-np-version-upgrade-wxkr7l-gs2n5hfrc7cj" to finish deleting: GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f609346f-a626-48c2-b3f7-2432c0d4f1c1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f609346f-a626-48c2-b3f7-2432c0d4f1c1",
      "name": "f609346f-a626-48c2-b3f7-2432c0d4f1c1",
      "status": "Failed",
      "startTime": "2026-10-21T15:16:30.030111335Z",
      "endTime": "2026-10-21T15:40:31.183335705Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tj8166hsuqubnhvhjdq3a2dlhjeqtpc still exists (deletion dispatched at 2026-10-21T15:16:30Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f609346f-a626-48c2-b3f7-2432c0d4f1c1
            correlation ID: 7043ac86-0234-4e2b-aec8-d9e0460c5a5c
            client request ID: 8cc99b04-9a95-4db0-933a-c9718f38aca5
occurred

Contributing tests (3)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Nodepool OS Disk Encryption should create a nodepool with customer-managed disk encryption via DES	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-21 12:58	PR #7635	2112891349849083904

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...

full failure pattern:

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [resourceTeardown] tearing down namespaces; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
    <*errors.joinError | 0x279359bea228>: 
    resource group "admin-api-version-pin-f5l942lsxnxx": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-version-pin-7g5shb" in resourcegroup="admin-api-version-pin-f5l942lsxnxx" to finish deleting: GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c9617226-f7ce-4606-a2f9-eb711857548d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c9617226-f7ce-4606-a2f9-eb711857548d",
      "name": "c9617226-f7ce-4606-a2f9-eb711857548d",
      "status": "Failed",
      "startTime": "2026-10-21T14:39:39.950651125Z",
      "endTime": "2026-10-21T15:03:42.224683778Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tj7lluppmfrbdhhdr0g5hdh5ec68bse still exists (deletion dispatched at 2026-10-21T14:39:40Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c9617226-f7ce-4606-a2f9-eb711857548d
            correlation ID: 48d5a3fe-e637-4aff-9f54-63cebfd278ec
            client request ID: d94bc996-f0e4-48a3-ab58-bf75318409cc
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
    <*errors.joinError | 0x279359bea228>: 
    resource group "admin-api-version-pin-f5l942lsxnxx": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-version-pin-7g5shb" in resourcegroup="admin-api-version-pin-f5l942lsxnxx" to finish deleting: GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c9617226-f7ce-4606-a2f9-eb711857548d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c9617226-f7ce-4606-a2f9-eb711857548d",
      "name": "c9617226-f7ce-4606-a2f9-eb711857548d",
      "status": "Failed",
      "startTime": "2026-10-21T14:39:39.950651125Z",
      "endTime": "2026-10-21T15:03:42.224683778Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tj7lluppmfrbdhhdr0g5hdh5ec68bse still exists (deletion dispatched at 2026-10-21T14:39:40Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c9617226-f7ce-4606-a2f9-eb711857548d
            correlation ID: 48d5a3fe-e637-4aff-9f54-63cebfd278ec
            client request ID: d94bc996-f0e4-48a3-ab58-bf75318409cc
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
    <*errors.joinError | 0x2cef6ec040a8>: 
    resource group "admin-api-breakglass-9q9nnh4t7882": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-9q9nnh4t7882" to finish deleting: GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8eaf5e6c-659a-4bc4-8c15-d54c813f8e68
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8eaf5e6c-659a-4bc4-8c15-d54c813f8e68",
      "name": "8eaf5e6c-659a-4bc4-8c15-d54c813f8e68",
      "status": "Failed",
      "startTime": "2026-10-21T14:29:52.564744569Z",
      "endTime": "2026-10-21T14:53:53.054355548Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tj7od9dtldg9atesqujpab56gdtktc3 still exists (deletion dispatched at 2026-10-21T14:29:53Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8eaf5e6c-659a-4bc4-8c15-d54c813f8e68
            correlation ID: ad5bbe53-529d-4033-95da-f2bd383b8927
            client request ID: 1402c96d-416f-418a-a03d-6b48908f76d1
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
    <*errors.joinError | 0x2cef6ec040a8>: 
    resource group "admin-api-breakglass-9q9nnh4t7882": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-9q9nnh4t7882" to finish deleting: GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8eaf5e6c-659a-4bc4-8c15-d54c813f8e68
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8eaf5e6c-659a-4bc4-8c15-d54c813f8e68",
      "name": "8eaf5e6c-659a-4bc4-8c15-d54c813f8e68",
      "status": "Failed",
      "startTime": "2026-10-21T14:29:52.564744569Z",
      "endTime": "2026-10-21T14:53:53.054355548Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tj7od9dtldg9atesqujpab56gdtktc3 still exists (deletion dispatched at 2026-10-21T14:29:53Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8eaf5e6c-659a-4bc4-8c15-d54c813f8e68
            correlation ID: ad5bbe53-529d-4033-95da-f2bd383b8927
            client request ID: 1402c96d-416f-418a-a03d-6b48908f76d1
occurred

Contributing tests (2)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	SRE should be able to log into a cluster via a breakglass session	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	SRE starts a control plane z-stream upgrade with an SRE pin	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-21 12:58	PR #7635	2112891349849083904

alert [hcp] KubeDeploymentReplicasMismatch fired

full failure pattern:

alert [hcp] KubeDeploymentReplicasMismatch fired

Signal:

alert

none

Full failure examples (1)

alert fired 12 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-21T15:11:47Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9083904-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.176:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9083904/providers/microsoft.monitor/accounts/hcps-j9083904", namespace="ocm-arohcpci01-2tj7od9dtldg9atesqujpab56gdtktc3-sre-hcp-cluster", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-sxf4w", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tj7od9dtldg9atesqujpab56gdtktc3-sre-hcp-cluster/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-21T15:11:47Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9083904-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.176:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9083904/providers/microsoft.monitor/accounts/hcps-j9083904", namespace="ocm-arohcpci01-2tj7od9dtldg9atesqujpab56gdtktc3-sre-hcp-cluster", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-sxf4w", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tj7od9dtldg9atesqujpab56gdtktc3-sre-hcp-cluster/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-21T15:11:47Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9083904-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.176:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9083904/providers/microsoft.monitor/accounts/hcps-j9083904", namespace="ocm-arohcpci01-2tj7od9dtldg9atesqujpab56gdtktc3-sre-hcp-cluster", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-sxf4w", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tj7od9dtldg9atesqujpab56gdtktc3-sre-hcp-cluster/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-21T15:11:48Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9083904-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.176:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9083904/providers/microsoft.monitor/accounts/hcps-j9083904", namespace="ocm-arohcpci01-2tj7od9dtldg9atesqujpab56gdtktc3-sre-hcp-cluster", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-sxf4w", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tj7od9dtldg9atesqujpab56gdtktc3-sre-hcp-cluster/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-10-21T15:20:46Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9083904-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.176:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9083904/providers/microsoft.monitor/accounts/hcps-j9083904", namespace="ocm-arohcpci01-2tj7lluppmfrbdhhdr0g5hdh5ec68bse-m8x0b3m3p3c6a3w", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-sxf4w", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tj7lluppmfrbdhhdr0g5hdh5ec68bse-m8x0b3m3p3c6a3w/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-10-21T15:20:46Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9083904-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.176:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9083904/providers/microsoft.monitor/accounts/hcps-j9083904", namespace="ocm-arohcpci01-2tj7lluppmfrbdhhdr0g5hdh5ec68bse-m8x0b3m3p3c6a3w", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-sxf4w", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tj7lluppmfrbdhhdr0g5hdh5ec68bse-m8x0b3m3p3c6a3w/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 7:
  State:       Fired (not resolved)
  Started:     2026-10-21T15:22:46Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9083904-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.206:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9083904/providers/microsoft.monitor/accounts/hcps-j9083904", namespace="ocm-arohcpci01-2tj7kg5i5v3qeqbhhu1aj4t26p20km4l-s6i4e5m7h6r7f2g", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-v576p", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tj7kg5i5v3qeqbhhu1aj4t26p20km4l-s6i4e5m7h6r7f2g/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 8:
  State:       Fired (not resolved)
  Started:     2026-10-21T15:22:46Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9083904-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.206:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9083904/providers/microsoft.monitor/accounts/hcps-j9083904", namespace="ocm-arohcpci01-2tj7kg5i5v3qeqbhhu1aj4t26p20km4l-s6i4e5m7h6r7f2g", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-v576p", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tj7kg5i5v3qeqbhhu1aj4t26p20km4l-s6i4e5m7h6r7f2g/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 9:
  State:       Fired (not resolved)
  Started:     2026-10-21T15:22:46Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9083904-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.206:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9083904/providers/microsoft.monitor/accounts/hcps-j9083904", namespace="ocm-arohcpci01-2tj7kg5i5v3qeqbhhu1aj4t26p20km4l-s6i4e5m7h6r7f2g", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-v576p", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tj7kg5i5v3qeqbhhu1aj4t26p20km4l-s6i4e5m7h6r7f2g/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 10:
  State:       Fired (not resolved)
  Started:     2026-10-21T15:22:46Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9083904-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.206:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9083904/providers/microsoft.monitor/accounts/hcps-j9083904", namespace="ocm-arohcpci01-2tj7kg5i5v3qeqbhhu1aj4t26p20km4l-s6i4e5m7h6r7f2g", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-v576p", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tj7kg5i5v3qeqbhhu1aj4t26p20km4l-s6i4e5m7h6r7f2g/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 11:
  State:       Fired (not resolved)
  Started:     2026-10-21T15:23:45Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9083904-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.176:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9083904/providers/microsoft.monitor/accounts/hcps-j9083904", namespace="ocm-arohcpci01-2tj7lluppmfrbdhhdr0g5hdh5ec68bse-m8x0b3m3p3c6a3w", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-sxf4w", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tj7lluppmfrbdhhdr0g5hdh5ec68bse-m8x0b3m3p3c6a3w/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 12:
  State:       Fired (not resolved)
  Started:     2026-10-21T15:23:47Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9083904-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.176:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9083904/providers/microsoft.monitor/accounts/hcps-j9083904", namespace="ocm-arohcpci01-2tj7lluppmfrbdhhdr0g5hdh5ec68bse-m8x0b3m3p3c6a3w", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-sxf4w", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tj7lluppmfrbdhhdr0g5hdh5ec68bse-m8x0b3m3p3c6a3w/cluster-api has not matched the expected number of replicas for longer than 30 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-21 12:58	PR #7635	2112891349849083904

DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): co...

full failure pattern:

DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:201]: failed to verify simple web app on back-level cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0x36cd28d2c720>: 
    DNS for route host agnhost-e2e-sample-app-272d9.apps.aro.g1g2s2z9p9f8q5l.qgj6.j6469632.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-272d9.apps.aro.g1g2s2z9p9f8q5l.qgj6.j6469632.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-272d9.apps.aro.g1g2s2z9p9f8q5l.qgj6.j6469632.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-272d9.apps.aro.g1g2s2z9p9f8q5l.qgj6.j6469632.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-272d9.apps.aro.g1g2s2z9p9f8q5l.qgj6.j6469632.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-272d9.apps.aro.g1g2s2z9p9f8q5l.qgj6.j6469632.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x36cd28c224c0>{
            msg: "DNS for agnhost-e2e-sample-app-272d9.apps.aro.g1g2s2z9p9f8q5l.qgj6.j6469632.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-272d9.apps.aro.g1g2s2z9p9f8q5l.qgj6.j6469632.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:201]: failed to verify simple web app on back-level cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0x36cd28d2c720>: 
    DNS for route host agnhost-e2e-sample-app-272d9.apps.aro.g1g2s2z9p9f8q5l.qgj6.j6469632.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-272d9.apps.aro.g1g2s2z9p9f8q5l.qgj6.j6469632.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-272d9.apps.aro.g1g2s2z9p9f8q5l.qgj6.j6469632.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-272d9.apps.aro.g1g2s2z9p9f8q5l.qgj6.j6469632.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-272d9.apps.aro.g1g2s2z9p9f8q5l.qgj6.j6469632.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-272d9.apps.aro.g1g2s2z9p9f8q5l.qgj6.j6469632.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x36cd28c224c0>{
            msg: "DNS for agnhost-e2e-sample-app-272d9.apps.aro.g1g2s2z9p9f8q5l.qgj6.j6469632.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-272d9.apps.aro.g1g2s2z9p9f8q5l.qgj6.j6469632.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with back-level version 4.20	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-21 16:54	PR #7582	2112950731366469632

alert [svc] MaestroAgentMetricsProxyDown fired

full failure pattern:

alert [svc] MaestroAgentMetricsProxyDown fired

Signal:

alert

none

Full failure examples (1)

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-21T18:12:50Z
  Severity:    Sev3
  Labels:      alertname="MaestroAgentMetricsProxyDown", cluster="ci01-j5592832-mgmt-1", component="monitoring-infrastructure", region="canadacentral", severity="3"
  Description: Prometheus has not scraped maestro-agent /metrics through the metrics-proxy sidecar (namespace=maestro, endpoint=metrics) for 10 minutes, while a kube-state-metrics scrape series still exists for this management cluster. Maestro agent metrics and any alerts that depend on them may be missing.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-21T18:12:50Z
  Severity:    Sev3
  Labels:      alertname="MaestroAgentMetricsProxyDown", cluster="ci01-j5592832-mgmt-2", component="monitoring-infrastructure", region="canadacentral", severity="3"
  Description: Prometheus has not scraped maestro-agent /metrics through the metrics-proxy sidecar (namespace=maestro, endpoint=metrics) for 10 minutes, while a kube-state-metrics scrape series still exists for this management cluster. Maestro agent metrics and any alerts that depend on them may be missing.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert MaestroAgentMetricsProxyDown does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-21 16:58	PR #7665	2112951578825592832

ERROR CODE: 429; provider Microsoft.ManagedIdentity

full failure pattern:

ERROR CODE: 429; provider Microsoft.ManagedIdentity

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:111]: failed to get cluster-api-azure managed identity
Unexpected error:
    <*exported.ResponseError | 0xc16479d48a0>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard1-02-07/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure
    --------------------------------------------------------------------------------
    RESPONSE 429: 429 Too Many Requests
    ERROR CODE: 429
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "429",
        "message": "Request is temporarily throttled because tenant XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 5 seconds."
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard1-02-07/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure
            correlation ID: 9f0dbad1-e718-4aa2-b236-36e96b6de447
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:111]: failed to get cluster-api-azure managed identity
Unexpected error:
    <*exported.ResponseError | 0xc16479d48a0>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard1-02-07/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure
    --------------------------------------------------------------------------------
    RESPONSE 429: 429 Too Many Requests
    ERROR CODE: 429
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "429",
        "message": "Request is temporarily throttled because tenant XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 5 seconds."
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard1-02-07/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure
            correlation ID: 9f0dbad1-e718-4aa2-b236-36e96b6de447
            client request ID: unavailable
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Nodepool OS Disk Encryption should create a nodepool with customer-managed disk encryption via DES	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-21 18:37	PR #7650	2112976501128302592

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.43
Unexpected error:
    <*fmt.wrapError | 0x1e8393e0bee0>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-gbdj5r" in resourcegroup="rg-np-version-upgrade-gbdj5r-ql9t2zm9w49h" to finish creating: GET https://rp.j5207424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6d0eb9e1-4d4f-4249-8838-c2cc78760cf4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6d0eb9e1-4d4f-4249-8838-c2cc78760cf4",
      "name": "6d0eb9e1-4d4f-4249-8838-c2cc78760cf4",
      "status": "Failed",
      "startTime": "2026-10-21T11:27:06.832183502Z",
      "endTime": "2026-10-21T11:46:10.834576567Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine z0c9h6g7p4h7x7v-npupgrade-4-20-9ds9t-fxlcn: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine z0c9h6g7p4h7x7v-npupgrade-4-20-9ds9t-fxlcn: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5207424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6d0eb9e1-4d4f-4249-8838-c2cc78760cf4
            correlation ID: ede1199e-dd9c-4499-8177-d42c4436eb90
            client request ID: d0c900cc-972d-4998-89ab-51274a7d36d4
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.43
Unexpected error:
    <*fmt.wrapError | 0x1e8393e0bee0>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-gbdj5r" in resourcegroup="rg-np-version-upgrade-gbdj5r-ql9t2zm9w49h" to finish creating: GET https://rp.j5207424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6d0eb9e1-4d4f-4249-8838-c2cc78760cf4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6d0eb9e1-4d4f-4249-8838-c2cc78760cf4",
      "name": "6d0eb9e1-4d4f-4249-8838-c2cc78760cf4",
      "status": "Failed",
      "startTime": "2026-10-21T11:27:06.832183502Z",
      "endTime": "2026-10-21T11:46:10.834576567Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine z0c9h6g7p4h7x7v-npupgrade-4-20-9ds9t-fxlcn: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine z0c9h6g7p4h7x7v-npupgrade-4-20-9ds9t-fxlcn: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5207424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6d0eb9e1-4d4f-4249-8838-c2cc78760cf4
            correlation ID: ede1199e-dd9c-4499-8177-d42c4436eb90
            client request ID: d0c900cc-972d-4998-89ab-51274a7d36d4
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-21 10:03	PR #7598	2112847311775207424

| take 101, Post "<url>": context deadline exceeded

full failure pattern:

| take 101, Post "<url>": context deadline exceeded

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-npskt6qgktdr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x3fb9cc54d40>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-npskt6qgktdr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = "dev"; let rgn = "centralus"; let cl = "ci01-j4218880-mgmt-1"; let ns = "ocm-arohcpci01-2tjdecblfn866doucbvsck82v6djrsrv-p7h2c2c6w4m5q7f"; let start = datetime(2026-10-21T20:47:23.6514068Z);
    let scope = strcat(env, "/", rgn, "/", cl, "/", ns);
    let inventory = materialize(kubernetesResourceSnapshots
    | where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns
    | where objectKind in ("Pod", "Deployment")
    | summarize arg_max(timestamp, *) by objectKind, name
    | where event != "Delete" and isempty(object.metadata.deletionTimestamp));
    let pods = inventory
    | where objectKind == "Pod" and object.metadata.labels.app == "private-router" and object.status.phase == "Running"
    | project UID = uid, Node = tostring(object.spec.nodeName);
    let desired = toscalar(inventory
    | where objectKind == "Deployment" and name == "router"
    | project tolong(object.spec.replicas));
    let current = toscalar(pods | count);
    let summaries = containerLogs
    | where timestamp >= start and environment == env and region == rgn and cluster == cl
    | where container_name == "swift-recorder"
    | where log.controller_name == "swift-router-check" and log.field == "summary"
    | where tostring(log.pod_namespace) == ns
    | extend UID = tostring(log.pod_uid)
    | summarize arg_max(timestamp, *) by UID
    | project UID, SummaryUID = tostring(log.pod_uid),
        SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);
    pods
    | join kind=leftouter summaries on UID
    | extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))
    | project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary
    | order by Scope asc, UID asc
    | take 101, Post "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query": context deadline exceeded
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-npskt6qgktdr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j4218880-mgmt-1\"; let ns = \"ocm-arohcpci01-2tjdecblfn866doucbvsck82v6djrsrv-p7h2c2c6w4m5q7f\"; let start = datetime(2026-10-21T20:47:23.6514068Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
        err: <*fmt.wrapError | 0x3fb9cc54d20>{
            msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j4218880-mgmt-1\"; let ns = \"ocm-arohcpci01-2tjdecblfn866doucbvsck82v6djrsrv-p7h2c2c6w4m5q7f\"; let start = datetime(2026-10-21T20:47:23.6514068Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
            err: <*errors.Error | 0x3fb9be4c6e0>{
                Op: 1,
                Kind: 7,
                Err: <*fmt.wrapError | 0x3fb9cc54d00>{
                    msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j4218880-mgmt-1\"; let ns = \"ocm-arohcpci01-2tjdecblfn866doucbvsck82v6djrsrv-p7h2c2c6w4m5q7f\"; let start = datetime(2026-10-21T20:47:23.6514068Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                    err: <*url.Error | 0x3fb9dd08330>{
                        Op: "Post",
                        URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                        Err: <context.deadlineExceededError>{},
                    },
                },
                restErrMsg: nil,
                decoded: nil,
                permanent: false,
                inner: nil,
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-npskt6qgktdr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x3fb9cc54d40>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-npskt6qgktdr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = "dev"; let rgn = "centralus"; let cl = "ci01-j4218880-mgmt-1"; let ns = "ocm-arohcpci01-2tjdecblfn866doucbvsck82v6djrsrv-p7h2c2c6w4m5q7f"; let start = datetime(2026-10-21T20:47:23.6514068Z);
    let scope = strcat(env, "/", rgn, "/", cl, "/", ns);
    let inventory = materialize(kubernetesResourceSnapshots
    | where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns
    | where objectKind in ("Pod", "Deployment")
    | summarize arg_max(timestamp, *) by objectKind, name
    | where event != "Delete" and isempty(object.metadata.deletionTimestamp));
    let pods = inventory
    | where objectKind == "Pod" and object.metadata.labels.app == "private-router" and object.status.phase == "Running"
    | project UID = uid, Node = tostring(object.spec.nodeName);
    let desired = toscalar(inventory
    | where objectKind == "Deployment" and name == "router"
    | project tolong(object.spec.replicas));
    let current = toscalar(pods | count);
    let summaries = containerLogs
    | where timestamp >= start and environment == env and region == rgn and cluster == cl
    | where container_name == "swift-recorder"
    | where log.controller_name == "swift-router-check" and log.field == "summary"
    | where tostring(log.pod_namespace) == ns
    | extend UID = tostring(log.pod_uid)
    | summarize arg_max(timestamp, *) by UID
    | project UID, SummaryUID = tostring(log.pod_uid),
        SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);
    pods
    | join kind=leftouter summaries on UID
    | extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))
    | project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary
    | order by Scope asc, UID asc
    | take 101, Post "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query": context deadline exceeded
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-npskt6qgktdr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j4218880-mgmt-1\"; let ns = \"ocm-arohcpci01-2tjdecblfn866doucbvsck82v6djrsrv-p7h2c2c6w4m5q7f\"; let start = datetime(2026-10-21T20:47:23.6514068Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
        err: <*fmt.wrapError | 0x3fb9cc54d20>{
            msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j4218880-mgmt-1\"; let ns = \"ocm-arohcpci01-2tjdecblfn866doucbvsck82v6djrsrv-p7h2c2c6w4m5q7f\"; let start = datetime(2026-10-21T20:47:23.6514068Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
            err: <*errors.Error | 0x3fb9be4c6e0>{
                Op: 1,
                Kind: 7,
                Err: <*fmt.wrapError | 0x3fb9cc54d00>{
                    msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j4218880-mgmt-1\"; let ns = \"ocm-arohcpci01-2tjdecblfn866doucbvsck82v6djrsrv-p7h2c2c6w4m5q7f\"; let start = datetime(2026-10-21T20:47:23.6514068Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                    err: <*url.Error | 0x3fb9dd08330>{
                        Op: "Post",
                        URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                        Err: <context.deadlineExceededError>{},
                    },
                },
                restErrMsg: nil,
                decoded: nil,
                permanent: false,
                inner: nil,
            },
        },
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jdsxztvcznpr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x2fae15cf05c0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jdsxztvcznpr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jdsxztvcznpr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster"; let start = datetime(2026-10-21T19:48:49.875795Z);
    let desires = cosmosResourceSnapshots
    | where resourceID startswith strcat(target, "/")
    | where resourceType =~ "microsoft.redhatopenshift/hcpopenshiftclusters/readdesires"
    | extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
    | summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts
    | summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID
    | where isempty(content.deletionTimestamp)
    | extend manifest = content.properties.status.kubeContent
    | where manifest.kind == "HostedCluster" and isempty(manifest.metadata.deletionTimestamp)
    | extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),
        managementCluster = tolower(tostring(content.properties.spec.managementCluster))
    | where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)
    | project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, "-", hostedClusterName);
    let fleet = cosmosResourceSnapshots
    | where resourceType =~ "microsoft.redhatopenshift/stamps/managementclusters"
    | extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
    | join kind=leftsemi desires on environment, region, sourceCluster, managementCluster
    | summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts
    | summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster
    | where isempty(content.deletionTimestamp)
    | extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), "/")[-1])
    | where isnotempty(clusterName)
    | project environment, region, sourceCluster, managementCluster, cluster = clusterName;
    let hcp = desires
    | join kind=inner fleet on environment, region, sourceCluster, managementCluster
    | distinct environment, region, cluster, namespace;
    let inventory = materialize(kubernetesResourceSnapshots
    | where timestamp >= start and objectKind in ("Pod", "Deployment")
    | lookup kind=inner hcp on environment, region, cluster, namespace
    | summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name
    | where event != "Delete" and isempty(object.metadata.deletionTimestamp));
    let pods = inventory
    | where objectKind == "Pod" and object.metadata.labels.app == "private-router" and object.status.phase == "Running"
    | project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);
    let counts = pods | summarize Current = count() by environment, region, cluster, namespace;
    let deployments = inventory
    | where objectKind == "Deployment" and name == "router"
    | project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);
    let summaries = containerLogs
    | where timestamp >= start and container_name == "swift-recorder"
    | where log.controller_name == "swift-router-check" and log.field == "summary"
    | extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)
    | lookup kind=inner hcp on environment, region, cluster, namespace
    | summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID
    | project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),
        SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);
    hcp
    | join kind=leftouter deployments on environment, region, cluster, namespace
    | join kind=leftouter counts on environment, region, cluster, namespace
    | join kind=leftouter pods on environment, region, cluster, namespace
    | join kind=leftouter summaries on environment, region, cluster, namespace, UID
    | project Scope = strcat(environment, "/", region, "/", cluster, "/", namespace),
        Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary
    | order by Scope asc, UID asc
    | take 101, Post "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query": context deadline exceeded
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jdsxztvcznpr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jdsxztvcznpr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-21T19:48:49.875795Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
        err: <*fmt.wrapError | 0x2fae15cf05a0>{
            msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jdsxztvcznpr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-21T19:48:49.875795Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
            err: <*errors.Error | 0x2fae15532370>{
                Op: 1,
                Kind: 7,
                Err: <*fmt.wrapError | 0x2fae15cf0580>{
                    msg: "With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jdsxztvcznpr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-21T19:48:49.875795Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                    err: <*url.Error | 0x2fae16f952c0>{
                        Op: "Post",
                        URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                        Err: <context.deadlineExceededError>{},
                    },
                },
                restErrMsg: nil,
                decoded: nil,
                permanent: false,
                inner: nil,
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jdsxztvcznpr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x2fae15cf05c0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jdsxztvcznpr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jdsxztvcznpr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster"; let start = datetime(2026-10-21T19:48:49.875795Z);
    let desires = cosmosResourceSnapshots
    | where resourceID startswith strcat(target, "/")
    | where resourceType =~ "microsoft.redhatopenshift/hcpopenshiftclusters/readdesires"
    | extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
    | summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts
    | summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID
    | where isempty(content.deletionTimestamp)
    | extend manifest = content.properties.status.kubeContent
    | where manifest.kind == "HostedCluster" and isempty(manifest.metadata.deletionTimestamp)
    | extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),
        managementCluster = tolower(tostring(content.properties.spec.managementCluster))
    | where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)
    | project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, "-", hostedClusterName);
    let fleet = cosmosResourceSnapshots
    | where resourceType =~ "microsoft.redhatopenshift/stamps/managementclusters"
    | extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
    | join kind=leftsemi desires on environment, region, sourceCluster, managementCluster
    | summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts
    | summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster
    | where isempty(content.deletionTimestamp)
    | extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), "/")[-1])
    | where isnotempty(clusterName)
    | project environment, region, sourceCluster, managementCluster, cluster = clusterName;
    let hcp = desires
    | join kind=inner fleet on environment, region, sourceCluster, managementCluster
    | distinct environment, region, cluster, namespace;
    let inventory = materialize(kubernetesResourceSnapshots
    | where timestamp >= start and objectKind in ("Pod", "Deployment")
    | lookup kind=inner hcp on environment, region, cluster, namespace
    | summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name
    | where event != "Delete" and isempty(object.metadata.deletionTimestamp));
    let pods = inventory
    | where objectKind == "Pod" and object.metadata.labels.app == "private-router" and object.status.phase == "Running"
    | project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);
    let counts = pods | summarize Current = count() by environment, region, cluster, namespace;
    let deployments = inventory
    | where objectKind == "Deployment" and name == "router"
    | project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);
    let summaries = containerLogs
    | where timestamp >= start and container_name == "swift-recorder"
    | where log.controller_name == "swift-router-check" and log.field == "summary"
    | extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)
    | lookup kind=inner hcp on environment, region, cluster, namespace
    | summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID
    | project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),
        SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);
    hcp
    | join kind=leftouter deployments on environment, region, cluster, namespace
    | join kind=leftouter counts on environment, region, cluster, namespace
    | join kind=leftouter pods on environment, region, cluster, namespace
    | join kind=leftouter summaries on environment, region, cluster, namespace, UID
    | project Scope = strcat(environment, "/", region, "/", cluster, "/", namespace),
        Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary
    | order by Scope asc, UID asc
    | take 101, Post "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query": context deadline exceeded
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jdsxztvcznpr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jdsxztvcznpr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-21T19:48:49.875795Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
        err: <*fmt.wrapError | 0x2fae15cf05a0>{
            msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jdsxztvcznpr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-21T19:48:49.875795Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
            err: <*errors.Error | 0x2fae15532370>{
                Op: 1,
                Kind: 7,
                Err: <*fmt.wrapError | 0x2fae15cf0580>{
                    msg: "With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jdsxztvcznpr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-21T19:48:49.875795Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                    err: <*url.Error | 0x2fae16f952c0>{
                        Op: "Post",
                        URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                        Err: <context.deadlineExceededError>{},
                    },
                },
                restErrMsg: nil,
                decoded: nil,
                permanent: false,
                inner: nil,
            },
        },
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2x98xj2zxslb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x30fb1a9da3c0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2x98xj2zxslb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2x98xj2zxslb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster"; let start = datetime(2026-10-21T19:20:56.4222237Z);
    let desires = cosmosResourceSnapshots
    | where resourceID startswith strcat(target, "/")
    | where resourceType =~ "microsoft.redhatopenshift/hcpopenshiftclusters/readdesires"
    | extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
    | summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts
    | summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID
    | where isempty(content.deletionTimestamp)
    | extend manifest = content.properties.status.kubeContent
    | where manifest.kind == "HostedCluster" and isempty(manifest.metadata.deletionTimestamp)
    | extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),
        managementCluster = tolower(tostring(content.properties.spec.managementCluster))
    | where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)
    | project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, "-", hostedClusterName);
    let fleet = cosmosResourceSnapshots
    | where resourceType =~ "microsoft.redhatopenshift/stamps/managementclusters"
    | extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
    | join kind=leftsemi desires on environment, region, sourceCluster, managementCluster
    | summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts
    | summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster
    | where isempty(content.deletionTimestamp)
    | extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), "/")[-1])
    | where isnotempty(clusterName)
    | project environment, region, sourceCluster, managementCluster, cluster = clusterName;
    let hcp = desires
    | join kind=inner fleet on environment, region, sourceCluster, managementCluster
    | distinct environment, region, cluster, namespace;
    let inventory = materialize(kubernetesResourceSnapshots
    | where timestamp >= start and objectKind in ("Pod", "Deployment")
    | lookup kind=inner hcp on environment, region, cluster, namespace
    | summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name
    | where event != "Delete" and isempty(object.metadata.deletionTimestamp));
    let pods = inventory
    | where objectKind == "Pod" and object.metadata.labels.app == "private-router" and object.status.phase == "Running"
    | project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);
    let counts = pods | summarize Current = count() by environment, region, cluster, namespace;
    let deployments = inventory
    | where objectKind == "Deployment" and name == "router"
    | project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);
    let summaries = containerLogs
    | where timestamp >= start and container_name == "swift-recorder"
    | where log.controller_name == "swift-router-check" and log.field == "summary"
    | extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)
    | lookup kind=inner hcp on environment, region, cluster, namespace
    | summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID
    | project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),
        SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);
    hcp
    | join kind=leftouter deployments on environment, region, cluster, namespace
    | join kind=leftouter counts on environment, region, cluster, namespace
    | join kind=leftouter pods on environment, region, cluster, namespace
    | join kind=leftouter summaries on environment, region, cluster, namespace, UID
    | project Scope = strcat(environment, "/", region, "/", cluster, "/", namespace),
        Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary
    | order by Scope asc, UID asc
    | take 101, Post "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query": context deadline exceeded
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2x98xj2zxslb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2x98xj2zxslb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-21T19:20:56.4222237Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
        err: <*fmt.wrapError | 0x30fb1a9da3a0>{
            msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2x98xj2zxslb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-21T19:20:56.4222237Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
            err: <*errors.Error | 0x30fb198a6050>{
                Op: 1,
                Kind: 7,
                Err: <*fmt.wrapError | 0x30fb1a9da380>{
                    msg: "With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2x98xj2zxslb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-21T19:20:56.4222237Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                    err: <*url.Error | 0x30fb1b7ece70>{
                        Op: "Post",
                        URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                        Err: <context.deadlineExceededError>{},
                    },
                },
                restErrMsg: nil,
                decoded: nil,
                permanent: false,
                inner: nil,
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2x98xj2zxslb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x30fb1a9da3c0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2x98xj2zxslb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2x98xj2zxslb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster"; let start = datetime(2026-10-21T19:20:56.4222237Z);
    let desires = cosmosResourceSnapshots
    | where resourceID startswith strcat(target, "/")
    | where resourceType =~ "microsoft.redhatopenshift/hcpopenshiftclusters/readdesires"
    | extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
    | summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts
    | summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID
    | where isempty(content.deletionTimestamp)
    | extend manifest = content.properties.status.kubeContent
    | where manifest.kind == "HostedCluster" and isempty(manifest.metadata.deletionTimestamp)
    | extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),
        managementCluster = tolower(tostring(content.properties.spec.managementCluster))
    | where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)
    | project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, "-", hostedClusterName);
    let fleet = cosmosResourceSnapshots
    | where resourceType =~ "microsoft.redhatopenshift/stamps/managementclusters"
    | extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
    | join kind=leftsemi desires on environment, region, sourceCluster, managementCluster
    | summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts
    | summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster
    | where isempty(content.deletionTimestamp)
    | extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), "/")[-1])
    | where isnotempty(clusterName)
    | project environment, region, sourceCluster, managementCluster, cluster = clusterName;
    let hcp = desires
    | join kind=inner fleet on environment, region, sourceCluster, managementCluster
    | distinct environment, region, cluster, namespace;
    let inventory = materialize(kubernetesResourceSnapshots
    | where timestamp >= start and objectKind in ("Pod", "Deployment")
    | lookup kind=inner hcp on environment, region, cluster, namespace
    | summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name
    | where event != "Delete" and isempty(object.metadata.deletionTimestamp));
    let pods = inventory
    | where objectKind == "Pod" and object.metadata.labels.app == "private-router" and object.status.phase == "Running"
    | project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);
    let counts = pods | summarize Current = count() by environment, region, cluster, namespace;
    let deployments = inventory
    | where objectKind == "Deployment" and name == "router"
    | project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);
    let summaries = containerLogs
    | where timestamp >= start and container_name == "swift-recorder"
    | where log.controller_name == "swift-router-check" and log.field == "summary"
    | extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)
    | lookup kind=inner hcp on environment, region, cluster, namespace
    | summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID
    | project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),
        SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);
    hcp
    | join kind=leftouter deployments on environment, region, cluster, namespace
    | join kind=leftouter counts on environment, region, cluster, namespace
    | join kind=leftouter pods on environment, region, cluster, namespace
    | join kind=leftouter summaries on environment, region, cluster, namespace, UID
    | project Scope = strcat(environment, "/", region, "/", cluster, "/", namespace),
        Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary
    | order by Scope asc, UID asc
    | take 101, Post "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query": context deadline exceeded
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2x98xj2zxslb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2x98xj2zxslb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-21T19:20:56.4222237Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
        err: <*fmt.wrapError | 0x30fb1a9da3a0>{
            msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2x98xj2zxslb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-21T19:20:56.4222237Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
            err: <*errors.Error | 0x30fb198a6050>{
                Op: 1,
                Kind: 7,
                Err: <*fmt.wrapError | 0x30fb1a9da380>{
                    msg: "With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2x98xj2zxslb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-21T19:20:56.4222237Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                    err: <*url.Error | 0x30fb1b7ece70>{
                        Op: "Post",
                        URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                        Err: <context.deadlineExceededError>{},
                    },
                },
                restErrMsg: nil,
                decoded: nil,
                permanent: false,
                inner: nil,
            },
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	9

Affected runs (9)

Date (UTC)	Associated PR	Prow job
2026-10-21 19:43	PR #7573	2112993261764218880
2026-10-21 18:51	PR #7646	2112980138302902272
2026-10-21 18:37	PR #7650	2112976501128302592
2026-10-21 16:35	PR #7592	2112945930595471360
2026-10-21 16:04	PR #7646	2112937980879441920
2026-10-21 15:39	PR #7441	2112931863092793344
2026-10-21 14:12	PR #7641	2112909828169928704
2026-10-21 10:03	PR #7598	2112847311775207424
2026-10-21 02:38	PR #7548	2112735125815955456

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.42
Unexpected error:
    <*fmt.wrapError | 0x208e48105340>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-qzrclp" in resourcegroup="rg-np-noedge-qzrclp-s7mgb584cvbg" to finish creating: GET https://rp.j8936960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/94d119b5-933d-4011-900a-944cda0c66bd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/94d119b5-933d-4011-900a-944cda0c66bd",
      "name": "94d119b5-933d-4011-900a-944cda0c66bd",
      "status": "Failed",
      "startTime": "2026-10-22T00:23:29.776626313Z",
      "endTime": "2026-10-22T00:42:32.225121177Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine r0b1s8w2x6j2z6y-npnoedge-4-20-wks4l-pwh8n: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-qzrclp-s7mgb584cvbg-np-ne-qzrclp--managed/providers/Microsoft.Compute/virtualMachines/r0b1s8w2x6j2z6y-npnoedge-4-20-wks4l-pwh8n to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine r0b1s8w2x6j2z6y-npnoedge-4-20-wks4l-pwh8n: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-qzrclp-s7mgb584cvbg-np-ne-qzrclp--managed/providers/Microsoft.Compute/virtualMachines/r0b1s8w2x6j2z6y-npnoedge-4-20-wks4l-pwh8n to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8936960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/94d119b5-933d-4011-900a-944cda0c66bd
            correlation ID: 9d04cab9-1988-4f3b-99fa-6267fc865f5b
            client request ID: e2044c6b-9c9e-4046-9821-e1d653a15511
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.42
Unexpected error:
    <*fmt.wrapError | 0x208e48105340>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-qzrclp" in resourcegroup="rg-np-noedge-qzrclp-s7mgb584cvbg" to finish creating: GET https://rp.j8936960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/94d119b5-933d-4011-900a-944cda0c66bd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/94d119b5-933d-4011-900a-944cda0c66bd",
      "name": "94d119b5-933d-4011-900a-944cda0c66bd",
      "status": "Failed",
      "startTime": "2026-10-22T00:23:29.776626313Z",
      "endTime": "2026-10-22T00:42:32.225121177Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine r0b1s8w2x6j2z6y-npnoedge-4-20-wks4l-pwh8n: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-qzrclp-s7mgb584cvbg-np-ne-qzrclp--managed/providers/Microsoft.Compute/virtualMachines/r0b1s8w2x6j2z6y-npnoedge-4-20-wks4l-pwh8n to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine r0b1s8w2x6j2z6y-npnoedge-4-20-wks4l-pwh8n: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-qzrclp-s7mgb584cvbg-np-ne-qzrclp--managed/providers/Microsoft.Compute/virtualMachines/r0b1s8w2x6j2z6y-npnoedge-4-20-wks4l-pwh8n to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8936960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/94d119b5-933d-4011-900a-944cda0c66bd
            correlation ID: 9d04cab9-1988-4f3b-99fa-6267fc865f5b
            client request ID: e2044c6b-9c9e-4046-9821-e1d653a15511
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.43
Unexpected error:
    <*fmt.wrapError | 0x15c75a6b8be0>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-nctrfg" in resourcegroup="rg-np-version-upgrade-nctrfg-45222jrbbpsn" to finish creating: GET https://rp.j4218880.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7ae99b05-0591-45be-992e-01736d913464
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7ae99b05-0591-45be-992e-01736d913464",
      "name": "7ae99b05-0591-45be-992e-01736d913464",
      "status": "Failed",
      "startTime": "2026-10-21T21:13:06.466186109Z",
      "endTime": "2026-10-21T21:32:09.368172425Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine x7u1u4c8y7v8s7l-npupgrade-4-20-2k82t-k6pq4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-nctrfg-45222jrbbpsn-np-upgrade-nc-dd441e35/providers/Microsoft.Compute/virtualMachines/x7u1u4c8y7v8s7l-npupgrade-4-20-2k82t-k6pq4 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine x7u1u4c8y7v8s7l-npupgrade-4-20-2k82t-k6pq4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-nctrfg-45222jrbbpsn-np-upgrade-nc-dd441e35/providers/Microsoft.Compute/virtualMachines/x7u1u4c8y7v8s7l-npupgrade-4-20-2k82t-k6pq4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j4218880.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7ae99b05-0591-45be-992e-01736d913464
            correlation ID: e2ddc6fc-2b3c-4d77-8c0d-3a130a41d2d3
            client request ID: 5ad213ec-faaf-4144-a539-72b8c88d6c1f
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.43
Unexpected error:
    <*fmt.wrapError | 0x15c75a6b8be0>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-nctrfg" in resourcegroup="rg-np-version-upgrade-nctrfg-45222jrbbpsn" to finish creating: GET https://rp.j4218880.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7ae99b05-0591-45be-992e-01736d913464
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7ae99b05-0591-45be-992e-01736d913464",
      "name": "7ae99b05-0591-45be-992e-01736d913464",
      "status": "Failed",
      "startTime": "2026-10-21T21:13:06.466186109Z",
      "endTime": "2026-10-21T21:32:09.368172425Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine x7u1u4c8y7v8s7l-npupgrade-4-20-2k82t-k6pq4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-nctrfg-45222jrbbpsn-np-upgrade-nc-dd441e35/providers/Microsoft.Compute/virtualMachines/x7u1u4c8y7v8s7l-npupgrade-4-20-2k82t-k6pq4 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine x7u1u4c8y7v8s7l-npupgrade-4-20-2k82t-k6pq4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-nctrfg-45222jrbbpsn-np-upgrade-nc-dd441e35/providers/Microsoft.Compute/virtualMachines/x7u1u4c8y7v8s7l-npupgrade-4-20-2k82t-k6pq4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j4218880.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7ae99b05-0591-45be-992e-01736d913464
            correlation ID: e2ddc6fc-2b3c-4d77-8c0d-3a130a41d2d3
            client request ID: 5ad213ec-faaf-4144-a539-72b8c88d6c1f
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:177]: failed to create node pool "np-5msk5q" for cluster "cluster-candidate-4-21-5msk5q"
Unexpected error:
    <*fmt.wrapError | 0x2326984a6a0>: 
    failed to create NodePool np-5msk5q: failed waiting for nodepool="np-5msk5q" for cluster "cluster-candidate-4-21-5msk5q" in resourcegroup="rg-candidate-4-21-sv65px5xxhg7" to finish creating: GET https://rp.j7966080.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1106c4fd-dc58-4724-97cb-b35ad7ddc532
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1106c4fd-dc58-4724-97cb-b35ad7ddc532",
      "name": "1106c4fd-dc58-4724-97cb-b35ad7ddc532",
      "status": "Failed",
      "startTime": "2026-10-21T15:12:47.403558891Z",
      "endTime": "2026-10-21T15:31:50.806949626Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine e7l9m0o7f4b1b7k-np-5msk5q-6jh4f-4xx9j: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-sv65px5xxhg7-candidate-5msk5q--managed/providers/Microsoft.Compute/virtualMachines/e7l9m0o7f4b1b7k-np-5msk5q-6jh4f-4xx9j to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine e7l9m0o7f4b1b7k-np-5msk5q-6jh4f-4xx9j: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-sv65px5xxhg7-candidate-5msk5q--managed/providers/Microsoft.Compute/virtualMachines/e7l9m0o7f4b1b7k-np-5msk5q-6jh4f-4xx9j to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7966080.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1106c4fd-dc58-4724-97cb-b35ad7ddc532
            correlation ID: e5915f53-e0ec-4f8c-9b0b-55f6298c1316
            client request ID: 7cdad002-adc5-457c-ae3d-798682bcbe96
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:177]: failed to create node pool "np-5msk5q" for cluster "cluster-candidate-4-21-5msk5q"
Unexpected error:
    <*fmt.wrapError | 0x2326984a6a0>: 
    failed to create NodePool np-5msk5q: failed waiting for nodepool="np-5msk5q" for cluster "cluster-candidate-4-21-5msk5q" in resourcegroup="rg-candidate-4-21-sv65px5xxhg7" to finish creating: GET https://rp.j7966080.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1106c4fd-dc58-4724-97cb-b35ad7ddc532
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1106c4fd-dc58-4724-97cb-b35ad7ddc532",
      "name": "1106c4fd-dc58-4724-97cb-b35ad7ddc532",
      "status": "Failed",
      "startTime": "2026-10-21T15:12:47.403558891Z",
      "endTime": "2026-10-21T15:31:50.806949626Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine e7l9m0o7f4b1b7k-np-5msk5q-6jh4f-4xx9j: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-sv65px5xxhg7-candidate-5msk5q--managed/providers/Microsoft.Compute/virtualMachines/e7l9m0o7f4b1b7k-np-5msk5q-6jh4f-4xx9j to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine e7l9m0o7f4b1b7k-np-5msk5q-6jh4f-4xx9j: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-sv65px5xxhg7-candidate-5msk5q--managed/providers/Microsoft.Compute/virtualMachines/e7l9m0o7f4b1b7k-np-5msk5q-6jh4f-4xx9j to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7966080.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1106c4fd-dc58-4724-97cb-b35ad7ddc532
            correlation ID: e5915f53-e0ec-4f8c-9b0b-55f6298c1316
            client request ID: 7cdad002-adc5-457c-ae3d-798682bcbe96
occurred

Contributing tests (3)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest	1

Affected runs (3)

Date (UTC)	Associated PR	Prow job
2026-10-21 22:55	PR #7646	2113041621648936960
2026-10-21 19:43	PR #7573	2112993261764218880
2026-10-21 14:03	PR #7421	2112907679297966080

ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<u...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens... message truncated; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x3240dc3871e0>: 
    failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-chb4gbwwsmzl" to finish creating: GET https://rp.j0328704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d2c2b5e0-cf5c-4ee8-837f-773d00c86a5c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d2c2b5e0-cf5c-4ee8-837f-773d00c86a5c",
      "name": "d2c2b5e0-cf5c-4ee8-837f-773d00c86a5c",
      "status": "Failed",
      "startTime": "2026-10-21T19:55:46.076391095Z",
      "endTime": "2026-10-21T20:14:46.636602132Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine o5d8k9t4u7s8l8x-autoscale-az-qfbfg-r6wk5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-chb4gbwwsmzl--managed/providers/Microsoft.Compute/virtualMachines/o5d8k9t4u7s8l8x-autoscale-az-qfbfg-r6wk5 to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine o5d8k9t4u7s8l8x-autoscale-az-qfbfg-r6wk5: Failed: interfaces failed to create or update. err: failed to get existing resource np-autoscaling-chb4gbwwsmzl--managed/o5d8k9t4u7s8l8x-autoscale-az-qfbfg-r6wk5-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-chb4gbwwsmzl--managed/providers/Microsoft.Network/networkInterfaces/o5d8k9t4u7s8l8x-autoscale-az-qfbfg-r6wk5-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '39eefc4f-5e24-491d-9f09-bcc0ad573bcf' exceeded. Please try again after '10' seconds after additional tokens... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j0328704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d2c2b5e0-cf5c-4ee8-837f-773d00c86a5c
            correlation ID: d1604de5-04bc-4cfd-a5ad-f2222992db63
            client request ID: 22866604-4f96-48c8-8339-c93f8fcbb1b3
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x3240dc3871e0>: 
    failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-chb4gbwwsmzl" to finish creating: GET https://rp.j0328704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d2c2b5e0-cf5c-4ee8-837f-773d00c86a5c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d2c2b5e0-cf5c-4ee8-837f-773d00c86a5c",
      "name": "d2c2b5e0-cf5c-4ee8-837f-773d00c86a5c",
      "status": "Failed",
      "startTime": "2026-10-21T19:55:46.076391095Z",
      "endTime": "2026-10-21T20:14:46.636602132Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine o5d8k9t4u7s8l8x-autoscale-az-qfbfg-r6wk5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-chb4gbwwsmzl--managed/providers/Microsoft.Compute/virtualMachines/o5d8k9t4u7s8l8x-autoscale-az-qfbfg-r6wk5 to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine o5d8k9t4u7s8l8x-autoscale-az-qfbfg-r6wk5: Failed: interfaces failed to create or update. err: failed to get existing resource np-autoscaling-chb4gbwwsmzl--managed/o5d8k9t4u7s8l8x-autoscale-az-qfbfg-r6wk5-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-chb4gbwwsmzl--managed/providers/Microsoft.Network/networkInterfaces/o5d8k9t4u7s8l8x-autoscale-az-qfbfg-r6wk5-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '39eefc4f-5e24-491d-9f09-bcc0ad573bcf' exceeded. Please try again after '10' seconds after additional tokens... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j0328704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d2c2b5e0-cf5c-4ee8-837f-773d00c86a5c
            correlation ID: d1604de5-04bc-4cfd-a5ad-f2222992db63
            client request ID: 22866604-4f96-48c8-8339-c93f8fcbb1b3
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xde2a318980>: 
    failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-pzz9klzh962n" to finish creating: GET https://rp.j9261056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/5af4fa15-307d-466b-b9c9-c0a6855da512
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/5af4fa15-307d-466b-b9c9-c0a6855da512",
      "name": "5af4fa15-307d-466b-b9c9-c0a6855da512",
      "status": "Failed",
      "startTime": "2026-10-21T14:36:54.679484558Z",
      "endTime": "2026-10-21T14:56:01.447308725Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllMachinesReady is False: 2 of 4 machines are not ready\nMachine h1n9d3h2q0n6y9o-autoscale-az-p4nbn-q4z78: Failed: interfaces failed to create or update. err: failed to get existing resource np-autoscaling-pzz9klzh962n--managed/h1n9d3h2q0n6y9o-autoscale-az-p4nbn-q4z78-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-pzz9klzh962n--managed/providers/Microsoft.Network/networkInterfaces/h1n9d3h2q0n6y9o-autoscale-az-p4nbn-q4z78-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '0d25d885-2cd8-4972-b610-4d85881c1ec4' exceeded. Please try again after '10' seconds after additional tokens... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9261056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/5af4fa15-307d-466b-b9c9-c0a6855da512
            correlation ID: 873f9122-b54e-46d0-9f76-ed588ac965d9
            client request ID: 1463f1a9-eece-4333-af81-213c95629aa9
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xde2a318980>: 
    failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-pzz9klzh962n" to finish creating: GET https://rp.j9261056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/5af4fa15-307d-466b-b9c9-c0a6855da512
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/5af4fa15-307d-466b-b9c9-c0a6855da512",
      "name": "5af4fa15-307d-466b-b9c9-c0a6855da512",
      "status": "Failed",
      "startTime": "2026-10-21T14:36:54.679484558Z",
      "endTime": "2026-10-21T14:56:01.447308725Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllMachinesReady is False: 2 of 4 machines are not ready\nMachine h1n9d3h2q0n6y9o-autoscale-az-p4nbn-q4z78: Failed: interfaces failed to create or update. err: failed to get existing resource np-autoscaling-pzz9klzh962n--managed/h1n9d3h2q0n6y9o-autoscale-az-p4nbn-q4z78-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-pzz9klzh962n--managed/providers/Microsoft.Network/networkInterfaces/h1n9d3h2q0n6y9o-autoscale-az-p4nbn-q4z78-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '0d25d885-2cd8-4972-b610-4d85881c1ec4' exceeded. Please try again after '10' seconds after additional tokens... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9261056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/5af4fa15-307d-466b-b9c9-c0a6855da512
            correlation ID: 873f9122-b54e-46d0-9f76-ed588ac965d9
            client request ID: 1463f1a9-eece-4333-af81-213c95629aa9
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-21 18:57	PR #7662	2112981681840328704
2026-10-21 13:21	PR #7582	2112896971529261056

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: NotReady: * NodeHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:96]: failed to create node pool "nodepool-128GiB" with 128GiB osDisk
Unexpected error:
    <*fmt.wrapError | 0x13054070020>: 
    failed to create NodePool nodepool-128GiB: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-2lvhqm85ncwq" to finish creating: GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/15626b81-4f40-4cd8-b91d-a63c8ef6d9cb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/15626b81-4f40-4cd8-b91d-a63c8ef6d9cb",
      "name": "15626b81-4f40-4cd8-b91d-a63c8ef6d9cb",
      "status": "Failed",
      "startTime": "2026-10-21T14:23:04.095189902Z",
      "endTime": "2026-10-21T14:42:12.03625287Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine g2v1j2n5e4d4v3a-nodepool-128gib-x54k8-xjpcf: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine g2v1j2n5e4d4v3a-nodepool-128gib-x54k8-xjpcf: NotReady: * NodeHealthy:\n  * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/15626b81-4f40-4cd8-b91d-a63c8ef6d9cb
            correlation ID: 317d5d8b-fd68-46aa-9ea3-280cce8619ef
            client request ID: 9d6ff53f-a6f9-4a01-87a2-797532ad561c
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:96]: failed to create node pool "nodepool-128GiB" with 128GiB osDisk
Unexpected error:
    <*fmt.wrapError | 0x13054070020>: 
    failed to create NodePool nodepool-128GiB: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-2lvhqm85ncwq" to finish creating: GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/15626b81-4f40-4cd8-b91d-a63c8ef6d9cb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/15626b81-4f40-4cd8-b91d-a63c8ef6d9cb",
      "name": "15626b81-4f40-4cd8-b91d-a63c8ef6d9cb",
      "status": "Failed",
      "startTime": "2026-10-21T14:23:04.095189902Z",
      "endTime": "2026-10-21T14:42:12.03625287Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine g2v1j2n5e4d4v3a-nodepool-128gib-x54k8-xjpcf: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine g2v1j2n5e4d4v3a-nodepool-128gib-x54k8-xjpcf: NotReady: * NodeHealthy:\n  * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/15626b81-4f40-4cd8-b91d-a63c8ef6d9cb
            correlation ID: 317d5d8b-fd68-46aa-9ea3-280cce8619ef
            client request ID: 9d6ff53f-a6f9-4a01-87a2-797532ad561c
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.43
Unexpected error:
    <*fmt.wrapError | 0x36f5972f0be0>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-bc4dk5" in resourcegroup="rg-np-version-upgrade-bc4dk5-7d4d8bxtxcwp" to finish creating: GET https://rp.j9440128.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d7299924-a73f-441d-881d-7c973d7b4248
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d7299924-a73f-441d-881d-7c973d7b4248",
      "name": "d7299924-a73f-441d-881d-7c973d7b4248",
      "status": "Failed",
      "startTime": "2026-10-21T14:06:44.0057756Z",
      "endTime": "2026-10-21T14:25:46.108337887Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine a5s6i4v4i7m1o8n-npupgrade-4-20-cctdj-kqldf: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine a5s6i4v4i7m1o8n-npupgrade-4-20-cctdj-kqldf: NotReady: * NodeHealthy:\n  * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9440128.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d7299924-a73f-441d-881d-7c973d7b4248
            correlation ID: 335f4e13-f2a1-4c45-8bda-c2e1119ff7b6
            client request ID: d412cd2d-c02a-4e92-80cf-e84a62b46887
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.43
Unexpected error:
    <*fmt.wrapError | 0x36f5972f0be0>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-bc4dk5" in resourcegroup="rg-np-version-upgrade-bc4dk5-7d4d8bxtxcwp" to finish creating: GET https://rp.j9440128.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d7299924-a73f-441d-881d-7c973d7b4248
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d7299924-a73f-441d-881d-7c973d7b4248",
      "name": "d7299924-a73f-441d-881d-7c973d7b4248",
      "status": "Failed",
      "startTime": "2026-10-21T14:06:44.0057756Z",
      "endTime": "2026-10-21T14:25:46.108337887Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine a5s6i4v4i7m1o8n-npupgrade-4-20-cctdj-kqldf: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine a5s6i4v4i7m1o8n-npupgrade-4-20-cctdj-kqldf: NotReady: * NodeHealthy:\n  * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9440128.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d7299924-a73f-441d-881d-7c973d7b4248
            correlation ID: 335f4e13-f2a1-4c45-8bda-c2e1119ff7b6
            client request ID: d412cd2d-c02a-4e92-80cf-e84a62b46887
occurred

Contributing tests (2)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and custom node pool osDisk size	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-21 12:58	PR #7635	2112891349849083904
2026-10-21 12:35	PR #7608	2112885462539440128

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [descendantResources] remaining resources: serviceProviderClusters; provider...

full failure pattern:

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
    <*errors.joinError | 0x1258c35821b0>: 
    resource group "pullsecret-test-tqqf49s4vksc": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-tqqf49s4vksc" to finish deleting: GET https://rp.j8690432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/34d62dc1-1d60-4614-9fd4-7a691fd278e4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/34d62dc1-1d60-4614-9fd4-7a691fd278e4",
      "name": "34d62dc1-1d60-4614-9fd4-7a691fd278e4",
      "status": "Failed",
      "startTime": "2026-10-21T18:09:45.740287452Z",
      "endTime": "2026-10-21T18:33:49.851023477Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8690432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/34d62dc1-1d60-4614-9fd4-7a691fd278e4
            correlation ID: 73ce79e7-e503-497f-984d-52615b1cd398
            client request ID: 38899611-f32f-4f30-8353-0d70856187d8
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
    <*errors.joinError | 0x1258c35821b0>: 
    resource group "pullsecret-test-tqqf49s4vksc": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-tqqf49s4vksc" to finish deleting: GET https://rp.j8690432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/34d62dc1-1d60-4614-9fd4-7a691fd278e4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/34d62dc1-1d60-4614-9fd4-7a691fd278e4",
      "name": "34d62dc1-1d60-4614-9fd4-7a691fd278e4",
      "status": "Failed",
      "startTime": "2026-10-21T18:09:45.740287452Z",
      "endTime": "2026-10-21T18:33:49.851023477Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8690432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/34d62dc1-1d60-4614-9fd4-7a691fd278e4
            correlation ID: 73ce79e7-e503-497f-984d-52615b1cd398
            client request ID: 38899611-f32f-4f30-8353-0d70856187d8
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
    <*errors.joinError | 0x2e81818a0078>: 
    resource group "patch-tags-2h6vxldjkpd2": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-2h6vxldjkpd2" to finish deleting: GET https://rp.j5137024.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e23640fa-33a8-41b4-bf0c-88ace3164eea
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e23640fa-33a8-41b4-bf0c-88ace3164eea",
      "name": "e23640fa-33a8-41b4-bf0c-88ace3164eea",
      "status": "Failed",
      "startTime": "2026-10-21T17:06:29.350825983Z",
      "endTime": "2026-10-21T17:30:32.428573897Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5137024.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e23640fa-33a8-41b4-bf0c-88ace3164eea
            correlation ID: de464733-8684-42c7-8fd5-e0982a079068
            client request ID: 2dec17a5-5e85-44c4-af64-7517b9464dd3
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
    <*errors.joinError | 0x2e81818a0078>: 
    resource group "patch-tags-2h6vxldjkpd2": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-2h6vxldjkpd2" to finish deleting: GET https://rp.j5137024.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e23640fa-33a8-41b4-bf0c-88ace3164eea
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e23640fa-33a8-41b4-bf0c-88ace3164eea",
      "name": "e23640fa-33a8-41b4-bf0c-88ace3164eea",
      "status": "Failed",
      "startTime": "2026-10-21T17:06:29.350825983Z",
      "endTime": "2026-10-21T17:30:32.428573897Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5137024.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e23640fa-33a8-41b4-bf0c-88ace3164eea
            correlation ID: de464733-8684-42c7-8fd5-e0982a079068
            client request ID: 2dec17a5-5e85-44c4-af64-7517b9464dd3
occurred

Contributing tests (2)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-21 16:51	PR #7664	2112949864508690432
2026-10-21 15:53	PR #7662	2112935310475137024

alert [svc] KubePodCrashLooping fired

full failure pattern:

alert [svc] KubePodCrashLooping fired

Signal:

alert

none

Full failure examples (2)

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-22T00:15:27Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0588800-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.150:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0588800/providers/microsoft.monitor/accounts/services-j0588800", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7f6f9c56c5-dkzx8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="181b057e-60cb-49ad-af81-7495d4c18ee7"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7f6f9c56c5-dkzx8 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-22T00:15:27Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0588800-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.248:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0588800/providers/microsoft.monitor/accounts/services-j0588800", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7f6f9c56c5-s8wts", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="4f27896e-6819-4b13-8573-f17ef370d5ca"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7f6f9c56c5-s8wts (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-22T00:18:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0588800-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.248:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0588800/providers/microsoft.monitor/accounts/services-j0588800", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7f6f9c56c5-s8wts", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="4f27896e-6819-4b13-8573-f17ef370d5ca"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7f6f9c56c5-s8wts (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-22T00:18:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0588800-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.150:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0588800/providers/microsoft.monitor/accounts/services-j0588800", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7f6f9c56c5-dkzx8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="181b057e-60cb-49ad-af81-7495d4c18ee7"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7f6f9c56c5-dkzx8 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-21T08:50:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j9212928-svc", component="kubernetes-infrastructure", container="cilium-agent", endpoint="http", environment="ci01", instance="10.128.64.52:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9212928/providers/microsoft.monitor/accounts/services-j9212928", namespace="kube-system", pod="cilium-ksfwq", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="c90d39c5-332e-4d3e-8b00-995e03055f7c"
  Description: Pod kube-system/cilium-ksfwq (cilium-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-21T08:50:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j9212928-svc", component="kubernetes-infrastructure", container="cilium-agent", endpoint="http", environment="ci01", instance="10.128.64.52:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9212928/providers/microsoft.monitor/accounts/services-j9212928", namespace="kube-system", pod="cilium-ksfwq", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="c90d39c5-332e-4d3e-8b00-995e03055f7c"
  Description: Pod kube-system/cilium-ksfwq (cilium-agent) is in waiting state (reason: "CrashLoopBackOff").

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert KubePodCrashLooping does not fire	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-21 22:59	PR #7662	2113042472400588800
2026-10-21 07:35	PR #7592	2112809978489212928

failed to run istio upgrade step: failed to start canary: start-canary: ARM operation timed out or failed: context deadline exceeded

full failure pattern:

failed to run istio upgrade step: failed to start canary: start-canary: ARM operation timed out or failed: context deadline exceeded

Signal:

provision

none

Full failure examples (2)

time=2026-10-21T22:45:12.476Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade description="Step istio-upgrade\n  Kind: IstioUpgrade\n  AKSCluster: {{ svc.aks.name }}\n"
time=2026-10-21T22:45:13.969Z level=INFO msg="Istio upgrade -- cluster state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j3213568-svc versions=asm-1-30 logger=istio-upgrade k8sVersion=1.35 provisioningState=Succeeded installedRevisions=[asm-1-29] availableUpgrades=[asm-1-30] upgradeInProgress=false
time=2026-10-21T22:45:13.969Z level=INFO msg="Upgrading to svc.istio.versions target" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j3213568-svc versions=asm-1-30 logger=istio-upgrade from=asm-1-29 to=asm-1-30 k8sVersion=1.35
time=2026-10-21T22:45:14.834Z level=INFO msg="Istio upgrade -- mesh state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j3213568-svc versions=asm-1-30 logger=istio-upgrade namespaces=0 controlPlanes=asm-1-29(2/2) ingressGateways=1 webhooks=1 configMaps=2
time=2026-10-21T22:45:14.834Z level=INFO msg="Starting canary -- installing target alongside current" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j3213568-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-21T22:45:14.834Z level=INFO msg="Starting ARM operation" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade stepID=Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade operation=start-canary cluster=ci01-j3213568-svc
time=2026-10-21T23:00:17.748Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade err="failed to run istio upgrade step: failed to start canary: start-canary: ARM operation timed out or failed: context deadline exceeded"

time=2026-10-21T02:49:42.974Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade description="Step istio-upgrade\n  Kind: IstioUpgrade\n  AKSCluster: {{ svc.aks.name }}\n"
time=2026-10-21T02:49:44.531Z level=INFO msg="Istio upgrade -- cluster state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8353280-svc versions=asm-1-30 logger=istio-upgrade k8sVersion=1.35 provisioningState=Succeeded installedRevisions=[asm-1-29] availableUpgrades=[asm-1-30] upgradeInProgress=false
time=2026-10-21T02:49:44.531Z level=INFO msg="Upgrading to svc.istio.versions target" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8353280-svc versions=asm-1-30 logger=istio-upgrade from=asm-1-29 to=asm-1-30 k8sVersion=1.35
time=2026-10-21T02:49:45.285Z level=INFO msg="Istio upgrade -- mesh state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8353280-svc versions=asm-1-30 logger=istio-upgrade namespaces=0 controlPlanes=asm-1-29(2/2) ingressGateways=1 webhooks=1 configMaps=2
time=2026-10-21T02:49:45.285Z level=INFO msg="Starting canary -- installing target alongside current" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8353280-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-21T02:49:45.285Z level=INFO msg="Starting ARM operation" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade stepID=Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade operation=start-canary cluster=ci01-j8353280-svc
time=2026-10-21T03:04:48.226Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade err="failed to run istio upgrade step: failed to start canary: start-canary: ARM operation timed out or failed: context deadline exceeded"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-10-21 21:41	PR #7668	2113022983743213568
2026-10-21 01:56	PR #7073	2112724660528353280

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-4-22-spmngrg5sbjv/cluster-candidate-4-22-rgcfjt should provision
Unexpected error:
    <*fmt.wrapError | 0x306e6fa50b60>: 
    failed to create HCP cluster cluster-candidate-4-22-rgcfjt: failed waiting for cluster="cluster-candidate-4-22-rgcfjt" in resourcegroup="rg-candidate-4-22-spmngrg5sbjv" to finish creating: GET https://rp.j4079360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2fe0ff0e-edb6-4d7f-be94-4e97f7a4cb44
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2fe0ff0e-edb6-4d7f-be94-4e97f7a4cb44",
      "name": "2fe0ff0e-edb6-4d7f-be94-4e97f7a4cb44",
      "status": "Failed",
      "startTime": "2026-10-21T05:26:28.567903276Z",
      "endTime": "2026-10-21T05:55:29.560038066Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.19 is Partial (want Completed), started 22m58s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j4079360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2fe0ff0e-edb6-4d7f-be94-4e97f7a4cb44
            correlation ID: eb19cda9-4b93-4643-b784-46181a5fe6fb
            client request ID: b8394245-59f5-4d02-b42c-bba6984ea587
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-4-22-spmngrg5sbjv/cluster-candidate-4-22-rgcfjt should provision
Unexpected error:
    <*fmt.wrapError | 0x306e6fa50b60>: 
    failed to create HCP cluster cluster-candidate-4-22-rgcfjt: failed waiting for cluster="cluster-candidate-4-22-rgcfjt" in resourcegroup="rg-candidate-4-22-spmngrg5sbjv" to finish creating: GET https://rp.j4079360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2fe0ff0e-edb6-4d7f-be94-4e97f7a4cb44
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2fe0ff0e-edb6-4d7f-be94-4e97f7a4cb44",
      "name": "2fe0ff0e-edb6-4d7f-be94-4e97f7a4cb44",
      "status": "Failed",
      "startTime": "2026-10-21T05:26:28.567903276Z",
      "endTime": "2026-10-21T05:55:29.560038066Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.19 is Partial (want Completed), started 22m58s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j4079360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2fe0ff0e-edb6-4d7f-be94-4e97f7a4cb44
            correlation ID: eb19cda9-4b93-4643-b784-46181a5fe6fb
            client request ID: b8394245-59f5-4d02-b42c-bba6984ea587
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-21 04:34	PR #7133	2112764404264079360

ERROR CODE: DeploymentFailed; detail code OverconstrainedZonalAllocationRequest; detail message Allocation failed. VM(s) with the following constraints cannot be allocated, becaus...

full failure pattern:

ERROR CODE: DeploymentFailed; detail code OverconstrainedZonalAllocationRequest; detail message Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\ - Availability Zone\\ - Differencing (Ephemeral) Disks\\ - Networking Constraints (such as Accelerated Networking or IPv6)\\ - VM Size\\; provider Microsoft.Compute

Signal:

provision

none

Full failure examples (1)

time=2026-10-21T18:14:19.941Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-10-21T18:27:57.001Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 err="stamp 2: error running Shell Step, failed to execute shell command: {\"time\":\"2026-10-21T18:14:19.952926036Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j6794752-mgmt-2\",\"clusterName\":\"ci01-j6794752-mgmt-2\",\"region\":\"centralus\"}\n{\"time\":\"2026-10-21T18:14:22.491703162Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j6794752-mgmt-2\",\"clusterName\":\"ci01-j6794752-mgmt-2\",\"region\":\"centralus\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-10-21T18:14:22.594394416Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j6794752-mgmt-2\",\"clusterName\":\"ci01-j6794752-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n{\"time\":\"2026-10-21T18:27:56.998402918Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j6794752-mgmt-2\",\"clusterName\":\"ci01-j6794752-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"8102785885B170C8\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6794752-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j6794752-mgmt-2/agentPools/userswft3\",\"resourceName\":\"ci01-j6794752-mgmt-2/userswft3\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-10-21T18:23:06.0910187Z\",\"duration\":\"PT2M24.1616872S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-10-21T18:27:56.998464939Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j6794752-mgmt-2\",\"clusterName\":\"ci01-j6794752-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"0BDA719BF64D0BED\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6794752-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j6794752-mgmt-2/agentPools/infra1\",\"resourceName\":\"ci01-j6794752-mgmt-2/infra1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-10-21T18:14:24.88832Z\",\"duration\":\"PT13M8.3947089S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-10-21T18:27:56.99847726Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j6794752-mgmt-2\",\"clusterName\":\"ci01-j6794752-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"E62198E06C157714\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6794752-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j6794752-mgmt-2/agentPools/userswft1\",\"resourceName\":\"ci01-j6794752-mgmt-2/userswft1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-10-21T18:23:06.0910187Z\",\"duration\":\"PT1M35.4239028S\",\"statusCode\":\"Conflict\",\"statusMessage\":{\"error\":{\"code\":\"OverconstrainedZonalAllocationRequest\",\"details\":[{\"code\":\"OverconstrainedZonalAllocationRequest\",\"message\":\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\n  - Availability Zone\\n  - Differencing (Ephemeral) Disks\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\n  - VM Size\\n\",\"target\":\"2\"}],\"message\":\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6794752-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft1-13745880-vmss'. Please try alternative sizes 'Standard_D16s_v6' (zones 3, 1, 2), 'Standard_D16ds_v6' (zones 2, 1, 3), 'Standard_D16ds_v6' (eastus2, zones 3, 1, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6794752-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft1-13745880-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\n  - Availability Zone\\n  - Differencing (Ephemeral) Disks\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\n  - VM Size\\n\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-10-21T18:27:56.998529601Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j6794752-mgmt-2\",\"clusterName\":\"ci01-j6794752-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"DE9772B750D369C1\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6794752-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j6794752-mgmt-2/agentPools/infra2\",\"resourceName\":\"ci01-j6794752-mgmt-2/infra2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-10-21T18:23:06.0910187Z\",\"duration\":\"PT1M36.8052474S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-10-21T18:27:56.998540041Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j6794752-mgmt-2\",\"clusterName\":\"ci01-j6794752-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"C04DDBEF7CE6AEB7\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6794752-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j6794752-mgmt-2/agentPools/userswft2\",\"resourceName\":\"ci01-j6794752-mgmt-2/userswft2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-10-21T18:23:06.0910187Z\",\"duration\":\"PT2M53.1939471S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-10-21T18:27:56.998550371Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j6794752-mgmt-2\",\"clusterName\":\"ci01-j6794752-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"7938AC3EB9262313\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6794752-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j6794752-mgmt-2\",\"resourceName\":\"ci01-j6794752-mgmt-2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-10-21T18:14:24.88832Z\",\"duration\":\"PT8M41.1379305S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-10-21T18:27:56.998572712Z\",\"level\":\"ERROR\",\"msg\":\"run failed\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j6794752-mgmt-2\",\"clusterName\":\"ci01-j6794752-mgmt-2\",\"region\":\"centralus\",\"err\":\"waiting for deployment aks-cluster-create: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6794752-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-create/operationStatuses/08584090004226967454\\n--------------------------------------------------------------------------------\\nRESPONSE 200: 200 OK\\nERROR CODE: DeploymentFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"status\\\": \\\"Failed\\\",\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"DeploymentFailed\\\",\\n    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6794752-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-create\\\",\\n    \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\n    \\\"details\\\": [\\n      {\\n        \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6794752-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j6794752-mgmt-2/agentPools/userswft1\\\",\\n        \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\n        \\\"details\\\": [\\n          {\\n            \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\n            \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6794752-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft1-13745880-vmss'. Please try alternative sizes 'Standard_D16s_v6' (zones 3, 1, 2), 'Standard_D16ds_v6' (zones 2, 1, 3), 'Standard_D16ds_v6' (eastus2, zones 3, 1, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6794752-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft1-13745880-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n  - Availability Zone\\\\n  - Differencing (Ephemeral) Disks\\\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n  - VM Size\\\\n\\\",\\n            \\\"details\\\": [\\n              {\\n                \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\n                \\\"target\\\": \\\"2\\\",\\n                \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n  - Availability Zone\\\\n  - Differencing (Ephemeral) Disks\\\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n  - VM Size\\\\n\\\"\\n              }\\n            ]\\n          }\\n        ]\\n      }\\n    ]\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\",\"elapsed\":\"13m37.045666056s\"}\n exit status 1"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster-create (stamp=2)	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-21 17:43	PR #7178	2112962903886794752

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0x2f37765b920>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-7xsbkjfw6rsz" to finish creating: GET https://rp.j2736896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a4c6551c-33dc-4004-bb01-0a2aa772eecc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a4c6551c-33dc-4004-bb01-0a2aa772eecc",
      "name": "a4c6551c-33dc-4004-bb01-0a2aa772eecc",
      "status": "Failed",
      "startTime": "2026-10-21T06:55:23.209858283Z",
      "endTime": "2026-10-21T07:14:31.602256912Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine l0l3p9l4t2w4v4g-np-1-rqp7k-l24nh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-7xsbkjfw6rsz--managed/providers/Microsoft.Compute/virtualMachines/l0l3p9l4t2w4v4g-np-1-rqp7k-l24nh to exist\nMachine l0l3p9l4t2w4v4g-np-1-rqp7k-mhsz6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-7xsbkjfw6rsz--managed/providers/Microsoft.Compute/virtualMachines/l0l3p9l4t2w4v4g-np-1-rqp7k-mhsz6 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine l0l3p9l4t2w4v4g-np-1-rqp7k-l24nh: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-7xsbkjfw6rsz--managed/providers/Microsoft.Compute/virtualMachines/l0l3p9l4t2w4v4g-np-1-rqp7k-l24nh to exist\nMachine l0l3p9l4t2w4v4g-np-1-rqp7k-mhsz6: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-7xsbkjfw6rsz--managed/providers/Microsoft.Compute/virtualMachines/l0l3p9l4t2w4v4g-np-1-rqp7k-mhsz6 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2736896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a4c6551c-33dc-4004-bb01-0a2aa772eecc
            correlation ID: 6ad3d41d-5643-43b0-aa39-e8dd37fe508a
            client request ID: 2b4396d2-9eda-4e0c-b6dc-3f33c00bc172
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0x2f37765b920>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-7xsbkjfw6rsz" to finish creating: GET https://rp.j2736896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a4c6551c-33dc-4004-bb01-0a2aa772eecc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a4c6551c-33dc-4004-bb01-0a2aa772eecc",
      "name": "a4c6551c-33dc-4004-bb01-0a2aa772eecc",
      "status": "Failed",
      "startTime": "2026-10-21T06:55:23.209858283Z",
      "endTime": "2026-10-21T07:14:31.602256912Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine l0l3p9l4t2w4v4g-np-1-rqp7k-l24nh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-7xsbkjfw6rsz--managed/providers/Microsoft.Compute/virtualMachines/l0l3p9l4t2w4v4g-np-1-rqp7k-l24nh to exist\nMachine l0l3p9l4t2w4v4g-np-1-rqp7k-mhsz6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-7xsbkjfw6rsz--managed/providers/Microsoft.Compute/virtualMachines/l0l3p9l4t2w4v4g-np-1-rqp7k-mhsz6 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine l0l3p9l4t2w4v4g-np-1-rqp7k-l24nh: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-7xsbkjfw6rsz--managed/providers/Microsoft.Compute/virtualMachines/l0l3p9l4t2w4v4g-np-1-rqp7k-l24nh to exist\nMachine l0l3p9l4t2w4v4g-np-1-rqp7k-mhsz6: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-7xsbkjfw6rsz--managed/providers/Microsoft.Compute/virtualMachines/l0l3p9l4t2w4v4g-np-1-rqp7k-mhsz6 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2736896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a4c6551c-33dc-4004-bb01-0a2aa772eecc
            correlation ID: 6ad3d41d-5643-43b0-aa39-e8dd37fe508a
            client request ID: 2b4396d2-9eda-4e0c-b6dc-3f33c00bc172
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-21 05:55	PR #7073	2112784863382736896

VerifyAllAPIServicesAvailable failed: apiservice/v1beta1

full failure pattern:

VerifyAllAPIServicesAvailable failed: apiservice/v1beta1

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:180]: cluster viability check failed
Expected success, but got an error:
    <*errors.joinError | 0x27455a542030>: 
    VerifyAllAPIServicesAvailable failed: apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.129.0.21:10250/apis/metrics.k8s.io/v1beta1: Get "https://10.129.0.21:10250/apis/metrics.k8s.io/v1beta1": tls: failed to verify certificate: x509: certificate signed by unknown authority
    {
        errs: [
            <*fmt.wrapError | 0x27455a1f8020>{
                msg: "VerifyAllAPIServicesAvailable failed: apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.129.0.21:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.129.0.21:10250/apis/metrics.k8s.io/v1beta1\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
                err: <*errors.joinError | 0x27455a436c78>{
                    errs: [
                        <*errors.errorString | 0x27455ba317e0>{
                            s: "apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.129.0.21:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.129.0.21:10250/apis/metrics.k8s.io/v1beta1\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
                        },
                    ],
                },
            },
        ],
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:180]: cluster viability check failed
Expected success, but got an error:
    <*errors.joinError | 0x27455a542030>: 
    VerifyAllAPIServicesAvailable failed: apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.129.0.21:10250/apis/metrics.k8s.io/v1beta1: Get "https://10.129.0.21:10250/apis/metrics.k8s.io/v1beta1": tls: failed to verify certificate: x509: certificate signed by unknown authority
    {
        errs: [
            <*fmt.wrapError | 0x27455a1f8020>{
                msg: "VerifyAllAPIServicesAvailable failed: apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.129.0.21:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.129.0.21:10250/apis/metrics.k8s.io/v1beta1\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
                err: <*errors.joinError | 0x27455a436c78>{
                    errs: [
                        <*errors.errorString | 0x27455ba317e0>{
                            s: "apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.129.0.21:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.129.0.21:10250/apis/metrics.k8s.io/v1beta1\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
                        },
                    ],
                },
            },
        ],
    }

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 5.1	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-21 12:08	PR #7655	2112878574095568896

alert [svc] KubeDeploymentReplicasMismatch fired

full failure pattern:

alert [svc] KubeDeploymentReplicasMismatch fired

Signal:

alert

none

Full failure examples (1)

alert fired 24 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-22T00:37:19Z
  Ended:       2026-10-22T00:42:30Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0588800-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.248:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0588800/providers/microsoft.monitor/accounts/services-j0588800", namespace="klusterlet-2tjg9csqcrisiulhj27orktv3umnod15", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-msgjh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2tjg9csqcrisiulhj27orktv3umnod15/config-policy-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-22T00:37:19Z
  Ended:       2026-10-22T00:42:30Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0588800-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="governance-policy-framework", endpoint="http", environment="ci01", instance="10.128.64.248:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0588800/providers/microsoft.monitor/accounts/services-j0588800", namespace="klusterlet-2tjg9csqcrisiulhj27orktv3umnod15", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-msgjh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2tjg9csqcrisiulhj27orktv3umnod15/governance-policy-framework has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-22T00:53:27Z
  Ended:       2026-10-22T01:09:27Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0588800-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.248:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0588800/providers/microsoft.monitor/accounts/services-j0588800", namespace="klusterlet-2tjgi3mscvh8c352mcr946oq21vref8m", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-msgjh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2tjgi3mscvh8c352mcr946oq21vref8m/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-10-22T00:53:27Z
  Ended:       2026-10-22T01:10:27Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0588800-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="governance-policy-framework", endpoint="http", environment="ci01", instance="10.128.64.248:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0588800/providers/microsoft.monitor/accounts/services-j0588800", namespace="klusterlet-2tjgi3mscvh8c352mcr946oq21vref8m", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-msgjh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2tjgi3mscvh8c352mcr946oq21vref8m/governance-policy-framework has not matched the expected number of replicas for longer than 30 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-10-22T00:53:27Z
  Ended:       2026-10-22T01:10:27Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0588800-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.248:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0588800/providers/microsoft.monitor/accounts/services-j0588800", namespace="klusterlet-2tjgi3mscvh8c352mcr946oq21vref8m", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-msgjh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2tjgi3mscvh8c352mcr946oq21vref8m/config-policy-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-10-22T00:53:27Z
  Ended:       2026-10-22T01:09:27Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0588800-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.248:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0588800/providers/microsoft.monitor/accounts/services-j0588800", namespace="klusterlet-2tjgi3mscvh8c352mcr946oq21vref8m", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-msgjh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2tjgi3mscvh8c352mcr946oq21vref8m/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-10-22T00:53:27Z
  Ended:       2026-10-22T01:10:27Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0588800-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="governance-policy-framework", endpoint="http", environment="ci01", instance="10.128.64.248:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0588800/providers/microsoft.monitor/accounts/services-j0588800", namespace="klusterlet-2tjgi3mscvh8c352mcr946oq21vref8m", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-msgjh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2tjgi3mscvh8c352mcr946oq21vref8m/governance-policy-framework has not matched the expected number of replicas for longer than 30 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-10-22T00:53:27Z
  Ended:       2026-10-22T01:10:27Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0588800-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.248:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0588800/providers/microsoft.monitor/accounts/services-j0588800", namespace="klusterlet-2tjgi3mscvh8c352mcr946oq21vref8m", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-msgjh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2tjgi3mscvh8c352mcr946oq21vref8m/config-policy-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-10-22T00:55:29Z
  Ended:       2026-10-22T01:01:29Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0588800-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.248:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0588800/providers/microsoft.monitor/accounts/services-j0588800", namespace="klusterlet-2tjgit1lgtucsgcuhe4ui1l694121mei", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-msgjh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2tjgit1lgtucsgcuhe4ui1l694121mei/config-policy-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-10-22T00:55:29Z
  Ended:       2026-10-22T01:02:29Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0588800-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="governance-policy-framework", endpoint="http", environment="ci01", instance="10.128.64.248:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0588800/providers/microsoft.monitor/accounts/services-j0588800", namespace="klusterlet-2tjgit1lgtucsgcuhe4ui1l694121mei", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-msgjh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2tjgit1lgtucsgcuhe4ui1l694121mei/governance-policy-framework has not matched the expected number of replicas for longer than 30 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-10-22T00:56:27Z
  Ended:       2026-10-22T01:02:30Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0588800-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="governance-policy-framework", endpoint="http", environment="ci01", instance="10.128.64.248:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0588800/providers/microsoft.monitor/accounts/services-j0588800", namespace="klusterlet-2tjgit1lgtucsgcuhe4ui1l694121mei", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-msgjh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2tjgit1lgtucsgcuhe4ui1l694121mei/governance-policy-framework has not matched the expected number of replicas for longer than 30 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-10-22T00:56:27Z
  Ended:       2026-10-22T01:02:29Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0588800-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.248:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0588800/providers/microsoft.monitor/accounts/services-j0588800", namespace="klusterlet-2tjgit1lgtucsgcuhe4ui1l694121mei", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-msgjh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2tjgit1lgtucsgcuhe4ui1l694121mei/config-policy-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-10-22T00:57:32Z
  Ended:       2026-10-22T01:07:33Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0588800-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.248:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0588800/providers/microsoft.monitor/accounts/services-j0588800", namespace="klusterlet-2tjgjtr217jj3vq9vq4ij0qgjg910j8v", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-msgjh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2tjgjtr217jj3vq9vq4ij0qgjg910j8v/config-policy-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-10-22T00:57:32Z
  Ended:       2026-10-22T01:07:33Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0588800-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.248:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0588800/providers/microsoft.monitor/accounts/services-j0588800", namespace="klusterlet-2tjgjs0baehlrda8ov4icjda3mdfuseh", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-msgjh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2tjgjs0baehlrda8ov4icjda3mdfuseh/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-10-22T00:57:32Z
  Ended:       2026-10-22T01:07:33Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0588800-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="governance-policy-framework", endpoint="http", environment="ci01", instance="10.128.64.248:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0588800/providers/microsoft.monitor/accounts/services-j0588800", namespace="klusterlet-2tjgjtr217jj3vq9vq4ij0qgjg910j8v", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-msgjh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2tjgjtr217jj3vq9vq4ij0qgjg910j8v/governance-policy-framework has not matched the expected number of replicas for longer than 30 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-10-22T00:57:32Z
  Ended:       2026-10-22T01:08:28Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0588800-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="governance-policy-framework", endpoint="http", environment="ci01", instance="10.128.64.248:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0588800/providers/microsoft.monitor/accounts/services-j0588800", namespace="klusterlet-2tjgjs0baehlrda8ov4icjda3mdfuseh", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-msgjh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2tjgjs0baehlrda8ov4icjda3mdfuseh/governance-policy-framework has not matched the expected number of replicas for longer than 30 minutes.
Firing 17:
  State:       Resolved
  Started:     2026-10-22T00:57:32Z
  Ended:       2026-10-22T01:07:33Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0588800-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.248:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0588800/providers/microsoft.monitor/accounts/services-j0588800", namespace="klusterlet-2tjgjs0baehlrda8ov4icjda3mdfuseh", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-msgjh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2tjgjs0baehlrda8ov4icjda3mdfuseh/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 18:
  State:       Resolved
  Started:     2026-10-22T00:57:32Z
  Ended:       2026-10-22T01:07:33Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0588800-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="governance-policy-framework", endpoint="http", environment="ci01", instance="10.128.64.248:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0588800/providers/microsoft.monitor/accounts/services-j0588800", namespace="klusterlet-2tjgjtr217jj3vq9vq4ij0qgjg910j8v", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-msgjh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2tjgjtr217jj3vq9vq4ij0qgjg910j8v/governance-policy-framework has not matched the expected number of replicas for longer than 30 minutes.
Firing 19:
  State:       Resolved
  Started:     2026-10-22T00:57:32Z
  Ended:       2026-10-22T01:06:27Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0588800-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.248:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0588800/providers/microsoft.monitor/accounts/services-j0588800", namespace="klusterlet-2tjgjtr217jj3vq9vq4ij0qgjg910j8v", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-msgjh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2tjgjtr217jj3vq9vq4ij0qgjg910j8v/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 20:
  State:       Resolved
  Started:     2026-10-22T00:57:32Z
  Ended:       2026-10-22T01:08:28Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0588800-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="governance-policy-framework", endpoint="http", environment="ci01", instance="10.128.64.248:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0588800/providers/microsoft.monitor/accounts/services-j0588800", namespace="klusterlet-2tjgjs0baehlrda8ov4icjda3mdfuseh", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-msgjh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2tjgjs0baehlrda8ov4icjda3mdfuseh/governance-policy-framework has not matched the expected number of replicas for longer than 30 minutes.
Firing 21:
  State:       Resolved
  Started:     2026-10-22T00:57:32Z
  Ended:       2026-10-22T01:05:32Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0588800-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.248:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0588800/providers/microsoft.monitor/accounts/services-j0588800", namespace="klusterlet-2tjgjtr217jj3vq9vq4ij0qgjg910j8v", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-msgjh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2tjgjtr217jj3vq9vq4ij0qgjg910j8v/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 22:
  State:       Resolved
  Started:     2026-10-22T00:57:32Z
  Ended:       2026-10-22T01:08:29Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0588800-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.248:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0588800/providers/microsoft.monitor/accounts/services-j0588800", namespace="klusterlet-2tjgjs0baehlrda8ov4icjda3mdfuseh", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-msgjh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2tjgjs0baehlrda8ov4icjda3mdfuseh/config-policy-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 23:
  State:       Resolved
  Started:     2026-10-22T00:57:32Z
  Ended:       2026-10-22T01:07:33Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0588800-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.248:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0588800/providers/microsoft.monitor/accounts/services-j0588800", namespace="klusterlet-2tjgjtr217jj3vq9vq4ij0qgjg910j8v", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-msgjh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2tjgjtr217jj3vq9vq4ij0qgjg910j8v/config-policy-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 24:
  State:       Resolved
  Started:     2026-10-22T00:57:33Z
  Ended:       2026-10-22T01:08:29Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0588800-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.248:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0588800/providers/microsoft.monitor/accounts/services-j0588800", namespace="klusterlet-2tjgjs0baehlrda8ov4icjda3mdfuseh", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-msgjh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2tjgjs0baehlrda8ov4icjda3mdfuseh/config-policy-controller has not matched the expected number of replicas for longer than 30 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert KubeDeploymentReplicasMismatch does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-21 22:59	PR #7662	2113042472400588800

ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<u...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '3' s... message truncated; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:177]: failed to create node pool "np-5jgckd" for cluster "cluster-candidate-4-22-5jgckd"
Unexpected error:
    <*fmt.wrapError | 0x32059462a000>: 
    failed to create NodePool np-5jgckd: failed waiting for nodepool="np-5jgckd" for cluster "cluster-candidate-4-22-5jgckd" in resourcegroup="rg-candidate-4-22-rnz97cmn598x" to finish creating: GET https://rp.j1024128.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8d50fab0-864e-4bf4-9092-8e2ed8d00995
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8d50fab0-864e-4bf4-9092-8e2ed8d00995",
      "name": "8d50fab0-864e-4bf4-9092-8e2ed8d00995",
      "status": "Failed",
      "startTime": "2026-10-21T19:20:40.084231689Z",
      "endTime": "2026-10-21T19:39:46.06586035Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u3w2j2u3b3n4m0t-np-5jgckd-4ppsc-48r8p: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-rnz97cmn598x-candidate-5jgckd--managed/providers/Microsoft.Compute/virtualMachines/u3w2j2u3b3n4m0t-np-5jgckd-4ppsc-48r8p to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine u3w2j2u3b3n4m0t-np-5jgckd-4ppsc-48r8p: Failed: interfaces failed to create or update. err: failed to get existing resource rg-candidate-4-22-rnz97cmn598x-candidate-5jgckd--managed/u3w2j2u3b3n4m0t-np-5jgckd-4ppsc-48r8p-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-rnz97cmn598x-candidate-5jgckd--managed/providers/Microsoft.Network/networkInterfaces/u3w2j2u3b3n4m0t-np-5jgckd-4ppsc-48r8p-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'e92b9f76-b040-4cf6-a4dd-5c8bdc759e69' exceeded. Please try again after '3' s... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1024128.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8d50fab0-864e-4bf4-9092-8e2ed8d00995
            correlation ID: a2a0560d-7e26-4028-86b5-5b49e70771b9
            client request ID: c2e68888-288a-4197-9beb-bcebbb7b170a
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:177]: failed to create node pool "np-5jgckd" for cluster "cluster-candidate-4-22-5jgckd"
Unexpected error:
    <*fmt.wrapError | 0x32059462a000>: 
    failed to create NodePool np-5jgckd: failed waiting for nodepool="np-5jgckd" for cluster "cluster-candidate-4-22-5jgckd" in resourcegroup="rg-candidate-4-22-rnz97cmn598x" to finish creating: GET https://rp.j1024128.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8d50fab0-864e-4bf4-9092-8e2ed8d00995
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8d50fab0-864e-4bf4-9092-8e2ed8d00995",
      "name": "8d50fab0-864e-4bf4-9092-8e2ed8d00995",
      "status": "Failed",
      "startTime": "2026-10-21T19:20:40.084231689Z",
      "endTime": "2026-10-21T19:39:46.06586035Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u3w2j2u3b3n4m0t-np-5jgckd-4ppsc-48r8p: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-rnz97cmn598x-candidate-5jgckd--managed/providers/Microsoft.Compute/virtualMachines/u3w2j2u3b3n4m0t-np-5jgckd-4ppsc-48r8p to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine u3w2j2u3b3n4m0t-np-5jgckd-4ppsc-48r8p: Failed: interfaces failed to create or update. err: failed to get existing resource rg-candidate-4-22-rnz97cmn598x-candidate-5jgckd--managed/u3w2j2u3b3n4m0t-np-5jgckd-4ppsc-48r8p-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-rnz97cmn598x-candidate-5jgckd--managed/providers/Microsoft.Network/networkInterfaces/u3w2j2u3b3n4m0t-np-5jgckd-4ppsc-48r8p-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'e92b9f76-b040-4cf6-a4dd-5c8bdc759e69' exceeded. Please try again after '3' s... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1024128.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8d50fab0-864e-4bf4-9092-8e2ed8d00995
            correlation ID: a2a0560d-7e26-4028-86b5-5b49e70771b9
            client request ID: c2e68888-288a-4197-9beb-bcebbb7b170a
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-21 18:21	PR #7641	2112972661071024128

ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<u...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.... message truncated; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
    <*fmt.wrapError | 0x139ab7392a40>: 
    failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-2xb7hfs7wzsp" to finish creating: GET https://rp.j3007872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/89019ec8-1120-4c25-a583-ac3d503cf668
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/89019ec8-1120-4c25-a583-ac3d503cf668",
      "name": "89019ec8-1120-4c25-a583-ac3d503cf668",
      "status": "Failed",
      "startTime": "2026-10-21T23:26:25.049762723Z",
      "endTime": "2026-10-21T23:45:34.818809633Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s0b8e5p5e1a2t1q-np-1-bqzxs-c4d2f: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-2xb7hfs7wzsp--managed/providers/Microsoft.Compute/virtualMachines/s0b8e5p5e1a2t1q-np-1-bqzxs-c4d2f to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine s0b8e5p5e1a2t1q-np-1-bqzxs-c4d2f: Failed: interfaces failed to create or update. err: failed to get existing resource rg-np-labels-taints-2xb7hfs7wzsp--managed/s0b8e5p5e1a2t1q-np-1-bqzxs-c4d2f-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-2xb7hfs7wzsp--managed/providers/Microsoft.Network/networkInterfaces/s0b8e5p5e1a2t1q-np-1-bqzxs-c4d2f-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '39eefc4f-5e24-491d-9f09-bcc0ad573bcf' exceeded. Please try again after '2' seconds after additional tokens are available.... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3007872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/89019ec8-1120-4c25-a583-ac3d503cf668
            correlation ID: 2530e3f0-1ee5-450b-9d0e-3e90d3290756
            client request ID: bcb2fe64-318b-44b4-82d6-9001a598a161
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
    <*fmt.wrapError | 0x139ab7392a40>: 
    failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-2xb7hfs7wzsp" to finish creating: GET https://rp.j3007872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/89019ec8-1120-4c25-a583-ac3d503cf668
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/89019ec8-1120-4c25-a583-ac3d503cf668",
      "name": "89019ec8-1120-4c25-a583-ac3d503cf668",
      "status": "Failed",
      "startTime": "2026-10-21T23:26:25.049762723Z",
      "endTime": "2026-10-21T23:45:34.818809633Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s0b8e5p5e1a2t1q-np-1-bqzxs-c4d2f: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-2xb7hfs7wzsp--managed/providers/Microsoft.Compute/virtualMachines/s0b8e5p5e1a2t1q-np-1-bqzxs-c4d2f to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine s0b8e5p5e1a2t1q-np-1-bqzxs-c4d2f: Failed: interfaces failed to create or update. err: failed to get existing resource rg-np-labels-taints-2xb7hfs7wzsp--managed/s0b8e5p5e1a2t1q-np-1-bqzxs-c4d2f-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-2xb7hfs7wzsp--managed/providers/Microsoft.Network/networkInterfaces/s0b8e5p5e1a2t1q-np-1-bqzxs-c4d2f-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '39eefc4f-5e24-491d-9f09-bcc0ad573bcf' exceeded. Please try again after '2' seconds after additional tokens are available.... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3007872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/89019ec8-1120-4c25-a583-ac3d503cf668
            correlation ID: 2530e3f0-1ee5-450b-9d0e-3e90d3290756
            client request ID: bcb2fe64-318b-44b4-82d6-9001a598a161
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update node pool labels and taints	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-21 22:24	PR #7675	2113033770973007872

ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<u...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0x3ca35c792040>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-cc4cdqjlzb2g" to finish creating: GET https://rp.j5494784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5d31c4ce-e10b-4cce-835c-391a0bb4ac85
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5d31c4ce-e10b-4cce-835c-391a0bb4ac85",
      "name": "5d31c4ce-e10b-4cce-835c-391a0bb4ac85",
      "status": "Failed",
      "startTime": "2026-10-21T19:13:43.567943485Z",
      "endTime": "2026-10-21T19:32:47.211157666Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine x1z4m5p8d9d3i3v-np-1-cbfq7-5dn9v: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-cc4cdqjlzb2g--managed/providers/Microsoft.Compute/virtualMachines/x1z4m5p8d9d3i3v-np-1-cbfq7-5dn9v to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine x1z4m5p8d9d3i3v-np-1-cbfq7-5dn9v: Failed: interfaces failed to create or update. err: failed to get existing resource fips-enabled-cc4cdqjlzb2g--managed/x1z4m5p8d9d3i3v-np-1-cbfq7-5dn9v-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-cc4cdqjlzb2g--managed/providers/Microsoft.Network/networkInterfaces/x1z4m5p8d9d3i3v-np-1-cbfq7-5dn9v-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'b26c2343-0863-4a55-bb7e-dfb2544999c1' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to http... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5494784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5d31c4ce-e10b-4cce-835c-391a0bb4ac85
            correlation ID: 6e04a42b-11b2-4a9c-addd-af84a9c30cb9
            client request ID: dbfea4e9-55ac-41dd-b8ed-ecc52650efa5
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0x3ca35c792040>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-cc4cdqjlzb2g" to finish creating: GET https://rp.j5494784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5d31c4ce-e10b-4cce-835c-391a0bb4ac85
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5d31c4ce-e10b-4cce-835c-391a0bb4ac85",
      "name": "5d31c4ce-e10b-4cce-835c-391a0bb4ac85",
      "status": "Failed",
      "startTime": "2026-10-21T19:13:43.567943485Z",
      "endTime": "2026-10-21T19:32:47.211157666Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine x1z4m5p8d9d3i3v-np-1-cbfq7-5dn9v: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-cc4cdqjlzb2g--managed/providers/Microsoft.Compute/virtualMachines/x1z4m5p8d9d3i3v-np-1-cbfq7-5dn9v to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine x1z4m5p8d9d3i3v-np-1-cbfq7-5dn9v: Failed: interfaces failed to create or update. err: failed to get existing resource fips-enabled-cc4cdqjlzb2g--managed/x1z4m5p8d9d3i3v-np-1-cbfq7-5dn9v-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-cc4cdqjlzb2g--managed/providers/Microsoft.Network/networkInterfaces/x1z4m5p8d9d3i3v-np-1-cbfq7-5dn9v-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'b26c2343-0863-4a55-bb7e-dfb2544999c1' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to http... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5494784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5d31c4ce-e10b-4cce-835c-391a0bb4ac85
            correlation ID: 6e04a42b-11b2-4a9c-addd-af84a9c30cb9
            client request ID: dbfea4e9-55ac-41dd-b8ed-ecc52650efa5
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-21 17:52	PR #7667	2112965263845494784

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/des-encrypt-des-np-t5fkw-frfcv to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/des-encrypt-des-np-t5fkw-frfcv to exist; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:143]: failed to create nodepool des-np with DES
Unexpected error:
    <*fmt.wrapError | 0x7664fd40020>: 
    failed to create NodePool des-np: failed waiting for nodepool="des-np" for cluster "des-encrypt" in resourcegroup="des-encrypt-l6t4l44h7mwd" to finish creating: GET https://rp.j9308160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/74947d21-0259-46f2-b622-dc34ae7ef665
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/74947d21-0259-46f2-b622-dc34ae7ef665",
      "name": "74947d21-0259-46f2-b622-dc34ae7ef665",
      "status": "Failed",
      "startTime": "2026-10-21T05:20:31.296701641Z",
      "endTime": "2026-10-21T05:39:32.421681481Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine des-encrypt-des-np-t5fkw-frfcv: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/des-encrypt-l6t4l44h7mwd--managed/providers/Microsoft.Compute/virtualMachines/des-encrypt-des-np-t5fkw-frfcv to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine des-encrypt-des-np-t5fkw-frfcv: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/des-encrypt-l6t4l44h7mwd--managed/providers/Microsoft.Compute/virtualMachines/des-encrypt-des-np-t5fkw-frfcv to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9308160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/74947d21-0259-46f2-b622-dc34ae7ef665
            correlation ID: 2689c7c9-dcff-46e2-8f5f-9f51c04d066b
            client request ID: 1c9831ba-b564-46d5-9462-612964a4e4e1
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:143]: failed to create nodepool des-np with DES
Unexpected error:
    <*fmt.wrapError | 0x7664fd40020>: 
    failed to create NodePool des-np: failed waiting for nodepool="des-np" for cluster "des-encrypt" in resourcegroup="des-encrypt-l6t4l44h7mwd" to finish creating: GET https://rp.j9308160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/74947d21-0259-46f2-b622-dc34ae7ef665
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/74947d21-0259-46f2-b622-dc34ae7ef665",
      "name": "74947d21-0259-46f2-b622-dc34ae7ef665",
      "status": "Failed",
      "startTime": "2026-10-21T05:20:31.296701641Z",
      "endTime": "2026-10-21T05:39:32.421681481Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine des-encrypt-des-np-t5fkw-frfcv: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/des-encrypt-l6t4l44h7mwd--managed/providers/Microsoft.Compute/virtualMachines/des-encrypt-des-np-t5fkw-frfcv to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine des-encrypt-des-np-t5fkw-frfcv: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/des-encrypt-l6t4l44h7mwd--managed/providers/Microsoft.Compute/virtualMachines/des-encrypt-des-np-t5fkw-frfcv to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9308160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/74947d21-0259-46f2-b622-dc34ae7ef665
            correlation ID: 2689c7c9-dcff-46e2-8f5f-9f51c04d066b
            client request ID: 1c9831ba-b564-46d5-9462-612964a4e4e1
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Nodepool OS Disk Encryption should create a nodepool with customer-managed disk encryption via DES	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-21 04:03	PR #7629	2112756690729308160

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...

full failure pattern:

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [resourceTeardown] tearing down namespaces; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
    <*errors.joinError | 0xfdd760f41e0>: 
    resource group "rg-np-skip-minor-fgcrg6-74ql4tphgbdm": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-skip-minor-fgcrg6" in resourcegroup="rg-np-skip-minor-fgcrg6-74ql4tphgbdm" to finish deleting: GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/61e56fe3-a33c-4f0b-9948-8d6043c67076
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/61e56fe3-a33c-4f0b-9948-8d6043c67076",
      "name": "61e56fe3-a33c-4f0b-9948-8d6043c67076",
      "status": "Failed",
      "startTime": "2026-10-21T15:16:34.475724714Z",
      "endTime": "2026-10-21T15:40:41.195179605Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tj8245n3hvirree84hvsumfles8p8ks still exists (deletion dispatched at 2026-10-21T15:16:34Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/61e56fe3-a33c-4f0b-9948-8d6043c67076
            correlation ID: cd153f0f-7a7c-424d-b3ee-0368d0322ea3
            client request ID: 907b5bd2-c9ac-4c9c-8240-9b1e7c8dc931
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
    <*errors.joinError | 0xfdd760f41e0>: 
    resource group "rg-np-skip-minor-fgcrg6-74ql4tphgbdm": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-skip-minor-fgcrg6" in resourcegroup="rg-np-skip-minor-fgcrg6-74ql4tphgbdm" to finish deleting: GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/61e56fe3-a33c-4f0b-9948-8d6043c67076
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/61e56fe3-a33c-4f0b-9948-8d6043c67076",
      "name": "61e56fe3-a33c-4f0b-9948-8d6043c67076",
      "status": "Failed",
      "startTime": "2026-10-21T15:16:34.475724714Z",
      "endTime": "2026-10-21T15:40:41.195179605Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tj8245n3hvirree84hvsumfles8p8ks still exists (deletion dispatched at 2026-10-21T15:16:34Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/61e56fe3-a33c-4f0b-9948-8d6043c67076
            correlation ID: cd153f0f-7a7c-424d-b3ee-0368d0322ea3
            client request ID: 907b5bd2-c9ac-4c9c-8240-9b1e7c8dc931
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-21 12:58	PR #7635	2112891349849083904

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [descendantRes...

full failure pattern:

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
    <*errors.joinError | 0x1e295ecea018>: 
    resource group "rg-np-noedge-qc2jn7-9vdqhpgwfbsz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-noedge-qc2jn7" in resourcegroup="rg-np-noedge-qc2jn7-9vdqhpgwfbsz" to finish deleting: GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/baf0b153-b7d4-4908-8682-0862f2ea6e46
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/baf0b153-b7d4-4908-8682-0862f2ea6e46",
      "name": "baf0b153-b7d4-4908-8682-0862f2ea6e46",
      "status": "Failed",
      "startTime": "2026-10-21T14:55:07.97017731Z",
      "endTime": "2026-10-21T15:19:11.127007148Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tj822velhkipn0rdu6sccipp887shou still exists (deletion dispatched at 2026-10-21T14:55:08Z); [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/baf0b153-b7d4-4908-8682-0862f2ea6e46
            correlation ID: 397cd54c-667d-4f35-89d7-acd336d7e091
            client request ID: 1fdb4833-29c6-49dc-b5c0-cbeda17a4258
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
    <*errors.joinError | 0x1e295ecea018>: 
    resource group "rg-np-noedge-qc2jn7-9vdqhpgwfbsz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-noedge-qc2jn7" in resourcegroup="rg-np-noedge-qc2jn7-9vdqhpgwfbsz" to finish deleting: GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/baf0b153-b7d4-4908-8682-0862f2ea6e46
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/baf0b153-b7d4-4908-8682-0862f2ea6e46",
      "name": "baf0b153-b7d4-4908-8682-0862f2ea6e46",
      "status": "Failed",
      "startTime": "2026-10-21T14:55:07.97017731Z",
      "endTime": "2026-10-21T15:19:11.127007148Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tj822velhkipn0rdu6sccipp887shou still exists (deletion dispatched at 2026-10-21T14:55:08Z); [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/baf0b153-b7d4-4908-8682-0862f2ea6e46
            correlation ID: 397cd54c-667d-4f35-89d7-acd336d7e091
            client request ID: 1fdb4833-29c6-49dc-b5c0-cbeda17a4258
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-21 12:58	PR #7635	2112891349849083904

VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.001s: failed to list pods in namespace pullsecret-image-test: context deadline exceeded

full failure pattern:

VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.001s: failed to list pods in namespace pullsecret-image-test: context deadline exceeded

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:297]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0x268bb264c8a0>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.001s: failed to list pods in namespace pullsecret-image-test: context deadline exceeded
    {
        msg: "VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.001s: failed to list pods in namespace pullsecret-image-test: context deadline exceeded",
        err: <*fmt.wrapError | 0x268bb264c840>{
            msg: "failed to list pods in namespace pullsecret-image-test: context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:297]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0x268bb264c8a0>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.001s: failed to list pods in namespace pullsecret-image-test: context deadline exceeded
    {
        msg: "VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.001s: failed to list pods in namespace pullsecret-image-test: context deadline exceeded",
        err: <*fmt.wrapError | 0x268bb264c840>{
            msg: "failed to list pods in namespace pullsecret-image-test: context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-21 22:23	PR #7592	2113033399156346880

VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.003s: expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: image pull error...

full failure pattern:

VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.003s: expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: image pull errors:

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:297]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0x25ba9dd67100>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.003s: expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: image pull errors:
      - pod pull-secret-test container pull-test: ImagePullBackOff - Back-off pulling image "registry.redhat.io/ubi9/ubi-minimal:latest": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://registry.redhat.io/ubi9/ubi-minimal:latest: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable; artifact err: get manifest: build image source: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable
    {
        msg: "VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.003s: expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: image pull errors:\n  - pod pull-secret-test container pull-test: ImagePullBackOff - Back-off pulling image \"registry.redhat.io/ubi9/ubi-minimal:latest\": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://registry.redhat.io/ubi9/ubi-minimal:latest: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable; artifact err: get manifest: build image source: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable",
        err: <*errors.errorString | 0x25ba9e656230>{
            s: "expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: image pull errors:\n  - pod pull-secret-test container pull-test: ImagePullBackOff - Back-off pulling image \"registry.redhat.io/ubi9/ubi-minimal:latest\": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://registry.redhat.io/ubi9/ubi-minimal:latest: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable; artifact err: get manifest: build image source: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable",
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:297]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0x25ba9dd67100>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.003s: expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: image pull errors:
      - pod pull-secret-test container pull-test: ImagePullBackOff - Back-off pulling image "registry.redhat.io/ubi9/ubi-minimal:latest": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://registry.redhat.io/ubi9/ubi-minimal:latest: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable; artifact err: get manifest: build image source: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable
    {
        msg: "VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.003s: expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: image pull errors:\n  - pod pull-secret-test container pull-test: ImagePullBackOff - Back-off pulling image \"registry.redhat.io/ubi9/ubi-minimal:latest\": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://registry.redhat.io/ubi9/ubi-minimal:latest: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable; artifact err: get manifest: build image source: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable",
        err: <*errors.errorString | 0x25ba9e656230>{
            s: "expected image from registry.redhat.io/ubi-minimal in namespace pullsecret-image-test: image pull errors:\n  - pod pull-secret-test container pull-test: ImagePullBackOff - Back-off pulling image \"registry.redhat.io/ubi9/ubi-minimal:latest\": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://registry.redhat.io/ubi9/ubi-minimal:latest: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable; artifact err: get manifest: build image source: Requesting bearer token: received unexpected HTTP status: 503 Service Unavailable",
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-21 22:56	PR #7573	2113041676711759872

VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0s: failed to list pods in namespace pullsecret-image-test: context deadline exceeded

full failure pattern:

VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0s: failed to list pods in namespace pullsecret-image-test: context deadline exceeded

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:297]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0x1af71017f080>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0s: failed to list pods in namespace pullsecret-image-test: context deadline exceeded
    {
        msg: "VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0s: failed to list pods in namespace pullsecret-image-test: context deadline exceeded",
        err: <*fmt.wrapError | 0x1af71017f060>{
            msg: "failed to list pods in namespace pullsecret-image-test: context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:297]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0x1af71017f080>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0s: failed to list pods in namespace pullsecret-image-test: context deadline exceeded
    {
        msg: "VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0s: failed to list pods in namespace pullsecret-image-test: context deadline exceeded",
        err: <*fmt.wrapError | 0x1af71017f060>{
            msg: "failed to list pods in namespace pullsecret-image-test: context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-21 22:09	PR #7641	2113029875274092544

alert [svc] KubeDeploymentRolloutStuck fired

full failure pattern:

alert [svc] KubeDeploymentRolloutStuck fired

Signal:

alert

none

Full failure examples (1)

alert fired 6 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-22T01:03:32Z
  Ended:       2026-10-22T01:09:28Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j0588800-mgmt-1", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.248:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0588800/providers/microsoft.monitor/accounts/services-j0588800", namespace="klusterlet-2tjgi3mscvh8c352mcr946oq21vref8m", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-msgjh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment klusterlet-2tjgi3mscvh8c352mcr946oq21vref8m/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-22T01:03:32Z
  Ended:       2026-10-22T01:10:28Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j0588800-mgmt-1", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.248:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0588800/providers/microsoft.monitor/accounts/services-j0588800", namespace="klusterlet-2tjgi3mscvh8c352mcr946oq21vref8m", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-msgjh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment klusterlet-2tjgi3mscvh8c352mcr946oq21vref8m/config-policy-controller is not progressing for longer than 30 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-22T01:03:33Z
  Ended:       2026-10-22T01:10:28Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j0588800-mgmt-1", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.248:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0588800/providers/microsoft.monitor/accounts/services-j0588800", namespace="klusterlet-2tjgi3mscvh8c352mcr946oq21vref8m", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-msgjh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment klusterlet-2tjgi3mscvh8c352mcr946oq21vref8m/config-policy-controller is not progressing for longer than 30 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-10-22T01:03:33Z
  Ended:       2026-10-22T01:10:28Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j0588800-mgmt-1", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="governance-policy-framework", endpoint="http", environment="ci01", instance="10.128.64.248:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0588800/providers/microsoft.monitor/accounts/services-j0588800", namespace="klusterlet-2tjgi3mscvh8c352mcr946oq21vref8m", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-msgjh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment klusterlet-2tjgi3mscvh8c352mcr946oq21vref8m/governance-policy-framework is not progressing for longer than 30 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-10-22T01:03:33Z
  Ended:       2026-10-22T01:09:28Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j0588800-mgmt-1", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.248:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0588800/providers/microsoft.monitor/accounts/services-j0588800", namespace="klusterlet-2tjgi3mscvh8c352mcr946oq21vref8m", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-msgjh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment klusterlet-2tjgi3mscvh8c352mcr946oq21vref8m/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-10-22T01:03:33Z
  Ended:       2026-10-22T01:10:28Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j0588800-mgmt-1", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="governance-policy-framework", endpoint="http", environment="ci01", instance="10.128.64.248:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0588800/providers/microsoft.monitor/accounts/services-j0588800", namespace="klusterlet-2tjgi3mscvh8c352mcr946oq21vref8m", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-msgjh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment klusterlet-2tjgi3mscvh8c352mcr946oq21vref8m/governance-policy-framework is not progressing for longer than 30 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert KubeDeploymentRolloutStuck does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-21 22:59	PR #7662	2113042472400588800

failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>

full failure pattern:

failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:119]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0x31d9eb504ae0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "default-ingress-cert" in resourcegroup="ingress-cert-le-z6nkl9f5sbc4" to finish creating: GET https://rp.j5471360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/17f3d6d4-06d1-4780-9e78-cf9749f6ba8b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/17f3d6d4-06d1-4780-9e78-cf9749f6ba8b",
      "name": "17f3d6d4-06d1-4780-9e78-cf9749f6ba8b",
      "status": "Failed",
      "startTime": "2026-10-21T18:00:06.904278761Z",
      "endTime": "2026-10-21T18:19:13.24922172Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v2z3o8a0b5o6d5o-np-1-tgcm2-k8hzl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-z6nkl9f5sbc4--managed/providers/Microsoft.Compute/virtualMachines/v2z3o8a0b5o6d5o-np-1-tgcm2-k8hzl to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v2z3o8a0b5o6d5o-np-1-tgcm2-k8hzl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-z6nkl9f5sbc4--managed/providers/Microsoft.Compute/virtualMachines/v2z3o8a0b5o6d5o-np-1-tgcm2-k8hzl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5471360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/17f3d6d4-06d1-4780-9e78-cf9749f6ba8b
            correlation ID: cc57308c-c983-4f14-a613-2d5dd8c79cbb
            client request ID: 38a10c4b-f563-4456-898f-02a08bc824fc
fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:119]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0x31d9eb504ae0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "default-ingress-cert" in resourcegroup="ingress-cert-le-z6nkl9f5sbc4" to finish creating: GET https://rp.j5471360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/17f3d6d4-06d1-4780-9e78-cf9749f6ba8b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/17f3d6d4-06d1-4780-9e78-cf9749f6ba8b",
      "name": "17f3d6d4-06d1-4780-9e78-cf9749f6ba8b",
      "status": "Failed",
      "startTime": "2026-10-21T18:00:06.904278761Z",
      "endTime": "2026-10-21T18:19:13.24922172Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v2z3o8a0b5o6d5o-np-1-tgcm2-k8hzl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-z6nkl9f5sbc4--managed/providers/Microsoft.Compute/virtualMachines/v2z3o8a0b5o6d5o-np-1-tgcm2-k8hzl to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v2z3o8a0b5o6d5o-np-1-tgcm2-k8hzl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-z6nkl9f5sbc4--managed/providers/Microsoft.Compute/virtualMachines/v2z3o8a0b5o6d5o-np-1-tgcm2-k8hzl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5471360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/17f3d6d4-06d1-4780-9e78-cf9749f6ba8b
            correlation ID: cc57308c-c983-4f14-a613-2d5dd8c79cbb
            client request ID: 38a10c4b-f563-4456-898f-02a08bc824fc

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should serve the default ingress certificate for 5.1	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-21 16:35	PR #7592	2112945930595471360

failed to roll out Helm release: execution error at (swift-recorder/templates/daemonset.yaml:37:79): swift-recorder requires an image digest when enabled

full failure pattern:

failed to roll out Helm release: execution error at (swift-recorder/templates/daemonset.yaml:37:79): swift-recorder requires an image digest when enabled

Signal:

provision

none

Full failure examples (1)

time=2026-10-21T22:26:12.219Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SwiftRecorder resourceGroup=management step=deploy stamp=2 description="Step deploy\n Kind: Helm\n"
time=2026-10-21T22:26:12.557Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.SwiftRecorder resourceGroup=management step=deploy stamp=2 values="map[captureMode:all clusterName:ci01-j3775744-mgmt-2 enabled:true environment:ci01 image:map[digest: registry:arohcpsvcdev.azurecr.io repository:swift-recorder] region:canadacentral resources:map[limits:map[memory:256Mi] requests:map[cpu:20m memory:64Mi]] serviceAccount:map[name:swift-recorder]]"
time=2026-10-21T22:26:12.557Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.SwiftRecorder resourceGroup=management step=deploy stamp=2
time=2026-10-21T22:26:12.728Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.SwiftRecorder resourceGroup=management step=deploy stamp=2 namespace=swift-recorder
time=2026-10-21T22:26:12.944Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SwiftRecorder resourceGroup=management step=deploy stamp=2 dryRun=false
time=2026-10-21T22:26:12.944Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.SwiftRecorder resourceGroup=management step=deploy stamp=2
time=2026-10-21T22:26:12.999Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.SwiftRecorder resourceGroup=management step=deploy stamp=2
time=2026-10-21T22:26:13.152Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SwiftRecorder resourceGroup=management step=deploy stamp=2 err="execution error at (swift-recorder/templates/daemonset.yaml:37:79): swift-recorder requires an image digest when enabled"
time=2026-10-21T22:26:13.152Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SwiftRecorder resourceGroup=management step=deploy stamp=2 err="stamp 2: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: execution error at (swift-recorder/templates/daemonset.yaml:37:79): swift-recorder requires an image digest when enabled"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.SwiftRecorder/management/deploy (stamp=2)	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-21 21:11	PR #7672	2113015450223775744

DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline...

full failure pattern:

DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline exceeded

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:121]: failed to verify simple web app runs on v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0x1ea8925d4400>: 
    DNS for route host agnhost-e2e-sample-app-5pdz7.apps.aro.v20260901.wbc4.j8936960.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-5pdz7.apps.aro.v20260901.wbc4.j8936960.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5pdz7.apps.aro.v20260901.wbc4.j8936960.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-5pdz7.apps.aro.v20260901.wbc4.j8936960.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-5pdz7.apps.aro.v20260901.wbc4.j8936960.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5pdz7.apps.aro.v20260901.wbc4.j8936960.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x1ea891ce22e0>{
            msg: "DNS for agnhost-e2e-sample-app-5pdz7.apps.aro.v20260901.wbc4.j8936960.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5pdz7.apps.aro.v20260901.wbc4.j8936960.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:121]: failed to verify simple web app runs on v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0x1ea8925d4400>: 
    DNS for route host agnhost-e2e-sample-app-5pdz7.apps.aro.v20260901.wbc4.j8936960.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-5pdz7.apps.aro.v20260901.wbc4.j8936960.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5pdz7.apps.aro.v20260901.wbc4.j8936960.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-5pdz7.apps.aro.v20260901.wbc4.j8936960.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-5pdz7.apps.aro.v20260901.wbc4.j8936960.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5pdz7.apps.aro.v20260901.wbc4.j8936960.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x1ea891ce22e0>{
            msg: "DNS for agnhost-e2e-sample-app-5pdz7.apps.aro.v20260901.wbc4.j8936960.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5pdz7.apps.aro.v20260901.wbc4.j8936960.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster using v20260901preview API and verify cluster health	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-10-21 22:55	PR #7646	2113041621648936960