CIHealth Failure Patterns

Environment: DEV

Failure patterns: 53 · Failures matched: 265 · Total runs: 107 · Runs affected: 83

Failure pattern

Failed at

Trend

Also in

VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil

full failure pattern:

VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:151]: failed to verify cluster health or IngressController scope for cluster "pi-4-21"
Unexpected error:
    <*errors.joinError | 0xc001589cb0>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:151]: failed to verify cluster health or IngressController scope for cluster "pi-4-21"
Unexpected error:
    <*errors.joinError | 0xc001589cb0>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:151]: failed to verify cluster health or IngressController scope for cluster "pi-4-20"
Unexpected error:
    <*errors.joinError | 0xc00122eff0>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:151]: failed to verify cluster health or IngressController scope for cluster "pi-4-20"
Unexpected error:
    <*errors.joinError | 0xc00122eff0>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:151]: failed to verify cluster health or IngressController scope for cluster "pi-4-21"
Unexpected error:
    <*errors.joinError | 0xc00152e078>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:151]: failed to verify cluster health or IngressController scope for cluster "pi-4-21"
Unexpected error:
    <*errors.joinError | 0xc00152e078>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    ...
occurred

Contributing tests (2)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal for 4.20	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal for 4.21	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-29 20:58	PR #7231	2105039456577785856
2026-09-29 15:40	PR #7231	2104959408780873728

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:83]: HCP cluster rg-hs-pre--vbl2fffvpm77/cluster-hs-pre-6h5wx2 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000fa6f80>: 
    failed to create HCP cluster cluster-hs-pre-6h5wx2: failed waiting for cluster="cluster-hs-pre-6h5wx2" in resourcegroup="rg-hs-pre--vbl2fffvpm77" to finish creating: GET https://rp.j6245760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ed26df33-28d6-48f7-aad8-139388423746
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ed26df33-28d6-48f7-aad8-139388423746",
      "name": "ed26df33-28d6-48f7-aad8-139388423746",
      "status": "Failed",
      "startTime": "2026-09-29T03:00:05.091215108Z",
      "endTime": "2026-09-29T03:29:10.065852363Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:83]: HCP cluster rg-hs-pre--vbl2fffvpm77/cluster-hs-pre-6h5wx2 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000fa6f80>: 
    failed to create HCP cluster cluster-hs-pre-6h5wx2: failed waiting for cluster="cluster-hs-pre-6h5wx2" in resourcegroup="rg-hs-pre--vbl2fffvpm77" to finish creating: GET https://rp.j6245760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ed26df33-28d6-48f7-aad8-139388423746
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ed26df33-28d6-48f7-aad8-139388423746",
      "name": "ed26df33-28d6-48f7-aad8-139388423746",
      "status": "Failed",
      "startTime": "2026-09-29T03:00:05.091215108Z",
      "endTime": "2026-09-29T03:29:10.065852363Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-29 01:55	PR #7178	2104751945536245760

ERROR CODE: RoleAssignmentLimitExceeded; provider Microsoft.Authorization

full failure pattern:

ERROR CODE: RoleAssignmentLimitExceeded; provider Microsoft.Authorization

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:348]: failed to grant AcrPull to second MI on ACR
Unexpected error:
    <*exported.ResponseError | 0xc001023830>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/acr-pull-create-92f84hwfsjvp/providers/Microsoft.ContainerRegistry/registries/acrpullcreate92f84hwfsjvp/providers/Microsoft.Authorization/roleAssignments/500314c4-ec16-5346-bdf6-ef93cc9ce507
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: RoleAssignmentLimitExceeded
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "RoleAssignmentLimitExceeded",
        "message": "No more role assignments can be created."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:348]: failed to grant AcrPull to second MI on ACR
Unexpected error:
    <*exported.ResponseError | 0xc001023830>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/acr-pull-create-92f84hwfsjvp/providers/Microsoft.ContainerRegistry/registries/acrpullcreate92f84hwfsjvp/providers/Microsoft.Authorization/roleAssignments/500314c4-ec16-5346-bdf6-ef93cc9ce507
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: RoleAssignmentLimitExceeded
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "RoleAssignmentLimitExceeded",
        "message": "No more role assignments can be created."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with ACR pull via managed identity and pull from a private ACR	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-29 01:37	PR #7224	2104747427830435840

failed to roll out Helm release: resource Deployment/clusters-service/clusters-service not ready. status: Failed, message: Progress deadline exceeded

full failure pattern:

failed to roll out Helm release: resource Deployment/clusters-service/clusters-service not ready. status: Failed, message: Progress deadline exceeded

Signal:

provision

none

Full failure examples (1)

time=2026-09-29T12:22:57.834Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n"
time=2026-09-29T12:22:58.550Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy values="map[azureArmHelperIdentityCertName:armHelperPoolCert-31 azureArmHelperIdentityClientId:15d9575a-485d-4e5f-9c06-003e8353383b azureArmHelperMockFpaPrincipalId:47f69502-0065-4d9a-b19b-d403e183d2f4 azureCloudEnvironmentName:AzurePublicCloud azureCsMiClientId:79ab0919-0aa8-453d-80c5-59ac393ff1e8 azureFirstPartyApplicationClientId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX azureMiMockServicePrincipalCertName:msiMockPoolCert-6 azureMiMockServicePrincipalClientId:fe0ac67d-d1d8-45f9-a347-362201553034 azureMiMockServicePrincipalPrincipalId:f8be0ede-39df-41cf-aed7-7f3626f23a5a azureOperatorsMI:map[dev:map[cloudControllerManager:map[roleDefinitions:[map[id:a1f96423-95ce-4224-ab27-4e3dc72facd4 name:Azure Red Hat OpenShift Cloud Controller Manager]]] cloudNetworkConfig:map[roleDefinitions:[map[id:be7a6435-15ae-4171-8f30-4a343eff9e8f name:Azure Red Hat OpenShift Network Operator]]] clusterApiAzure:map[roleDefinitions:[map[id:88366f10-ed47-4cc0-9fab-c8a06148393e name:Azure Red Hat OpenShift Hosted Control Planes Cluster API Provider]]] controlPlane:map[roleDefinitions:[map[id:fc0c873f-45e9-4d0d-a7d1-585aab30c6ed name:Azure Red Hat OpenShift Hosted Control Planes Control Plane Operator]]] diskCsiDriver:map[roleDefinitions:[map[id:5b7237c5-45e1-49d6-bc18-a1f62f400748 name:Azure Red Hat OpenShift Disk Storage Operator]]] fileCsiDriver:map[roleDefinitions:[map[id:0d7aedc0-15fd-4a67-a412-efad370c947e name:Azure Red Hat OpenShift File Storage Operator]]] imageRegistry:map[roleDefinitions:[map[id:8b32b316-c2f5-4ddf-b05b-83dacd2d08b5 name:Azure Red Hat OpenShift Image Registry Operator]]] ingress:map[roleDefinitions:[map[id:0336e1d3-7a87-462b-b6db-342b63f7802c name:Azure Red Hat OpenShift Cluster Ingress Operator]]] kms:map[roleDefinitions:[map[id:12338af0-0e69-4776-bea7-57ae8d297424 name:Key Vault Crypto User]]] serviceManagedIdentity:map[roleDefinitions:[map[id:c0ff367d-66d8-445e-917c-583feb0ef0d4 name:Azure Red Hat OpenShift Hosted Control Planes Service Managed Identity]]]] public:map[cloudControllerManager:map[roleDefinitions:[map[id:a1f96423-95ce-4224-ab27-4e3dc72facd4 name:Azure Red Hat OpenShift Cloud Controller Manager]]] cloudNetworkConfig:map[roleDefinitions:[map[id:be7a6435-15ae-4171-8f30-4a343eff9e8f name:Azure Red Hat OpenShift Network Operator]]] clusterApiAzure:map[roleDefinitions:[map[id:88366f10-ed47-4cc0-9fab-c8a06148393e name:Azure Red Hat OpenShift Hosted Control Planes Cluster API Provider]]] controlPlane:map[roleDefinitions:[map[id:fc0c873f-45e9-4d0d-a7d1-585aab30c6ed name:Azure Red Hat OpenShift Hosted Control Planes Control Plane Operator]]] diskCsiDriver:map[roleDefinitions:[map[id:5b7237c5-45e1-49d6-bc18-a1f62f400748 name:Azure Red Hat OpenShift Disk Storage Operator]]] fileCsiDriver:map[roleDefinitions:[map[id:0d7aedc0-15fd-4a67-a412-efad370c947e name:Azure Red Hat OpenShift File Storage Operator]]] imageRegistry:map[roleDefinitions:[map[id:8b32b316-c2f5-4ddf-b05b-83dacd2d08b5 name:Azure Red Hat OpenShift Image Registry Operator]]] ingress:map[roleDefinitions:[map[id:0336e1d3-7a87-462b-b6db-342b63f7802c name:Azure Red Hat OpenShift Cluster Ingress Operator]]] kms:map[roleDefinitions:[map[id:12338af0-0e69-4776-bea7-57ae8d297424 name:Key Vault Crypto User]]] serviceManagedIdentity:map[roleDefinitions:[map[id:c0ff367d-66d8-445e-917c-583feb0ef0d4 name:Azure Red Hat OpenShift Hosted Control Planes Service Managed Identity]]]] roleSetName:dev] backplaneURL: batchProcesses: batchProcessesDryRun:true clientId:foo clientScopes:openid clientSecret:bar clusterErrorReport:false cosmosDatabaseName:arohcpci01-rp-j9715712 cosmosDatabaseUrl:https://arohcpci01-rp-j9715712.documents.azure.com:443/ cpuRequest:630m csDeploymentStrategy:map[rollingUpdate:map[maxSurge:1 maxUnavailable:0] type:RollingUpdate] dataPlaneHAProxyImage:arohcpocpdev.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678 databaseAuthMethod:postgres databaseDisableTls:true databaseHost:ocm-cs-db databaseK8sSecretName:ocm-cs-db databaseName:ocm-cs-db databasePassword:TheBlurstOfTimes databasePort:5432 databaseServiceName:ocm-cs-db databaseUser:ocm databaseVolumeCapacity:512Mi debugMaxDuration:5m debugPort: defaultExpiration:0 deployment:map[zoneCount:3] environment:arohcpci01 forceMigration: fpaCertName:firstPartyCert2 gatewayUrl:http://127.0.0.1:9090 global:map[imageDigest:sha256:11c94e058bf6e19d183bb655461fb5d37d7368243382944adabc1c0fdf10ea9b] imageDigest:sha256:11c94e058bf6e19d183bb655461fb5d37d7368243382944adabc1c0fdf10ea9b imageRegistry:arohcpsvcdev.azurecr.io imageRepository:mvacula/cs insecure:false jwksUrl:http://localhost logLevel:4 machinePoolMigrationWorkerPeriod:1h managedIdentitiesDataPlaneAudienceResource:https://dummy.org maximumExpiration:0 memoryLimit:unlimited memoryRequest:150Mi ocpAcrResourceId:/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev ocpAcrResourceUrl:arohcpocpdev.azurecr.io oidcIssuerBaseUrl:https://arohcpci01oidcj9715712.z1.web.core.windows.net/ oidcIssuerBlobServiceUrl:https://arohcpci01oidcj9715712.blob.core.windows.net/ postgresqlVersion:12 provisionShardClusterLimit:100 pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:mvacula/cs:pull workloadIdentityClientId:c964e23b-7c92-4e91-82c7-8ed9b8848c7e workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] region:westus3 replicas:3 runtimeMode:aro-hcp serviceAccountName:clusters-service serviceKeyVaultName:aro-hcp-dev-svc-kv tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX tlsCertificatesEnabled:true tlsCertificatesIssuer:Self tokenUrl:http://localhost useAzureDB:false userDefinedDnsBaseDomain:i1.devshift.org]"
time=2026-09-29T12:22:58.550Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy
time=2026-09-29T12:22:58.851Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy namespace=clusters-service
time=2026-09-29T12:22:59.239Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy dryRun=false
time=2026-09-29T12:22:59.239Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy
time=2026-09-29T12:22:59.359Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy
time=2026-09-29T12:33:06.293Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy err="resource Deployment/clusters-service/clusters-service not ready. status: Failed, message: Progress deadline exceeded"
time=2026-09-29T12:33:06.294Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy err="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource Deployment/clusters-service/clusters-service not ready. status: Failed, message: Progress deadline exceeded"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.ClusterService/service/deploy	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-29 11:48	PR #6141	2104901125449715712

alert [svc] BackendControllerQueueDepthHigh fired

full failure pattern:

alert [svc] BackendControllerQueueDepthHigh fired

Signal:

alert

none

Full failure examples (3)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-29T22:45:32Z
  Ended:       2026-09-29T22:56:28Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j9852672-svc", component="backend", name="clusterresources", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-29T21:30:31Z
  Ended:       2026-09-29T21:40:23Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j8987136-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-29T21:12:57Z
  Ended:       2026-09-29T21:27:58Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j3763456-svc", component="backend", name="clusterresources", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire	11

Affected runs (11)

Date (UTC)	Associated PR	Prow job
2026-09-29 21:37	PR #7001	2105049386189852672
2026-09-29 20:31	PR #7251	2105032680448987136
2026-09-29 19:19	PR #7238	2105026278703763456
2026-09-29 16:29	PR #7246	2104971730622615552
2026-09-29 15:44	PR #7245	2104960395847405568
2026-09-29 15:00	PR #7183	2104949577500069888
2026-09-29 14:11	PR #7054	2104937201925099520
2026-09-29 06:52	PR #7221	2104826606282346496
2026-09-29 06:51	PR #7224	2104826504838909952
2026-09-29 01:37	PR #6900	2104747399384666112
2026-09-29 01:37	PR #7170	2104747285383483392

alert [hcp] KubePodNotReady fired

full failure pattern:

alert [hcp] KubePodNotReady fired

Signal:

alert

none

Full failure examples (3)

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-29T23:07:08Z
  Ended:       2026-09-29T23:34:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6848896-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t4v70gg0dc77jh6i9l6rr93v81st7vh-k2r6k8a7f3k5y2e", pod="router-5c4bbd5cf4-5bqs8", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t4v70gg0dc77jh6i9l6rr93v81st7vh-k2r6k8a7f3k5y2e/router-5c4bbd5cf4-5bqs8 has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-29T23:07:08Z
  Ended:       2026-09-29T23:34:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6848896-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t4v6v9k2f4p2jhh963s28dni94k6h3d-n7f2l3l7q3v7r9c", pod="router-68d995764-szrlt", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t4v6v9k2f4p2jhh963s28dni94k6h3d-n7f2l3l7q3v7r9c/router-68d995764-szrlt has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-29T12:31:18Z
  Ended:       2026-09-29T12:40:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8217088-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t4lslm5i8q6f369o8t992bo6sevfbrh-p0z2h6h5a9y0w0j", pod="router-8577fddd95-8qm6v", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t4lslm5i8q6f369o8t992bo6sevfbrh-p0z2h6h5a9y0w0j/router-8577fddd95-8qm6v has been in a non-ready state for longer than 5 minutes.

alert fired 3 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-29T12:10:11Z
  Ended:       2026-09-29T12:36:10Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4324224-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t4ljqd98a0ltiqds61qu1uqil6ns75e-v20261001", pod="router-c85f86bb6-554fp", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t4ljqd98a0ltiqds61qu1uqil6ns75e-v20261001/router-c85f86bb6-554fp has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-29T12:11:12Z
  Ended:       2026-09-29T12:39:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4324224-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t4ljt7cibn81sb7upchqksdk1o3pk6k-l2c2c2w5b6r7a5t", pod="router-6c8dc8d9f9-k5vdh", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t4ljt7cibn81sb7upchqksdk1o3pk6k-l2c2c2w5b6r7a5t/router-6c8dc8d9f9-k5vdh has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-29T12:12:13Z
  Ended:       2026-09-29T12:43:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4324224-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t4ll36tvrrrt72ee2s4e8n92bvd8a7l-agg-cluster", pod="router-5c8466799d-l8fcc", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t4ll36tvrrrt72ee2s4e8n92bvd8a7l-agg-cluster/router-5c8466799d-l8fcc has been in a non-ready state for longer than 5 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubePodNotReady does not fire	7

Affected runs (7)

Date (UTC)	Associated PR	Prow job
2026-09-29 22:01	PR #7254	2105055507696848896
2026-09-29 11:29	PR #7169	2104896461568217088
2026-09-29 11:13	PR #7001	2104892283974324224
2026-09-29 09:49	PR #7221	2104871248742846464
2026-09-29 08:50	PR #7236	2104856376063823872
2026-09-29 06:52	PR #7068	2104826553132126208
2026-09-29 01:37	PR #7169	2104747271315787776

alert [svc] KubePodNotReady fired

full failure pattern:

alert [svc] KubePodNotReady fired

Signal:

alert

none

Full failure examples (3)

alert fired 22 time(s) (2 unknown, 20 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-29T18:06:16Z
  Ended:       2026-09-29T18:12:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3617536-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t4qlf9g4q7g15no702p9kevvbmidhl9-f6u1x0l2r8k7x5n", pod="openshift-apiserver-5864fbdbd4-st9wn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t4qlf9g4q7g15no702p9kevvbmidhl9-f6u1x0l2r8k7x5n/openshift-apiserver-5864fbdbd4-st9wn has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-29T18:12:19Z
  Ended:       2026-09-29T18:18:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3617536-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t4q8gksdbb4mj4dub9bfiir7qb06n0b-e5v1y6s5y3h4c3s", pod="azure-disk-csi-driver-controller-84b49bf957-24qh8", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t4q8gksdbb4mj4dub9bfiir7qb06n0b-e5v1y6s5y3h4c3s/azure-disk-csi-driver-controller-84b49bf957-24qh8 has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-29T16:33:27Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0873728-mgmt-2", component="kubernetes-infrastructure", namespace="open-cluster-management-agent-addon", pod="klusterlet-addon-workmgr-78648d9c8d-gd726", region="canadacentral", severity="warning"
  Description: Pod open-cluster-management-agent-addon/klusterlet-addon-workmgr-78648d9c8d-gd726 has been in a non-ready state for longer than 5 minutes.

alert fired 3 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-29T12:28:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9715712-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-54c9ff565c-wwtqq", region="westus3", severity="warning"
  Description: Pod clusters-service/clusters-service-54c9ff565c-wwtqq has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-29T12:28:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9715712-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-54c9ff565c-rqgr7", region="westus3", severity="warning"
  Description: Pod clusters-service/clusters-service-54c9ff565c-rqgr7 has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-29T12:28:43Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9715712-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-54c9ff565c-dp5lq", region="westus3", severity="warning"
  Description: Pod clusters-service/clusters-service-54c9ff565c-dp5lq has been in a non-ready state for longer than 5 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert KubePodNotReady does not fire	6

Affected runs (6)

Date (UTC)	Associated PR	Prow job
2026-09-29 16:20	PR #7178	2104969648343617536
2026-09-29 15:40	PR #7231	2104959408780873728
2026-09-29 11:48	PR #6141	2104901125449715712
2026-09-29 11:13	PR #7001	2104892283974324224
2026-09-29 09:09	PR #6915	2104861197303025664
2026-09-29 01:55	PR #7178	2104751945536245760

alert [hcp] KubeStatefulSetReplicasMismatch fired

full failure pattern:

alert [hcp] KubeStatefulSetReplicasMismatch fired

Signal:

alert

none

Full failure examples (3)

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-29T19:56:54Z
  Ended:       2026-09-29T20:20:53Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0135168-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.15:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0135168/providers/microsoft.monitor/accounts/hcps-j0135168", namespace="ocm-arohcpci01-2t4s97ialb0v6jm7ucotiecf3ule87m0-w2j1q7d0a6j3n6k", pod="arohcp-monitor-kube-state-metrics-7988dc648c-cnk2l", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t4s97ialb0v6jm7ucotiecf3ule87m0-w2j1q7d0a6j3n6k/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-29T19:56:54Z
  Ended:       2026-09-29T20:24:52Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0135168-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.15:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0135168/providers/microsoft.monitor/accounts/hcps-j0135168", namespace="ocm-arohcpci01-2t4s97ialb0v6jm7ucotiecf3ule87m0-w2j1q7d0a6j3n6k", pod="arohcp-monitor-kube-state-metrics-7988dc648c-cnk2l", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t4s97ialb0v6jm7ucotiecf3ule87m0-w2j1q7d0a6j3n6k/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-29T19:35:51Z
  Ended:       2026-09-29T20:06:48Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j3543680-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.87:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3543680/providers/microsoft.monitor/accounts/hcps-j3543680", namespace="ocm-arohcpci01-2t4s0nlb3kmtdnv66shggj311g6csrra-basic-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-c4fhh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t4s0nlb3kmtdnv66shggj311g6csrra-basic-cluster/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-29T16:54:05Z
  Ended:       2026-09-29T17:20:06Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0873728-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0873728/providers/microsoft.monitor/accounts/hcps-j0873728", namespace="ocm-arohcpci01-2t4pkau0b0nuc1epds7jnuhhosgt2got-c8u2v9v3j9q8t4s", pod="arohcp-monitor-kube-state-metrics-7988dc648c-qqn8d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t4pkau0b0nuc1epds7jnuhhosgt2got-c8u2v9v3j9q8t4s/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-29T16:54:05Z
  Ended:       2026-09-29T17:24:04Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0873728-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0873728/providers/microsoft.monitor/accounts/hcps-j0873728", namespace="ocm-arohcpci01-2t4pkau0b0nuc1epds7jnuhhosgt2got-c8u2v9v3j9q8t4s", pod="arohcp-monitor-kube-state-metrics-7988dc648c-qqn8d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t4pkau0b0nuc1epds7jnuhhosgt2got-c8u2v9v3j9q8t4s/etcd has not matched the expected number of replicas for longer than 15 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubeStatefulSetReplicasMismatch does not fire	5

Affected runs (5)

Date (UTC)	Associated PR	Prow job
2026-09-29 18:25	PR #7141	2105006438400135168
2026-09-29 18:00	PR #6915	2105000598473543680
2026-09-29 15:40	PR #7231	2104959408780873728
2026-09-29 06:52	PR #7068	2104826553132126208
2026-09-29 01:37	PR #7169	2104747271315787776

alert [hcp] KubeDeploymentReplicasMismatch fired

full failure pattern:

alert [hcp] KubeDeploymentReplicasMismatch fired

Signal:

alert

none

Full failure examples (3)

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-29T20:10:50Z
  Ended:       2026-09-29T20:26:52Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0135168-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.15:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0135168/providers/microsoft.monitor/accounts/hcps-j0135168", namespace="ocm-arohcpci01-2t4s97ialb0v6jm7ucotiecf3ule87m0-w2j1q7d0a6j3n6k", pod="arohcp-monitor-kube-state-metrics-7988dc648c-cnk2l", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t4s97ialb0v6jm7ucotiecf3ule87m0-w2j1q7d0a6j3n6k/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-29T17:08:04Z
  Ended:       2026-09-29T17:26:05Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0873728-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0873728/providers/microsoft.monitor/accounts/hcps-j0873728", namespace="ocm-arohcpci01-2t4pkau0b0nuc1epds7jnuhhosgt2got-c8u2v9v3j9q8t4s", pod="arohcp-monitor-kube-state-metrics-7988dc648c-qqn8d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t4pkau0b0nuc1epds7jnuhhosgt2got-c8u2v9v3j9q8t4s/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

alert fired 12 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-29T08:30:07Z
  Ended:       2026-09-29T08:50:06Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2126208-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.203:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2126208/providers/microsoft.monitor/accounts/hcps-j2126208", namespace="ocm-arohcpci01-2t4i23bv73ftklp9paqra2ef1uvhfhih-z2g1x5t0y3e0n2m", pod="arohcp-monitor-kube-state-metrics-7988dc648c-njht2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t4i23bv73ftklp9paqra2ef1uvhfhih-z2g1x5t0y3e0n2m/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-29T08:30:07Z
  Ended:       2026-09-29T08:50:06Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2126208-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.203:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2126208/providers/microsoft.monitor/accounts/hcps-j2126208", namespace="ocm-arohcpci01-2t4i1qgjm100jlb0kv1qg3ij1gftkogs-n1u9p0c1i8f8h1v", pod="arohcp-monitor-kube-state-metrics-7988dc648c-njht2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t4i1qgjm100jlb0kv1qg3ij1gftkogs-n1u9p0c1i8f8h1v/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-29T08:30:07Z
  Ended:       2026-09-29T08:48:05Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2126208-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.203:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2126208/providers/microsoft.monitor/accounts/hcps-j2126208", namespace="ocm-arohcpci01-2t4i1v634q59ghjdn03ih5nveqi92bne-b3q8z4x9q6s0r6o", pod="arohcp-monitor-kube-state-metrics-7988dc648c-njht2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t4i1v634q59ghjdn03ih5nveqi92bne-b3q8z4x9q6s0r6o/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-29T08:30:08Z
  Ended:       2026-09-29T08:48:05Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2126208-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.203:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2126208/providers/microsoft.monitor/accounts/hcps-j2126208", namespace="ocm-arohcpci01-2t4i2598lplvrsf0s9b0lnetgap7c3pm-acr-pull-create", pod="arohcp-monitor-kube-state-metrics-7988dc648c-njht2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t4i2598lplvrsf0s9b0lnetgap7c3pm-acr-pull-create/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-09-29T08:30:08Z
  Ended:       2026-09-29T08:50:06Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2126208-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.203:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2126208/providers/microsoft.monitor/accounts/hcps-j2126208", namespace="ocm-arohcpci01-2t4i21j3vg3ak4ep8960ujatt7mie9ip-ea-listhcr4j8", pod="arohcp-monitor-kube-state-metrics-7988dc648c-njht2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t4i21j3vg3ak4ep8960ujatt7mie9ip-ea-listhcr4j8/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-09-29T08:31:08Z
  Ended:       2026-09-29T08:48:05Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2126208-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.203:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2126208/providers/microsoft.monitor/accounts/hcps-j2126208", namespace="ocm-arohcpci01-2t4i1nohuspbu2tqvi3lqnaqjiqmepii-u0v7m3x2h5g5c3y", pod="arohcp-monitor-kube-state-metrics-7988dc648c-njht2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t4i1nohuspbu2tqvi3lqnaqjiqmepii-u0v7m3x2h5g5c3y/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-09-29T08:31:08Z
  Ended:       2026-09-29T08:50:06Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2126208-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.203:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2126208/providers/microsoft.monitor/accounts/hcps-j2126208", namespace="ocm-arohcpci01-2t4i21lp6qql74kuc0jeund2dp8k3e5o-u1w7f0g9h5z6l6n", pod="arohcp-monitor-kube-state-metrics-7988dc648c-njht2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t4i21lp6qql74kuc0jeund2dp8k3e5o-u1w7f0g9h5z6l6n/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-09-29T08:31:08Z
  Ended:       2026-09-29T08:50:06Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2126208-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.203:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2126208/providers/microsoft.monitor/accounts/hcps-j2126208", namespace="ocm-arohcpci01-2t4i2fc1poue2ub3f5n014gqkv4othuk-r1v9h5p1o9x0i5z", pod="arohcp-monitor-kube-state-metrics-7988dc648c-njht2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t4i2fc1poue2ub3f5n014gqkv4othuk-r1v9h5p1o9x0i5z/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-09-29T08:31:08Z
  Ended:       2026-09-29T08:50:06Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2126208-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.203:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2126208/providers/microsoft.monitor/accounts/hcps-j2126208", namespace="ocm-arohcpci01-2t4i2b1m61e1uvlskbah1j29ct4ov9km-g0u9n1p2u6q7n2c", pod="arohcp-monitor-kube-state-metrics-7988dc648c-njht2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t4i2b1m61e1uvlskbah1j29ct4ov9km-g0u9n1p2u6q7n2c/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-09-29T08:31:08Z
  Ended:       2026-09-29T08:50:06Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2126208-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.203:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2126208/providers/microsoft.monitor/accounts/hcps-j2126208", namespace="ocm-arohcpci01-2t4i2fclju63g3kb2c1qlbdm3tgnarp2-z5b2k1s2b2d6r6p", pod="arohcp-monitor-kube-state-metrics-7988dc648c-njht2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t4i2fclju63g3kb2c1qlbdm3tgnarp2-z5b2k1s2b2d6r6p/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-09-29T08:32:10Z
  Ended:       2026-09-29T08:50:06Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2126208-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.203:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2126208/providers/microsoft.monitor/accounts/hcps-j2126208", namespace="ocm-arohcpci01-2t4i1q19vd24pc2dma39mndkh6d055c3-u8o7e3g2q8p1m7w", pod="arohcp-monitor-kube-state-metrics-7988dc648c-njht2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t4i1q19vd24pc2dma39mndkh6d055c3-u8o7e3g2q8p1m7w/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-09-29T08:33:07Z
  Ended:       2026-09-29T08:50:06Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2126208-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.203:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2126208/providers/microsoft.monitor/accounts/hcps-j2126208", namespace="ocm-arohcpci01-2t4i2j0k2qc17trgshchrbb31b6r0e12-cilium-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-njht2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t4i2j0k2qc17trgshchrbb31b6r0e12-cilium-cluster/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire	3

Affected runs (3)

Date (UTC)	Associated PR	Prow job
2026-09-29 18:25	PR #7141	2105006438400135168
2026-09-29 15:40	PR #7231	2104959408780873728
2026-09-29 06:52	PR #7068	2104826553132126208

ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor ...

full failure pattern:

ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.Compute

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000a02cc0>: 
    failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-7gnm6qbmvpcb" to finish creating: GET https://rp.j0435840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8167fae0-e6eb-4298-bee1-fee2d504da77
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8167fae0-e6eb-4298-bee1-fee2d504da77",
      "name": "8167fae0-e6eb-4298-bee1-fee2d504da77",
      "status": "Failed",
      "startTime": "2026-09-29T02:58:54.33539143Z",
      "endTime": "2026-09-29T02:59:36.376050844Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000a02cc0>: 
    failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-7gnm6qbmvpcb" to finish creating: GET https://rp.j0435840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8167fae0-e6eb-4298-bee1-fee2d504da77
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8167fae0-e6eb-4298-bee1-fee2d504da77",
      "name": "8167fae0-e6eb-4298-bee1-fee2d504da77",
      "status": "Failed",
      "startTime": "2026-09-29T02:58:54.33539143Z",
      "endTime": "2026-09-29T02:59:36.376050844Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:222]: failed to create nodepool "np-1" for authorized CIDRs cluster
Unexpected error:
    <*fmt.wrapError | 0xc0011bf9c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-2plrgsl5fwgr" to finish creating: GET https://rp.j0435840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/476732bf-65f0-4455-afd3-17ce246aa2ca
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/476732bf-65f0-4455-afd3-17ce246aa2ca",
      "name": "476732bf-65f0-4455-afd3-17ce246aa2ca",
      "status": "Failed",
      "startTime": "2026-09-29T03:00:32.197258881Z",
      "endTime": "2026-09-29T03:01:14.992686131Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:222]: failed to create nodepool "np-1" for authorized CIDRs cluster
Unexpected error:
    <*fmt.wrapError | 0xc0011bf9c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-2plrgsl5fwgr" to finish creating: GET https://rp.j0435840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/476732bf-65f0-4455-afd3-17ce246aa2ca
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/476732bf-65f0-4455-afd3-17ce246aa2ca",
      "name": "476732bf-65f0-4455-afd3-17ce246aa2ca",
      "status": "Failed",
      "startTime": "2026-09-29T03:00:32.197258881Z",
      "endTime": "2026-09-29T03:01:14.992686131Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-21 with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc000e962c0>: 
    failed to create NodePool npupgrade-4-21: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-8rl27d" in resourcegroup="rg-np-version-upgrade-8rl27d-cds8pn7rp5pj" to finish creating: GET https://rp.j0435840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/99532f4a-c494-477c-9235-28ce3b3dccdb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/99532f4a-c494-477c-9235-28ce3b3dccdb",
      "name": "99532f4a-c494-477c-9235-28ce3b3dccdb",
      "status": "Failed",
      "startTime": "2026-09-29T02:59:24.07754651Z",
      "endTime": "2026-09-29T03:00:05.923503248Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-21 with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc000e962c0>: 
    failed to create NodePool npupgrade-4-21: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-8rl27d" in resourcegroup="rg-np-version-upgrade-8rl27d-cds8pn7rp5pj" to finish creating: GET https://rp.j0435840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/99532f4a-c494-477c-9235-28ce3b3dccdb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/99532f4a-c494-477c-9235-28ce3b3dccdb",
      "name": "99532f4a-c494-477c-9235-28ce3b3dccdb",
      "status": "Failed",
      "startTime": "2026-09-29T02:59:24.07754651Z",
      "endTime": "2026-09-29T03:00:05.923503248Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (4)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-29 01:37	PR #7224	2104747427830435840
2026-09-29 01:37	PR #7215	2104747333580230656

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...

full failure pattern:

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc002a9c120>: 
    resource group "feature-aggregation-58s9mcwldsqm": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-58s9mcwldsqm" to finish deleting: GET https://rp.j4239616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c5803e91-8e02-4c30-b145-ec8ce0a8c23b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c5803e91-8e02-4c30-b145-ec8ce0a8c23b",
      "name": "c5803e91-8e02-4c30-b145-ec8ce0a8c23b",
      "status": "Failed",
      "startTime": "2026-09-29T21:25:15.122990062Z",
      "endTime": "2026-09-29T21:49:22.352588907Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t4tf85mf6cm5no630bapiv0pq2r6pgq still exists (deletion dispatched at 2026-09-29T21:25:17Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc002a9c120>: 
    resource group "feature-aggregation-58s9mcwldsqm": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-58s9mcwldsqm" to finish deleting: GET https://rp.j4239616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c5803e91-8e02-4c30-b145-ec8ce0a8c23b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c5803e91-8e02-4c30-b145-ec8ce0a8c23b",
      "name": "c5803e91-8e02-4c30-b145-ec8ce0a8c23b",
      "status": "Failed",
      "startTime": "2026-09-29T21:25:15.122990062Z",
      "endTime": "2026-09-29T21:49:22.352588907Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t4tf85mf6cm5no630bapiv0pq2r6pgq still exists (deletion dispatched at 2026-09-29T21:25:17Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0011b6150>: 
    resource group "patch-tags-vb572dcgrtxw": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-vb572dcgrtxw" to finish deleting: GET https://rp.j0230656.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/03f22ac5-a416-4ae3-a322-aefdc0c7dba6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/03f22ac5-a416-4ae3-a322-aefdc0c7dba6",
      "name": "03f22ac5-a416-4ae3-a322-aefdc0c7dba6",
      "status": "Failed",
      "startTime": "2026-09-29T03:00:11.459457944Z",
      "endTime": "2026-09-29T03:24:16.961720696Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t4dh8v1rajainbkav3mf373qfaraar5 still exists (deletion dispatched at 2026-09-29T03:00:13Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0011b6150>: 
    resource group "patch-tags-vb572dcgrtxw": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-vb572dcgrtxw" to finish deleting: GET https://rp.j0230656.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/03f22ac5-a416-4ae3-a322-aefdc0c7dba6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/03f22ac5-a416-4ae3-a322-aefdc0c7dba6",
      "name": "03f22ac5-a416-4ae3-a322-aefdc0c7dba6",
      "status": "Failed",
      "startTime": "2026-09-29T03:00:11.459457944Z",
      "endTime": "2026-09-29T03:24:16.961720696Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t4dh8v1rajainbkav3mf373qfaraar5 still exists (deletion dispatched at 2026-09-29T03:00:13Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (2)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-29 19:18	PR #7242	2105024277664239616
2026-09-29 01:37	PR #7215	2104747333580230656

alert [svc] ServiceCPUDrift fired

full failure pattern:

alert [svc] ServiceCPUDrift fired

Signal:

alert

none

Full failure examples (2)

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-29T17:44:34Z
  Ended:       2026-09-29T18:19:33Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3617536-svc", component="service-memory", container="aro-hcp-frontend", namespace="aro-hcp", pod="aro-hcp-frontend-67d6bd86-dxk5c", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-frontend in pod aro-hcp-frontend-67d6bd86-dxk5c (namespace aro-hcp) on cluster ci01-j3617536-svc has averaged 177.5% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-29T17:44:34Z
  Ended:       2026-09-29T18:31:36Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3617536-svc", component="service-memory", container="aro-hcp-frontend", namespace="aro-hcp", pod="aro-hcp-frontend-67d6bd86-wwm6n", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-frontend in pod aro-hcp-frontend-67d6bd86-wwm6n (namespace aro-hcp) on cluster ci01-j3617536-svc has averaged 220.8% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-29T03:34:52Z
  Ended:       2026-09-29T03:55:53Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6245760-svc", component="service-memory", container="aro-hcp-frontend", namespace="aro-hcp", pod="aro-hcp-frontend-588d586db6-sjv52", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-frontend in pod aro-hcp-frontend-588d586db6-sjv52 (namespace aro-hcp) on cluster ci01-j6245760-svc has averaged 159.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-29T03:34:52Z
  Ended:       2026-09-29T03:55:53Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6245760-svc", component="service-memory", container="aro-hcp-frontend", namespace="aro-hcp", pod="aro-hcp-frontend-588d586db6-ksmpx", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-frontend in pod aro-hcp-frontend-588d586db6-ksmpx (namespace aro-hcp) on cluster ci01-j6245760-svc has averaged 151.5% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert ServiceCPUDrift does not fire	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-29 16:20	PR #7178	2104969648343617536
2026-09-29 01:55	PR #7178	2104751945536245760

alert [hcp] KubePodCrashLooping fired

full failure pattern:

alert [hcp] KubePodCrashLooping fired

Signal:

alert

none

Full failure examples (2)

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-29T19:37:53Z
  Ended:       2026-09-29T20:07:50Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3543680-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.87:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3543680/providers/microsoft.monitor/accounts/hcps-j3543680", namespace="ocm-arohcpci01-2t4s0nlb3kmtdnv66shggj311g6csrra-basic-cluster", pod="etcd-2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="d56b7e50-ca7c-400c-bbda-078a761f4494"
  Description: Pod ocm-arohcpci01-2t4s0nlb3kmtdnv66shggj311g6csrra-basic-cluster/etcd-2 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-09-29T19:37:53Z
  Ended:       2026-09-29T20:06:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3543680-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.87:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3543680/providers/microsoft.monitor/accounts/hcps-j3543680", namespace="ocm-arohcpci01-2t4s0nlb3kmtdnv66shggj311g6csrra-basic-cluster", pod="etcd-2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="d56b7e50-ca7c-400c-bbda-078a761f4494"
  Description: Pod ocm-arohcpci01-2t4s0nlb3kmtdnv66shggj311g6csrra-basic-cluster/etcd-2 (etcd) is in waiting state (reason: "CrashLoopBackOff").

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-29T16:55:04Z
  Ended:       2026-09-29T17:23:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0873728-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0873728/providers/microsoft.monitor/accounts/hcps-j0873728", namespace="ocm-arohcpci01-2t4pkau0b0nuc1epds7jnuhhosgt2got-c8u2v9v3j9q8t4s", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="77d06f7d-9903-4077-b53b-f78669a34cec"
  Description: Pod ocm-arohcpci01-2t4pkau0b0nuc1epds7jnuhhosgt2got-c8u2v9v3j9q8t4s/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-09-29T16:55:04Z
  Ended:       2026-09-29T17:20:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0873728-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0873728/providers/microsoft.monitor/accounts/hcps-j0873728", namespace="ocm-arohcpci01-2t4pkau0b0nuc1epds7jnuhhosgt2got-c8u2v9v3j9q8t4s", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="77d06f7d-9903-4077-b53b-f78669a34cec"
  Description: Pod ocm-arohcpci01-2t4pkau0b0nuc1epds7jnuhhosgt2got-c8u2v9v3j9q8t4s/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert KubePodCrashLooping does not fire	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-29 18:00	PR #6915	2105000598473543680
2026-09-29 15:40	PR #7231	2104959408780873728

alert [svc] KubeDeploymentReplicasMismatch fired

full failure pattern:

alert [svc] KubeDeploymentReplicasMismatch fired

Signal:

alert

none

Full failure examples (2)

alert fired 8 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-29T16:58:29Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0873728-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0873728/providers/microsoft.monitor/accounts/services-j0873728", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7988dc648c-qqn8d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-29T16:58:29Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0873728-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0873728/providers/microsoft.monitor/accounts/services-j0873728", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7988dc648c-qqn8d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-29T17:07:31Z
  Ended:       2026-09-29T17:35:30Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0873728-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0873728/providers/microsoft.monitor/accounts/services-j0873728", namespace="klusterlet-2t4pk8k1eqei6ipqjn5eo2qde37f917q", pod="arohcp-monitor-kube-state-metrics-7988dc648c-qqn8d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t4pk8k1eqei6ipqjn5eo2qde37f917q/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-29T17:07:31Z
  Ended:       2026-09-29T17:35:30Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0873728-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0873728/providers/microsoft.monitor/accounts/services-j0873728", namespace="klusterlet-2t4pk8k1eqei6ipqjn5eo2qde37f917q", pod="arohcp-monitor-kube-state-metrics-7988dc648c-qqn8d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t4pk8k1eqei6ipqjn5eo2qde37f917q/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-09-29T17:29:29Z
  Ended:       2026-09-29T17:39:29Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0873728-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0873728/providers/microsoft.monitor/accounts/services-j0873728", namespace="klusterlet-2t4pvsjcuds2fgpd5nc13jc1b6vtv0ra", pod="arohcp-monitor-kube-state-metrics-7988dc648c-qqn8d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t4pvsjcuds2fgpd5nc13jc1b6vtv0ra/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-09-29T17:29:29Z
  Ended:       2026-09-29T17:39:29Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0873728-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0873728/providers/microsoft.monitor/accounts/services-j0873728", namespace="klusterlet-2t4pvsjcuds2fgpd5nc13jc1b6vtv0ra", pod="arohcp-monitor-kube-state-metrics-7988dc648c-qqn8d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t4pvsjcuds2fgpd5nc13jc1b6vtv0ra/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-09-29T17:34:30Z
  Ended:       2026-09-29T17:40:33Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0873728-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0873728/providers/microsoft.monitor/accounts/services-j0873728", namespace="klusterlet-2t4q266hvr5t8e6kmj10ip38mft9e346", pod="arohcp-monitor-kube-state-metrics-7988dc648c-qqn8d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t4q266hvr5t8e6kmj10ip38mft9e346/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-09-29T17:34:30Z
  Ended:       2026-09-29T17:41:29Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0873728-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0873728/providers/microsoft.monitor/accounts/services-j0873728", namespace="klusterlet-2t4q266hvr5t8e6kmj10ip38mft9e346", pod="arohcp-monitor-kube-state-metrics-7988dc648c-qqn8d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t4q266hvr5t8e6kmj10ip38mft9e346/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.

alert fired 14 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-29T10:17:54Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3025664-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.135:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3025664/providers/microsoft.monitor/accounts/services-j3025664", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7988dc648c-gxfn6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-29T10:17:54Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3025664-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.135:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3025664/providers/microsoft.monitor/accounts/services-j3025664", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7988dc648c-gxfn6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-29T10:22:52Z
  Ended:       2026-09-29T10:32:53Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3025664-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.135:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3025664/providers/microsoft.monitor/accounts/services-j3025664", namespace="klusterlet-2t4jnvs8kd2mj67qj3qjdr0prgmk3g16", pod="arohcp-monitor-kube-state-metrics-7988dc648c-gxfn6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t4jnvs8kd2mj67qj3qjdr0prgmk3g16/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-29T10:22:52Z
  Ended:       2026-09-29T10:32:53Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3025664-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.135:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3025664/providers/microsoft.monitor/accounts/services-j3025664", namespace="klusterlet-2t4jnvs8kd2mj67qj3qjdr0prgmk3g16", pod="arohcp-monitor-kube-state-metrics-7988dc648c-gxfn6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t4jnvs8kd2mj67qj3qjdr0prgmk3g16/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-09-29T10:23:53Z
  Ended:       2026-09-29T10:49:55Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3025664-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.135:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3025664/providers/microsoft.monitor/accounts/services-j3025664", namespace="klusterlet-2t4jo44vp2jv0nlkcl93iohtjucb2f9b", pod="arohcp-monitor-kube-state-metrics-7988dc648c-gxfn6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t4jo44vp2jv0nlkcl93iohtjucb2f9b/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-09-29T10:23:53Z
  Ended:       2026-09-29T10:49:55Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3025664-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.135:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3025664/providers/microsoft.monitor/accounts/services-j3025664", namespace="klusterlet-2t4jo44vp2jv0nlkcl93iohtjucb2f9b", pod="arohcp-monitor-kube-state-metrics-7988dc648c-gxfn6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t4jo44vp2jv0nlkcl93iohtjucb2f9b/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-09-29T10:41:53Z
  Ended:       2026-09-29T10:53:52Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3025664-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.135:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3025664/providers/microsoft.monitor/accounts/services-j3025664", namespace="klusterlet-2t4k0pk1qfbh6ckcvag872v8bpqll942", pod="arohcp-monitor-kube-state-metrics-7988dc648c-gxfn6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t4k0pk1qfbh6ckcvag872v8bpqll942/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-09-29T10:41:53Z
  Ended:       2026-09-29T10:50:53Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3025664-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.135:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3025664/providers/microsoft.monitor/accounts/services-j3025664", namespace="klusterlet-2t4k0nh3n7qatpi8mc239v6qipvko07d", pod="arohcp-monitor-kube-state-metrics-7988dc648c-gxfn6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t4k0nh3n7qatpi8mc239v6qipvko07d/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-09-29T10:41:53Z
  Ended:       2026-09-29T10:55:00Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3025664-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.135:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3025664/providers/microsoft.monitor/accounts/services-j3025664", namespace="klusterlet-2t4k0pk1qfbh6ckcvag872v8bpqll942", pod="arohcp-monitor-kube-state-metrics-7988dc648c-gxfn6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t4k0pk1qfbh6ckcvag872v8bpqll942/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-09-29T10:41:53Z
  Ended:       2026-09-29T10:50:53Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3025664-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.135:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3025664/providers/microsoft.monitor/accounts/services-j3025664", namespace="klusterlet-2t4k0nh3n7qatpi8mc239v6qipvko07d", pod="arohcp-monitor-kube-state-metrics-7988dc648c-gxfn6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t4k0nh3n7qatpi8mc239v6qipvko07d/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-09-29T10:45:53Z
  Ended:       2026-09-29T10:55:00Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3025664-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.135:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3025664/providers/microsoft.monitor/accounts/services-j3025664", namespace="klusterlet-2t4k2egjlp01qontmh09pkq2jcahc57j", pod="arohcp-monitor-kube-state-metrics-7988dc648c-gxfn6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t4k2egjlp01qontmh09pkq2jcahc57j/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-09-29T10:45:53Z
  Ended:       2026-09-29T10:55:00Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3025664-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.135:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3025664/providers/microsoft.monitor/accounts/services-j3025664", namespace="klusterlet-2t4k2egjlp01qontmh09pkq2jcahc57j", pod="arohcp-monitor-kube-state-metrics-7988dc648c-gxfn6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t4k2egjlp01qontmh09pkq2jcahc57j/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-09-29T10:46:53Z
  Ended:       2026-09-29T10:57:58Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3025664-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.135:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3025664/providers/microsoft.monitor/accounts/services-j3025664", namespace="klusterlet-2t4k3g6c5jmg9c3neja6l4kgrl2qckj3", pod="arohcp-monitor-kube-state-metrics-7988dc648c-gxfn6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t4k3g6c5jmg9c3neja6l4kgrl2qckj3/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-09-29T10:46:53Z
  Ended:       2026-09-29T10:57:58Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3025664-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.135:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3025664/providers/microsoft.monitor/accounts/services-j3025664", namespace="klusterlet-2t4k3g6c5jmg9c3neja6l4kgrl2qckj3", pod="arohcp-monitor-kube-state-metrics-7988dc648c-gxfn6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t4k3g6c5jmg9c3neja6l4kgrl2qckj3/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert KubeDeploymentReplicasMismatch does not fire	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-29 15:40	PR #7231	2104959408780873728
2026-09-29 09:09	PR #6915	2104861197303025664

alert [svc] KubeDeploymentRolloutStuck fired

full failure pattern:

alert [svc] KubeDeploymentRolloutStuck fired

Signal:

alert

none

Full failure examples (2)

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-29T17:08:31Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j0873728-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0873728/providers/microsoft.monitor/accounts/services-j0873728", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7988dc648c-qqn8d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-29T17:08:31Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j0873728-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0873728/providers/microsoft.monitor/accounts/services-j0873728", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7988dc648c-qqn8d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-29T17:17:28Z
  Ended:       2026-09-29T17:35:31Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j0873728-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0873728/providers/microsoft.monitor/accounts/services-j0873728", namespace="klusterlet-2t4pk8k1eqei6ipqjn5eo2qde37f917q", pod="arohcp-monitor-kube-state-metrics-7988dc648c-qqn8d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment klusterlet-2t4pk8k1eqei6ipqjn5eo2qde37f917q/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-29T17:17:28Z
  Ended:       2026-09-29T17:35:30Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j0873728-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0873728/providers/microsoft.monitor/accounts/services-j0873728", namespace="klusterlet-2t4pk8k1eqei6ipqjn5eo2qde37f917q", pod="arohcp-monitor-kube-state-metrics-7988dc648c-qqn8d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment klusterlet-2t4pk8k1eqei6ipqjn5eo2qde37f917q/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-29T10:27:54Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j3025664-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.135:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3025664/providers/microsoft.monitor/accounts/services-j3025664", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7988dc648c-gxfn6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-29T10:27:54Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j3025664-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.135:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3025664/providers/microsoft.monitor/accounts/services-j3025664", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7988dc648c-gxfn6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-29T10:33:53Z
  Ended:       2026-09-29T10:49:56Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j3025664-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.135:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3025664/providers/microsoft.monitor/accounts/services-j3025664", namespace="klusterlet-2t4jo44vp2jv0nlkcl93iohtjucb2f9b", pod="arohcp-monitor-kube-state-metrics-7988dc648c-gxfn6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment klusterlet-2t4jo44vp2jv0nlkcl93iohtjucb2f9b/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-29T10:33:53Z
  Ended:       2026-09-29T10:49:56Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j3025664-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.135:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3025664/providers/microsoft.monitor/accounts/services-j3025664", namespace="klusterlet-2t4jo44vp2jv0nlkcl93iohtjucb2f9b", pod="arohcp-monitor-kube-state-metrics-7988dc648c-gxfn6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment klusterlet-2t4jo44vp2jv0nlkcl93iohtjucb2f9b/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert KubeDeploymentRolloutStuck does not fire	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-29 15:40	PR #7231	2104959408780873728
2026-09-29 09:09	PR #6915	2104861197303025664

cluster entered terminal Failed state after role assignment deployment

full failure pattern:

cluster entered terminal Failed state after role assignment deployment

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.004s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.004s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-29 09:05	PR #7073	2104860015251689472

clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url>

full failure pattern:

clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url>

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:357]: Timed out after 300.002s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0xc000d0e4a0>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:47677->172.30.0.10:53: read: connection refused
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:357]: Timed out after 300.002s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0xc000d0e4a0>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:47677->172.30.0.10:53: read: connection refused
    ...

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-29 16:52	PR #7241	2104977540106948608

ERROR CODE: ResourceNotFound; detail message The resource 'locations/hcpOperationStatuses/<uuid>' under resource group '' was not found.; provider Microsoft.RedHatOpenShift

full failure pattern:

ERROR CODE: ResourceNotFound; detail message The resource 'locations/hcpOperationStatuses/<uuid>' under resource group '' was not found.; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:282]: failed to update cluster "agg-cluster" authorized CIDRs to include test runner IP
Unexpected error:
    <*exported.ResponseError | 0xc001519d40>: 
    GET https://rp.j7063040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/314c81fe-2478-47d0-ae67-1f8bdc8c0f20
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/314c81fe-2478-47d0-ae67-1f8bdc8c0f20' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/314c81fe-2478-47d0-ae67-1f8bdc8c0f20"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:282]: failed to update cluster "agg-cluster" authorized CIDRs to include test runner IP
Unexpected error:
    <*exported.ResponseError | 0xc001519d40>: 
    GET https://rp.j7063040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/314c81fe-2478-47d0-ae67-1f8bdc8c0f20
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/314c81fe-2478-47d0-ae67-1f8bdc8c0f20' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/314c81fe-2478-47d0-ae67-1f8bdc8c0f20"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-29 14:57	PR #6915	2104948570607063040

VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)

full failure pattern:

VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)

Signal:

e2e

INT, PROD, STG

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.006s.
Expected success, but got an error:
    <*errors.joinError | 0xc000c8f800>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.006s.
Expected success, but got an error:
    <*errors.joinError | 0xc000c8f800>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.003s.
Expected success, but got an error:
    <*errors.joinError | 0xc001542840>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.003s.
Expected success, but got an error:
    <*errors.joinError | 0xc001542840>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.007s.
Expected success, but got an error:
    <*errors.joinError | 0xc001065710>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.007s.
Expected success, but got an error:
    <*errors.joinError | 0xc001065710>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor	41

Affected runs (41)

Date (UTC)	Associated PR	Prow job
2026-09-29 18:44	PR #7053	2105012837859463168
2026-09-29 18:25	PR #7141	2105006438400135168
2026-09-29 18:00	PR #6915	2105000598473543680
2026-09-29 17:48	PR #7124	2104997947618889728
2026-09-29 17:29	PR #6577	2104995028081315840
2026-09-29 16:55	PR #7200	2104978275775287296
2026-09-29 16:52	PR #7241	2104977540106948608
2026-09-29 16:51	PR #6900	2104977492128305152
2026-09-29 16:51	PR #7170	2104977428710428672
2026-09-29 16:51	PR #7238	2104977337324933120
2026-09-29 16:29	PR #7246	2104971730622615552
2026-09-29 16:20	PR #7178	2104969648343617536
2026-09-29 15:43	PR #7169	2104960350616031232
2026-09-29 15:40	PR #7231	2104959408780873728
2026-09-29 15:08	PR #7141	2104951367071174656
2026-09-29 15:07	PR #7001	2104951111046664192
2026-09-29 15:06	PR #7181	2104950948592881664
2026-09-29 15:01	PR #7053	2104949753673420800
2026-09-29 15:00	PR #7183	2104949577500069888
2026-09-29 14:57	PR #6915	2104948570607063040
2026-09-29 14:11	PR #7054	2104937201925099520
2026-09-29 14:09	PR #7104	2104936582908743680
2026-09-29 13:45	PR #6976	2104930620709801984
2026-09-29 13:40	PR #7241	2104929398925824000
2026-09-29 12:49	PR #7220	2104916576108023808
2026-09-29 12:48	PR #7071	2104916338194518016
2026-09-29 12:46	PR #7238	2104915824979480576
2026-09-29 11:29	PR #7169	2104896461568217088
2026-09-29 11:27	PR #6915	2104895962802556928
2026-09-29 11:13	PR #7001	2104892283974324224
2026-09-29 10:18	PR #6746	2104878392305258496
2026-09-29 09:53	PR #7239	2104872117722943488
2026-09-29 09:50	PR #6900	2104871453622013952
2026-09-29 09:49	PR #6922	2104871291713490944
2026-09-29 09:49	PR #7221	2104871248742846464
2026-09-29 09:49	PR #7224	2104871228572438528
2026-09-29 09:49	PR #7068	2104871211610673152
2026-09-29 09:29	PR #7165	2104866139002638336
2026-09-29 09:09	PR #6915	2104861197303025664
2026-09-29 08:59	PR #7169	2104858665063288832
2026-09-29 08:50	PR #7236	2104856376063823872

ERROR CODE: DeadlineExceeded; provider Microsoft.Compute

full failure pattern:

ERROR CODE: DeadlineExceeded; provider Microsoft.Compute

Signal:

e2e

INT, PROD, STG

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0xc0006fe4e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001preview-q2dnllpw9nvn" to finish creating: GET https://rp.j9788032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e0f2d148-6c09-4fa7-92eb-2682cee14684
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e0f2d148-6c09-4fa7-92eb-2682cee14684",
      "name": "e0f2d148-6c09-4fa7-92eb-2682cee14684",
      "status": "Failed",
      "startTime": "2026-09-29T23:12:20.427830985Z",
      "endTime": "2026-09-29T23:31:23.466880978Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20261001-np-1-wjghl-dvxj8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-q2dnllpw9nvn--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-wjghl-dvxj8 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20261001-np-1-wjghl-dvxj8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-q2dnllpw9nvn--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-wjghl-dvxj8 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0xc0006fe4e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001preview-q2dnllpw9nvn" to finish creating: GET https://rp.j9788032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e0f2d148-6c09-4fa7-92eb-2682cee14684
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e0f2d148-6c09-4fa7-92eb-2682cee14684",
      "name": "e0f2d148-6c09-4fa7-92eb-2682cee14684",
      "status": "Failed",
      "startTime": "2026-09-29T23:12:20.427830985Z",
      "endTime": "2026-09-29T23:31:23.466880978Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20261001-np-1-wjghl-dvxj8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-q2dnllpw9nvn--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-wjghl-dvxj8 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20261001-np-1-wjghl-dvxj8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-q2dnllpw9nvn--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-wjghl-dvxj8 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:740]: failed to create nodepool npdg-422-420
Unexpected error:
    <*fmt.wrapError | 0xc001071820>: 
    failed to create NodePool npdg-422-420: failed waiting for nodepool="npdg-422-420" for cluster "np-dg-minor-pxnddw" in resourcegroup="rg-np-dg-minor-pxnddw-dsdsdnq4v6ch" to finish creating: GET https://rp.j1355136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bedd488d-acf4-4a7b-9e30-cb9921b62867
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bedd488d-acf4-4a7b-9e30-cb9921b62867",
      "name": "bedd488d-acf4-4a7b-9e30-cb9921b62867",
      "status": "Failed",
      "startTime": "2026-09-29T22:05:15.883513695Z",
      "endTime": "2026-09-29T22:24:16.137353481Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q1b0s9q1x0x2p8w-npdg-422-420-t77mj-czdm4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-pxnddw-dsdsdnq4v6ch-np-dgm-pxnddw--managed/providers/Microsoft.Compute/virtualMachines/q1b0s9q1x0x2p8w-npdg-422-420-t77mj-czdm4 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q1b0s9q1x0x2p8w-npdg-422-420-t77mj-czdm4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-pxnddw-dsdsdnq4v6ch-np-dgm-pxnddw--managed/providers/Microsoft.Compute/virtualMachines/q1b0s9q1x0x2p8w-npdg-422-420-t77mj-czdm4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:740]: failed to create nodepool npdg-422-420
Unexpected error:
    <*fmt.wrapError | 0xc001071820>: 
    failed to create NodePool npdg-422-420: failed waiting for nodepool="npdg-422-420" for cluster "np-dg-minor-pxnddw" in resourcegroup="rg-np-dg-minor-pxnddw-dsdsdnq4v6ch" to finish creating: GET https://rp.j1355136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bedd488d-acf4-4a7b-9e30-cb9921b62867
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bedd488d-acf4-4a7b-9e30-cb9921b62867",
      "name": "bedd488d-acf4-4a7b-9e30-cb9921b62867",
      "status": "Failed",
      "startTime": "2026-09-29T22:05:15.883513695Z",
      "endTime": "2026-09-29T22:24:16.137353481Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q1b0s9q1x0x2p8w-npdg-422-420-t77mj-czdm4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-pxnddw-dsdsdnq4v6ch-np-dgm-pxnddw--managed/providers/Microsoft.Compute/virtualMachines/q1b0s9q1x0x2p8w-npdg-422-420-t77mj-czdm4 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q1b0s9q1x0x2p8w-npdg-422-420-t77mj-czdm4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-pxnddw-dsdsdnq4v6ch-np-dgm-pxnddw--managed/providers/Microsoft.Compute/virtualMachines/q1b0s9q1x0x2p8w-npdg-422-420-t77mj-czdm4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:270]: failed to create node pool oidc-wi-np
Unexpected error:
    <*fmt.wrapError | 0xc000b835a0>: 
    failed to create NodePool oidc-wi-np: failed waiting for nodepool="oidc-wi-np" for cluster "oidc-wi-cluster" in resourcegroup="oidc-wi-d25xmd5lmjzd" to finish creating: GET https://rp.j4742400.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a260492b-096c-44ec-a66c-3ad97273eaa0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a260492b-096c-44ec-a66c-3ad97273eaa0",
      "name": "a260492b-096c-44ec-a66c-3ad97273eaa0",
      "status": "Failed",
      "startTime": "2026-09-29T21:43:34.965858019Z",
      "endTime": "2026-09-29T22:02:41.653989842Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine oidc-wi-cluster-oidc-wi-np-7hh2h-jn4bt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-d25xmd5lmjzd--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-7hh2h-jn4bt to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine oidc-wi-cluster-oidc-wi-np-7hh2h-jn4bt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-d25xmd5lmjzd--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-7hh2h-jn4bt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:270]: failed to create node pool oidc-wi-np
Unexpected error:
    <*fmt.wrapError | 0xc000b835a0>: 
    failed to create NodePool oidc-wi-np: failed waiting for nodepool="oidc-wi-np" for cluster "oidc-wi-cluster" in resourcegroup="oidc-wi-d25xmd5lmjzd" to finish creating: GET https://rp.j4742400.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a260492b-096c-44ec-a66c-3ad97273eaa0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a260492b-096c-44ec-a66c-3ad97273eaa0",
      "name": "a260492b-096c-44ec-a66c-3ad97273eaa0",
      "status": "Failed",
      "startTime": "2026-09-29T21:43:34.965858019Z",
      "endTime": "2026-09-29T22:02:41.653989842Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine oidc-wi-cluster-oidc-wi-np-7hh2h-jn4bt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-d25xmd5lmjzd--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-7hh2h-jn4bt to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine oidc-wi-cluster-oidc-wi-np-7hh2h-jn4bt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-d25xmd5lmjzd--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-7hh2h-jn4bt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (9)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.1	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to retrieve cluster and nodepool status active versions	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to use workload identity via the cluster OIDC issuer URL	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest	1

Affected runs (10)

Date (UTC)	Associated PR	Prow job
2026-09-29 21:50	PR #6900	2105052569519788032
2026-09-29 20:48	PR #6915	2105037155771355136
2026-09-29 20:15	PR #7250	2105029879664742400
2026-09-29 17:48	PR #7124	2104997947618889728
2026-09-29 13:45	PR #6976	2104930620709801984
2026-09-29 09:53	PR #7239	2104872117722943488
2026-09-29 09:49	PR #7068	2104871211610673152
2026-09-29 08:50	PR #7236	2104856376063823872
2026-09-29 01:37	PR #7224	2104747427830435840
2026-09-29 01:37	PR #7221	2104747373145100288

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:89]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc00114aa60>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-jglxzzbsxnh6" to finish creating: GET https://rp.j8692992.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2ba10d5a-1e61-4b26-8ed8-33a90f52a770
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2ba10d5a-1e61-4b26-8ed8-33a90f52a770",
      "name": "2ba10d5a-1e61-4b26-8ed8-33a90f52a770",
      "status": "Failed",
      "startTime": "2026-09-29T23:06:32.779086006Z",
      "endTime": "2026-09-29T23:35:37.525993022Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 20m35s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:89]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc00114aa60>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-jglxzzbsxnh6" to finish creating: GET https://rp.j8692992.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2ba10d5a-1e61-4b26-8ed8-33a90f52a770
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2ba10d5a-1e61-4b26-8ed8-33a90f52a770",
      "name": "2ba10d5a-1e61-4b26-8ed8-33a90f52a770",
      "status": "Failed",
      "startTime": "2026-09-29T23:06:32.779086006Z",
      "endTime": "2026-09-29T23:35:37.525993022Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 20m35s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:99]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000c2a520>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-ghf4jfbdlg72" to finish creating: GET https://rp.j8692992.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/596355a9-5b86-46b1-9839-13786177e8e4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/596355a9-5b86-46b1-9839-13786177e8e4",
      "name": "596355a9-5b86-46b1-9839-13786177e8e4",
      "status": "Failed",
      "startTime": "2026-09-29T23:06:53.563704915Z",
      "endTime": "2026-09-29T23:35:57.862623465Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.16 is Partial (want Completed), started 21m32s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:99]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000c2a520>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-ghf4jfbdlg72" to finish creating: GET https://rp.j8692992.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/596355a9-5b86-46b1-9839-13786177e8e4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/596355a9-5b86-46b1-9839-13786177e8e4",
      "name": "596355a9-5b86-46b1-9839-13786177e8e4",
      "status": "Failed",
      "startTime": "2026-09-29T23:06:53.563704915Z",
      "endTime": "2026-09-29T23:35:57.862623465Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.16 is Partial (want Completed), started 21m32s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000fdc1a0>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-s7n5pwj29kws" to finish creating: GET https://rp.j6848896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/74696105-f579-4b73-9d4f-01cd26fb675e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/74696105-f579-4b73-9d4f-01cd26fb675e",
      "name": "74696105-f579-4b73-9d4f-01cd26fb675e",
      "status": "Failed",
      "startTime": "2026-09-29T22:53:48.685100711Z",
      "endTime": "2026-09-29T23:22:58.576047333Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 22m3s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000fdc1a0>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-s7n5pwj29kws" to finish creating: GET https://rp.j6848896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/74696105-f579-4b73-9d4f-01cd26fb675e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/74696105-f579-4b73-9d4f-01cd26fb675e",
      "name": "74696105-f579-4b73-9d4f-01cd26fb675e",
      "status": "Failed",
      "startTime": "2026-09-29T22:53:48.685100711Z",
      "endTime": "2026-09-29T23:22:58.576047333Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 22m3s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (8)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to retrieve cluster and nodepool status active versions	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to delete a nodepool	1

Affected runs (6)

Date (UTC)	Associated PR	Prow job
2026-09-29 22:22	PR #7054	2105060628308692992
2026-09-29 22:01	PR #7254	2105055507696848896
2026-09-29 12:49	PR #7220	2104916576108023808
2026-09-29 11:13	PR #7001	2104892283974324224
2026-09-29 08:50	PR #7236	2104856376063823872
2026-09-29 01:55	PR #7178	2104751945536245760

ERROR CODE: DeploymentFailed; detail code OverconstrainedZonalAllocationRequest; detail message Allocation failed. VM(s) with the following constraints cannot be allocated, becaus...

full failure pattern:

ERROR CODE: DeploymentFailed; detail code OverconstrainedZonalAllocationRequest; detail message Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\ - Availability Zone\\ - Differencing (Ephemeral) Disks\\ - Networking Constraints (such as Accelerated Networking or IPv6)\\ - VM Size\\; provider Microsoft.Compute

Signal:

provision

none

Full failure examples (3)

time=2026-09-29T09:32:36.709Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-09-29T09:40:14.775Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 err="stamp 2: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-29T09:32:36.720660292Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j0227328-mgmt-2\",\"clusterName\":\"ci01-j0227328-mgmt-2\",\"region\":\"centralus\"}\n{\"time\":\"2026-09-29T09:32:38.364028078Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j0227328-mgmt-2\",\"clusterName\":\"ci01-j0227328-mgmt-2\",\"region\":\"centralus\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-29T09:32:38.559062049Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j0227328-mgmt-2\",\"clusterName\":\"ci01-j0227328-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n{\"time\":\"2026-09-29T09:40:14.773569619Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j0227328-mgmt-2\",\"clusterName\":\"ci01-j0227328-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"939DC55C48692422\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0227328-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j0227328-mgmt-2/agentPools/userswft2\",\"resourceName\":\"ci01-j0227328-mgmt-2/userswft2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-29T09:38:15.5365612Z\",\"duration\":\"PT1M53.6748858S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-29T09:40:14.773629341Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j0227328-mgmt-2\",\"clusterName\":\"ci01-j0227328-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"7244CB440D997254\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0227328-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j0227328-mgmt-2/agentPools/infra2\",\"resourceName\":\"ci01-j0227328-mgmt-2/infra2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-29T09:38:15.5365612Z\",\"duration\":\"PT1M23.3840019S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-29T09:40:14.773642111Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j0227328-mgmt-2\",\"clusterName\":\"ci01-j0227328-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"1CB00462E228B7F2\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0227328-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j0227328-mgmt-2/agentPools/userswft3\",\"resourceName\":\"ci01-j0227328-mgmt-2/userswft3\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-29T09:38:15.5365612Z\",\"duration\":\"PT1M22.8983991S\",\"statusCode\":\"Conflict\",\"statusMessage\":{\"error\":{\"code\":\"OverconstrainedZonalAllocationRequest\",\"details\":[{\"code\":\"OverconstrainedZonalAllocationRequest\",\"message\":\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\n  - Availability Zone\\n  - Differencing (Ephemeral) Disks\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\n  - VM Size\\n\",\"target\":\"0\"}],\"message\":\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0227328-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-52905658-vmss'. Please try alternative sizes 'Standard_D16s_v6' (zones 2, 1), 'Standard_D16ds_v7' (zones 2), 'Standard_D16ads_v7' (zones 3, 1, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0227328-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-52905658-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\n  - Availability Zone\\n  - Differencing (Ephemeral) Disks\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\n  - VM Size\\n\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-29T09:40:14.773711183Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j0227328-mgmt-2\",\"clusterName\":\"ci01-j0227328-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"2656F8058F363717\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0227328-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j0227328-mgmt-2/agentPools/userswft1\",\"resourceName\":\"ci01-j0227328-mgmt-2/userswft1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-29T09:38:15.5365612Z\",\"duration\":\"PT1M53.6953657S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-29T09:40:14.773723573Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j0227328-mgmt-2\",\"clusterName\":\"ci01-j0227328-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"14B4CF500D615D35\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0227328-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j0227328-mgmt-2/agentPools/infra1\",\"resourceName\":\"ci01-j0227328-mgmt-2/infra1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-29T09:32:43.8316068Z\",\"duration\":\"PT6M54.993084S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-29T09:40:14.773734363Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j0227328-mgmt-2\",\"clusterName\":\"ci01-j0227328-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"389A184917075DF4\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0227328-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j0227328-mgmt-2\",\"resourceName\":\"ci01-j0227328-mgmt-2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-29T09:32:43.8316068Z\",\"duration\":\"PT5M31.5756796S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-29T09:40:14.773756484Z\",\"level\":\"ERROR\",\"msg\":\"run failed\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j0227328-mgmt-2\",\"clusterName\":\"ci01-j0227328-mgmt-2\",\"region\":\"centralus\",\"err\":\"waiting for deployment aks-cluster-create: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0227328-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-create/operationStatuses/08584109325266766857\\n--------------------------------------------------------------------------------\\nRESPONSE 200: 200 OK\\nERROR CODE: DeploymentFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"status\\\": \\\"Failed\\\",\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"DeploymentFailed\\\",\\n    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0227328-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-create\\\",\\n    \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\n    \\\"details\\\": [\\n      {\\n        \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0227328-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j0227328-mgmt-2/agentPools/userswft3\\\",\\n        \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\n        \\\"details\\\": [\\n          {\\n            \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\n            \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0227328-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-52905658-vmss'. Please try alternative sizes 'Standard_D16s_v6' (zones 2, 1), 'Standard_D16ds_v7' (zones 2), 'Standard_D16ads_v7' (zones 3, 1, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0227328-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-52905658-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n  - Availability Zone\\\\n  - Differencing (Ephemeral) Disks\\\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n  - VM Size\\\\n\\\",\\n            \\\"details\\\": [\\n              {\\n                \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\n                \\\"target\\\": \\\"0\\\",\\n                \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n  - Availability Zone\\\\n  - Differencing (Ephemeral) Disks\\\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n  - VM Size\\\\n\\\"\\n              }\\n            ]\\n          }\\n        ]\\n      }\\n    ]\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\",\"elapsed\":\"7m38.053105252s\"}\n exit status 1"

time=2026-09-29T08:34:25.705Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-09-29T08:42:03.922Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 err="stamp 1: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-29T08:34:25.714664724Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j1163648-mgmt-1\",\"clusterName\":\"ci01-j1163648-mgmt-1\",\"region\":\"centralus\"}\n{\"time\":\"2026-09-29T08:34:27.934335048Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j1163648-mgmt-1\",\"clusterName\":\"ci01-j1163648-mgmt-1\",\"region\":\"centralus\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-29T08:34:28.153595232Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j1163648-mgmt-1\",\"clusterName\":\"ci01-j1163648-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n{\"time\":\"2026-09-29T08:42:03.919377937Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j1163648-mgmt-1\",\"clusterName\":\"ci01-j1163648-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"E0CC533F465FCB4C\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1163648-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j1163648-mgmt-1/agentPools/userswft1\",\"resourceName\":\"ci01-j1163648-mgmt-1/userswft1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-29T08:39:20.5956034Z\",\"duration\":\"PT1M53.0427309S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-29T08:42:03.919443378Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j1163648-mgmt-1\",\"clusterName\":\"ci01-j1163648-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"21FA5A5958E17E85\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1163648-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j1163648-mgmt-1/agentPools/infra2\",\"resourceName\":\"ci01-j1163648-mgmt-1/infra2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-29T08:39:20.5956034Z\",\"duration\":\"PT1M37.3664627S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-29T08:42:03.919464379Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j1163648-mgmt-1\",\"clusterName\":\"ci01-j1163648-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"3FBD33CFC93FD8AF\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1163648-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j1163648-mgmt-1/agentPools/userswft3\",\"resourceName\":\"ci01-j1163648-mgmt-1/userswft3\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-29T08:39:20.5956034Z\",\"duration\":\"PT35.412467S\",\"statusCode\":\"Conflict\",\"statusMessage\":{\"error\":{\"code\":\"OverconstrainedZonalAllocationRequest\",\"details\":[{\"code\":\"OverconstrainedZonalAllocationRequest\",\"message\":\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\n  - Availability Zone\\n  - Differencing (Ephemeral) Disks\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\n  - VM Size\\n\",\"target\":\"0\"}],\"message\":\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1163648-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-12237352-vmss'. Please try alternative sizes 'Standard_D16s_v6' (zones 2, 1), 'Standard_D16ds_v7' (zones 2), 'Standard_D16ads_v7' (zones 1, 3, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1163648-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-12237352-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\n  - Availability Zone\\n  - Differencing (Ephemeral) Disks\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\n  - VM Size\\n\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-29T08:42:03.919563681Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j1163648-mgmt-1\",\"clusterName\":\"ci01-j1163648-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"F8F7C66DAE59E102\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1163648-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j1163648-mgmt-1/agentPools/infra1\",\"resourceName\":\"ci01-j1163648-mgmt-1/infra1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-29T08:39:20.5956034Z\",\"duration\":\"PT1M22.2906097S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-29T08:42:03.919594142Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j1163648-mgmt-1\",\"clusterName\":\"ci01-j1163648-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"109FB9AA097998DF\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1163648-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j1163648-mgmt-1/agentPools/userswft2\",\"resourceName\":\"ci01-j1163648-mgmt-1/userswft2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-29T08:34:33.6987466Z\",\"duration\":\"PT7M26.7702251S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-29T08:42:03.919629593Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j1163648-mgmt-1\",\"clusterName\":\"ci01-j1163648-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"A22B902E63DA51E7\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1163648-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j1163648-mgmt-1\",\"resourceName\":\"ci01-j1163648-mgmt-1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-29T08:34:33.6987466Z\",\"duration\":\"PT4M46.8300264S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-29T08:42:03.919662274Z\",\"level\":\"ERROR\",\"msg\":\"run failed\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j1163648-mgmt-1\",\"clusterName\":\"ci01-j1163648-mgmt-1\",\"region\":\"centralus\",\"err\":\"waiting for deployment aks-cluster-create: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1163648-mgmt-1/providers/Microsoft.Resources/deployments/aks-cluster-create/operationStatuses/08584109360170854697\\n--------------------------------------------------------------------------------\\nRESPONSE 200: 200 OK\\nERROR CODE: DeploymentFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"status\\\": \\\"Failed\\\",\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"DeploymentFailed\\\",\\n    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1163648-mgmt-1/providers/Microsoft.Resources/deployments/aks-cluster-create\\\",\\n    \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\n    \\\"details\\\": [\\n      {\\n        \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1163648-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j1163648-mgmt-1/agentPools/userswft3\\\",\\n        \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\n        \\\"details\\\": [\\n          {\\n            \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\n            \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1163648-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-12237352-vmss'. Please try alternative sizes 'Standard_D16s_v6' (zones 2, 1), 'Standard_D16ds_v7' (zones 2), 'Standard_D16ads_v7' (zones 1, 3, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1163648-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-12237352-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n  - Availability Zone\\\\n  - Differencing (Ephemeral) Disks\\\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n  - VM Size\\\\n\\\",\\n            \\\"details\\\": [\\n              {\\n                \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\n                \\\"target\\\": \\\"0\\\",\\n                \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n  - Availability Zone\\\\n  - Differencing (Ephemeral) Disks\\\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n  - VM Size\\\\n\\\"\\n              }\\n            ]\\n          }\\n        ]\\n      }\\n    ]\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\",\"elapsed\":\"7m38.20500018s\"}\n exit status 1"

time=2026-09-29T07:29:07.654Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-09-29T07:37:16.632Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 err="stamp 1: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-29T07:29:07.663748341Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j0616320-mgmt-1\",\"clusterName\":\"ci01-j0616320-mgmt-1\",\"region\":\"centralus\"}\n{\"time\":\"2026-09-29T07:29:09.664580159Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j0616320-mgmt-1\",\"clusterName\":\"ci01-j0616320-mgmt-1\",\"region\":\"centralus\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-29T07:29:09.774367851Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j0616320-mgmt-1\",\"clusterName\":\"ci01-j0616320-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n{\"time\":\"2026-09-29T07:37:16.628236265Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j0616320-mgmt-1\",\"clusterName\":\"ci01-j0616320-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"E60E8144CEBBD0EE\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0616320-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j0616320-mgmt-1/agentPools/userswft1\",\"resourceName\":\"ci01-j0616320-mgmt-1/userswft1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-29T07:34:33.4549643Z\",\"duration\":\"PT1M54.5572403S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-29T07:37:16.628296067Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j0616320-mgmt-1\",\"clusterName\":\"ci01-j0616320-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"3B57D2CEB495CF19\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0616320-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j0616320-mgmt-1/agentPools/userswft3\",\"resourceName\":\"ci01-j0616320-mgmt-1/userswft3\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-29T07:34:33.4549643Z\",\"duration\":\"PT1M22.0705191S\",\"statusCode\":\"Conflict\",\"statusMessage\":{\"error\":{\"code\":\"OverconstrainedZonalAllocationRequest\",\"details\":[{\"code\":\"OverconstrainedZonalAllocationRequest\",\"message\":\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\n  - Availability Zone\\n  - Differencing (Ephemeral) Disks\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\n  - VM Size\\n\",\"target\":\"3\"}],\"message\":\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0616320-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-41550067-vmss'. Please try alternative sizes 'Standard_D16s_v6' (zones 2, 1), 'Standard_D16ds_v7' (zones 2), 'Standard_D16ads_v7' (zones 1, 3, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0616320-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-41550067-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\n  - Availability Zone\\n  - Differencing (Ephemeral) Disks\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\n  - VM Size\\n\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-29T07:37:16.628368178Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j0616320-mgmt-1\",\"clusterName\":\"ci01-j0616320-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"1FDC71FA470AB50C\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0616320-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j0616320-mgmt-1/agentPools/infra2\",\"resourceName\":\"ci01-j0616320-mgmt-1/infra2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-29T07:29:15.5482717Z\",\"duration\":\"PT6M40.5743573S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-29T07:37:16.628379928Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j0616320-mgmt-1\",\"clusterName\":\"ci01-j0616320-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"E9836E8873033651\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0616320-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j0616320-mgmt-1/agentPools/infra1\",\"resourceName\":\"ci01-j0616320-mgmt-1/infra1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-29T07:34:33.4549643Z\",\"duration\":\"PT1M38.3097269S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-29T07:37:16.628390179Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j0616320-mgmt-1\",\"clusterName\":\"ci01-j0616320-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"419B3E4C9E087C8E\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0616320-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j0616320-mgmt-1/agentPools/userswft2\",\"resourceName\":\"ci01-j0616320-mgmt-1/userswft2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-29T07:34:33.4549643Z\",\"duration\":\"PT2M23.5295769S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-29T07:37:16.628401639Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j0616320-mgmt-1\",\"clusterName\":\"ci01-j0616320-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"382BA92BCBB89114\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0616320-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j0616320-mgmt-1\",\"resourceName\":\"ci01-j0616320-mgmt-1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-29T07:29:15.5482717Z\",\"duration\":\"PT5M17.7846591S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-29T07:37:16.628426019Z\",\"level\":\"ERROR\",\"msg\":\"run failed\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j0616320-mgmt-1\",\"clusterName\":\"ci01-j0616320-mgmt-1\",\"region\":\"centralus\",\"err\":\"waiting for deployment aks-cluster-create: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0616320-mgmt-1/providers/Microsoft.Resources/deployments/aks-cluster-create/operationStatuses/08584109399346000714\\n--------------------------------------------------------------------------------\\nRESPONSE 200: 200 OK\\nERROR CODE: DeploymentFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"status\\\": \\\"Failed\\\",\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"DeploymentFailed\\\",\\n    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0616320-mgmt-1/providers/Microsoft.Resources/deployments/aks-cluster-create\\\",\\n    \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\n    \\\"details\\\": [\\n      {\\n        \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0616320-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j0616320-mgmt-1/agentPools/userswft3\\\",\\n        \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\n        \\\"details\\\": [\\n          {\\n            \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\n            \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0616320-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-41550067-vmss'. Please try alternative sizes 'Standard_D16s_v6' (zones 2, 1), 'Standard_D16ds_v7' (zones 2), 'Standard_D16ads_v7' (zones 1, 3, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0616320-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-41550067-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n  - Availability Zone\\\\n  - Differencing (Ephemeral) Disks\\\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n  - VM Size\\\\n\\\",\\n            \\\"details\\\": [\\n              {\\n                \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\n                \\\"target\\\": \\\"3\\\",\\n                \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n  - Availability Zone\\\\n  - Differencing (Ephemeral) Disks\\\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n  - VM Size\\\\n\\\"\\n              }\\n            ]\\n          }\\n        ]\\n      }\\n    ]\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\",\"elapsed\":\"8m8.964683458s\"}\n exit status 1"

Contributing tests (2)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster-create (stamp=2)	4
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster-create (stamp=1)	2

Affected runs (6)

Date (UTC)	Associated PR	Prow job
2026-09-29 09:10	PR #6944	2104861480800227328
2026-09-29 08:13	PR #7071	2104846989291163648
2026-09-29 07:05	PR #7236	2104829939730616320
2026-09-29 06:52	PR #6922	2104826523797164032
2026-09-29 06:51	PR #6900	2104826494294429696
2026-09-29 06:25	PR #7073	2104819949015928832

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-23-b9pvc2cjcshp/cluster-candidate-4-23-nm2hv7 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000ea4780>: 
    failed to create HCP cluster cluster-candidate-4-23-nm2hv7: failed waiting for cluster="cluster-candidate-4-23-nm2hv7" in resourcegroup="rg-candidate-4-23-b9pvc2cjcshp" to finish creating: GET https://rp.j0135168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7bbfe34a-1f3c-4ee3-924d-466a04a1edb8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7bbfe34a-1f3c-4ee3-924d-466a04a1edb8",
      "name": "7bbfe34a-1f3c-4ee3-924d-466a04a1edb8",
      "status": "Failed",
      "startTime": "2026-09-29T19:33:49.148595696Z",
      "endTime": "2026-09-29T20:02:49.618696201Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-23-b9pvc2cjcshp/cluster-candidate-4-23-nm2hv7 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000ea4780>: 
    failed to create HCP cluster cluster-candidate-4-23-nm2hv7: failed waiting for cluster="cluster-candidate-4-23-nm2hv7" in resourcegroup="rg-candidate-4-23-b9pvc2cjcshp" to finish creating: GET https://rp.j0135168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7bbfe34a-1f3c-4ee3-924d-466a04a1edb8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7bbfe34a-1f3c-4ee3-924d-466a04a1edb8",
      "name": "7bbfe34a-1f3c-4ee3-924d-466a04a1edb8",
      "status": "Failed",
      "startTime": "2026-09-29T19:33:49.148595696Z",
      "endTime": "2026-09-29T20:02:49.618696201Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc00112a620>: 
    failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-zg4mlcxxqtkw" to finish creating: GET https://rp.j3543680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9a237115-bd4b-4faf-99f3-a126cf693a5b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9a237115-bd4b-4faf-99f3-a126cf693a5b",
      "name": "9a237115-bd4b-4faf-99f3-a126cf693a5b",
      "status": "Failed",
      "startTime": "2026-09-29T19:15:41.667113509Z",
      "endTime": "2026-09-29T19:44:46.805281429Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc00112a620>: 
    failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-zg4mlcxxqtkw" to finish creating: GET https://rp.j3543680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9a237115-bd4b-4faf-99f3-a126cf693a5b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9a237115-bd4b-4faf-99f3-a126cf693a5b",
      "name": "9a237115-bd4b-4faf-99f3-a126cf693a5b",
      "status": "Failed",
      "startTime": "2026-09-29T19:15:41.667113509Z",
      "endTime": "2026-09-29T19:44:46.805281429Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-r85k6z" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0xc001224f00>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-r85k6z: failed waiting for cluster="cp-ystream-upgrade-4-23-r85k6z" in resourcegroup="rg-cp-ystream-upgrade-4-23-v5dv9ks6kpb7" to finish creating: GET https://rp.j0873728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1f6f7bc4-1ea5-4377-a861-1080166398f9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1f6f7bc4-1ea5-4377-a861-1080166398f9",
      "name": "1f6f7bc4-1ea5-4377-a861-1080166398f9",
      "status": "Failed",
      "startTime": "2026-09-29T16:32:43.227956369Z",
      "endTime": "2026-09-29T17:01:49.778635843Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-r85k6z" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0xc001224f00>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-r85k6z: failed waiting for cluster="cp-ystream-upgrade-4-23-r85k6z" in resourcegroup="rg-cp-ystream-upgrade-4-23-v5dv9ks6kpb7" to finish creating: GET https://rp.j0873728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1f6f7bc4-1ea5-4377-a861-1080166398f9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1f6f7bc4-1ea5-4377-a861-1080166398f9",
      "name": "1f6f7bc4-1ea5-4377-a861-1080166398f9",
      "status": "Failed",
      "startTime": "2026-09-29T16:32:43.227956369Z",
      "endTime": "2026-09-29T17:01:49.778635843Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (18)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with ACR pull via managed identity and pull from a private ACR	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to lifecycle and confirm external auth on a cluster	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to rotate KMS key for a cluster with version >= 4.22	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update nodepool replicas and autoscaling	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should not be able to reuse subnets and NSGs between clusters	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.23	1

Affected runs (5)

Date (UTC)	Associated PR	Prow job
2026-09-29 18:25	PR #7141	2105006438400135168
2026-09-29 18:00	PR #6915	2105000598473543680
2026-09-29 15:40	PR #7231	2104959408780873728
2026-09-29 06:52	PR #7068	2104826553132126208
2026-09-29 01:37	PR #7233	2104747306141093888

ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created.

full failure pattern:

ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created.

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:154]: failed to create customer resources for third cluster "basic-hcp-cluster3"
Unexpected error:
    <*fmt.wrapError | 0xc00088a640>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-basic-hcp-cluster3-xlptvl" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster3-xlptvl/operationStatuses/08584109556974857386
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-lxcj7rrjfmg7/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-igpwm4vimfr3y\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard0-03-31/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:154]: failed to create customer resources for third cluster "basic-hcp-cluster3"
Unexpected error:
    <*fmt.wrapError | 0xc00088a640>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-basic-hcp-cluster3-xlptvl" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster3-xlptvl/operationStatuses/08584109556974857386
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-lxcj7rrjfmg7/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-igpwm4vimfr3y\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard0-03-31/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:712]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc001744a60>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-dg-minor-ndgcc2-tplzd9" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-dg-minor-ndgcc2-tplzd9/operationStatuses/08584109554179204837
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-ndgcc2-kcw2dvn877rh/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-bmtu7bm547ml6\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:712]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc001744a60>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-dg-minor-ndgcc2-tplzd9" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-dg-minor-ndgcc2-tplzd9/operationStatuses/08584109554179204837
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-ndgcc2-kcw2dvn877rh/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-bmtu7bm547ml6\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:293]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc001486340>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-noedge-wzv4bm-g699tf" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-noedge-wzv4bm-g699tf/operationStatuses/08584109554270576231
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard0-00-09/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:293]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc001486340>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-noedge-wzv4bm-g699tf" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-noedge-wzv4bm-g699tf/operationStatuses/08584109554270576231
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard0-00-09/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (4)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20	1

Affected runs (4)

Date (UTC)	Associated PR	Prow job
2026-09-29 01:47	PR #7200	2104749868172972032
2026-09-29 01:37	PR #7224	2104747427830435840
2026-09-29 01:37	PR #6922	2104747348675530752
2026-09-29 01:37	PR #7215	2104747333580230656

failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>

full failure pattern:

failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>

Signal:

e2e

PROD

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc000eecfc0>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-f25q6rmztl9d" to finish creating: GET https://rp.j1394688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d3c2e124-eab6-4afe-a82b-d6f4e1f8cabf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d3c2e124-eab6-4afe-a82b-d6f4e1f8cabf",
      "name": "d3c2e124-eab6-4afe-a82b-d6f4e1f8cabf",
      "status": "Failed",
      "startTime": "2026-09-29T23:21:23.268055179Z",
      "endTime": "2026-09-29T23:40:26.327494387Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine m8s8k4n5s9t3q7q-np-delete-main-b4h85-fm9gd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-f25q6rmztl9d--managed/providers/Microsoft.Compute/virtualMachines/m8s8k4n5s9t3q7q-np-delete-main-b4h85-fm9gd to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine m8s8k4n5s9t3q7q-np-delete-main-b4h85-fm9gd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-f25q6rmztl9d--managed/providers/Microsoft.Compute/virtualMachines/m8s8k4n5s9t3q7q-np-delete-main-b4h85-fm9gd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc000eecfc0>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-f25q6rmztl9d" to finish creating: GET https://rp.j1394688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d3c2e124-eab6-4afe-a82b-d6f4e1f8cabf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d3c2e124-eab6-4afe-a82b-d6f4e1f8cabf",
      "name": "d3c2e124-eab6-4afe-a82b-d6f4e1f8cabf",
      "status": "Failed",
      "startTime": "2026-09-29T23:21:23.268055179Z",
      "endTime": "2026-09-29T23:40:26.327494387Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine m8s8k4n5s9t3q7q-np-delete-main-b4h85-fm9gd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-f25q6rmztl9d--managed/providers/Microsoft.Compute/virtualMachines/m8s8k4n5s9t3q7q-np-delete-main-b4h85-fm9gd to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine m8s8k4n5s9t3q7q-np-delete-main-b4h85-fm9gd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-f25q6rmztl9d--managed/providers/Microsoft.Compute/virtualMachines/m8s8k4n5s9t3q7q-np-delete-main-b4h85-fm9gd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc003710200>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-mvqbmn55cv5b" to finish creating: GET https://rp.j9463168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/adec164f-ea16-461c-9762-733a451ff4ec
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/adec164f-ea16-461c-9762-733a451ff4ec",
      "name": "adec164f-ea16-461c-9762-733a451ff4ec",
      "status": "Failed",
      "startTime": "2026-09-29T20:11:35.519536155Z",
      "endTime": "2026-09-29T20:30:41.781416805Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine n0c3u0l0p2x4r6y-np-deletable-4d2w7-wx26q: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-mvqbmn55cv5b--managed/providers/Microsoft.Compute/virtualMachines/n0c3u0l0p2x4r6y-np-deletable-4d2w7-wx26q to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine n0c3u0l0p2x4r6y-np-deletable-4d2w7-wx26q: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-mvqbmn55cv5b--managed/providers/Microsoft.Compute/virtualMachines/n0c3u0l0p2x4r6y-np-deletable-4d2w7-wx26q to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc003710200>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-mvqbmn55cv5b" to finish creating: GET https://rp.j9463168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/adec164f-ea16-461c-9762-733a451ff4ec
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/adec164f-ea16-461c-9762-733a451ff4ec",
      "name": "adec164f-ea16-461c-9762-733a451ff4ec",
      "status": "Failed",
      "startTime": "2026-09-29T20:11:35.519536155Z",
      "endTime": "2026-09-29T20:30:41.781416805Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine n0c3u0l0p2x4r6y-np-deletable-4d2w7-wx26q: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-mvqbmn55cv5b--managed/providers/Microsoft.Compute/virtualMachines/n0c3u0l0p2x4r6y-np-deletable-4d2w7-wx26q to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine n0c3u0l0p2x4r6y-np-deletable-4d2w7-wx26q: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-mvqbmn55cv5b--managed/providers/Microsoft.Compute/virtualMachines/n0c3u0l0p2x4r6y-np-deletable-4d2w7-wx26q to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc0008aa800>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-p7k2xh5g9wv9" to finish creating: GET https://rp.j5824000.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/be0a1eb9-b1b2-4a75-abe0-ce39a79c7bde
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/be0a1eb9-b1b2-4a75-abe0-ce39a79c7bde",
      "name": "be0a1eb9-b1b2-4a75-abe0-ce39a79c7bde",
      "status": "Failed",
      "startTime": "2026-09-29T14:34:49.184621543Z",
      "endTime": "2026-09-29T14:53:52.314346782Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine a0k9y7c0o7e2r3d-np-delete-main-m2wn4-jg6rj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-p7k2xh5g9wv9--managed/providers/Microsoft.Compute/virtualMachines/a0k9y7c0o7e2r3d-np-delete-main-m2wn4-jg6rj to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine a0k9y7c0o7e2r3d-np-delete-main-m2wn4-jg6rj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-p7k2xh5g9wv9--managed/providers/Microsoft.Compute/virtualMachines/a0k9y7c0o7e2r3d-np-delete-main-m2wn4-jg6rj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc0008aa800>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-p7k2xh5g9wv9" to finish creating: GET https://rp.j5824000.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/be0a1eb9-b1b2-4a75-abe0-ce39a79c7bde
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/be0a1eb9-b1b2-4a75-abe0-ce39a79c7bde",
      "name": "be0a1eb9-b1b2-4a75-abe0-ce39a79c7bde",
      "status": "Failed",
      "startTime": "2026-09-29T14:34:49.184621543Z",
      "endTime": "2026-09-29T14:53:52.314346782Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine a0k9y7c0o7e2r3d-np-delete-main-m2wn4-jg6rj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-p7k2xh5g9wv9--managed/providers/Microsoft.Compute/virtualMachines/a0k9y7c0o7e2r3d-np-delete-main-m2wn4-jg6rj to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine a0k9y7c0o7e2r3d-np-delete-main-m2wn4-jg6rj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-p7k2xh5g9wv9--managed/providers/Microsoft.Compute/virtualMachines/a0k9y7c0o7e2r3d-np-delete-main-m2wn4-jg6rj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to delete a nodepool	4

Affected runs (4)

Date (UTC)	Associated PR	Prow job
2026-09-29 22:03	PR #7229	2105055984161394688
2026-09-29 18:44	PR #7053	2105012837859463168
2026-09-29 13:40	PR #7241	2104929398925824000
2026-09-29 01:37	PR #7233	2104747306141093888

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...

full failure pattern:

ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001e14168>: 
    resource group "rg-candidate-4-20-w8n9ljjm626w": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-20-5l8bh4" in resourcegroup="rg-candidate-4-20-w8n9ljjm626w" to finish deleting: GET https://rp.j3617536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9e18bb44-b0e5-444f-94b9-63d3a9ff27a9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9e18bb44-b0e5-444f-94b9-63d3a9ff27a9",
      "name": "9e18bb44-b0e5-444f-94b9-63d3a9ff27a9",
      "status": "Failed",
      "startTime": "2026-09-29T17:42:10.150923038Z",
      "endTime": "2026-09-29T18:06:12.263898636Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t4q8dfpl0ib2cthssvd6at5j30an4b9 still exists (deletion dispatched at 2026-09-29T17:42:27Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001e14168>: 
    resource group "rg-candidate-4-20-w8n9ljjm626w": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-20-5l8bh4" in resourcegroup="rg-candidate-4-20-w8n9ljjm626w" to finish deleting: GET https://rp.j3617536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9e18bb44-b0e5-444f-94b9-63d3a9ff27a9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9e18bb44-b0e5-444f-94b9-63d3a9ff27a9",
      "name": "9e18bb44-b0e5-444f-94b9-63d3a9ff27a9",
      "status": "Failed",
      "startTime": "2026-09-29T17:42:10.150923038Z",
      "endTime": "2026-09-29T18:06:12.263898636Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t4q8dfpl0ib2cthssvd6at5j30an4b9 still exists (deletion dispatched at 2026-09-29T17:42:27Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001d822b8>: 
    resource group "rg-candidate-5-1-4df2rb88kslv": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-5-1-8lqd77" in resourcegroup="rg-candidate-5-1-4df2rb88kslv" to finish deleting: GET https://rp.j3617536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6f2dd559-18f1-4c7c-a090-7b2cb32f6352
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6f2dd559-18f1-4c7c-a090-7b2cb32f6352",
      "name": "6f2dd559-18f1-4c7c-a090-7b2cb32f6352",
      "status": "Failed",
      "startTime": "2026-09-29T17:42:25.116814076Z",
      "endTime": "2026-09-29T18:06:32.319934274Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t4q8hfn91loc9a4dn47b48hi462fllh still exists (deletion dispatched at 2026-09-29T17:42:27Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001d822b8>: 
    resource group "rg-candidate-5-1-4df2rb88kslv": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-5-1-8lqd77" in resourcegroup="rg-candidate-5-1-4df2rb88kslv" to finish deleting: GET https://rp.j3617536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6f2dd559-18f1-4c7c-a090-7b2cb32f6352
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6f2dd559-18f1-4c7c-a090-7b2cb32f6352",
      "name": "6f2dd559-18f1-4c7c-a090-7b2cb32f6352",
      "status": "Failed",
      "startTime": "2026-09-29T17:42:25.116814076Z",
      "endTime": "2026-09-29T18:06:32.319934274Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t4q8hfn91loc9a4dn47b48hi462fllh still exists (deletion dispatched at 2026-09-29T17:42:27Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00259e030>: 
    resource group "feature-aggregation-zkcnx8tdngds": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-zkcnx8tdngds" to finish deleting: GET https://rp.j3617536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bd8e3f22-69a7-45cb-9a4f-423ef0d53d3f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bd8e3f22-69a7-45cb-9a4f-423ef0d53d3f",
      "name": "bd8e3f22-69a7-45cb-9a4f-423ef0d53d3f",
      "status": "Failed",
      "startTime": "2026-09-29T17:53:02.426632555Z",
      "endTime": "2026-09-29T18:17:09.63164867Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t4q9u7coc31q0vdncuqjcta1uk5qsbm still exists (deletion dispatched at 2026-09-29T17:53:04Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00259e030>: 
    resource group "feature-aggregation-zkcnx8tdngds": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-zkcnx8tdngds" to finish deleting: GET https://rp.j3617536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bd8e3f22-69a7-45cb-9a4f-423ef0d53d3f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bd8e3f22-69a7-45cb-9a4f-423ef0d53d3f",
      "name": "bd8e3f22-69a7-45cb-9a4f-423ef0d53d3f",
      "status": "Failed",
      "startTime": "2026-09-29T17:53:02.426632555Z",
      "endTime": "2026-09-29T18:17:09.63164867Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t4q9u7coc31q0vdncuqjcta1uk5qsbm still exists (deletion dispatched at 2026-09-29T17:53:04Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (12)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.1	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to retrieve cluster and nodepool status active versions	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-29 16:20	PR #7178	2104969648343617536
2026-09-29 01:55	PR #7178	2104751945536245760

alert [svc] KustoLogsDataStale fired

full failure pattern:

alert [svc] KustoLogsDataStale fired

Signal:

alert

none

Full failure examples (2)

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-30T00:09:36Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j8264832-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.41:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8264832/providers/microsoft.monitor/accounts/services-j8264832", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-7bf4cc4544-w8cbp", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j8264832-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-30T00:09:36Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j8264832-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.41:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8264832/providers/microsoft.monitor/accounts/services-j8264832", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-7bf4cc4544-w8cbp", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j8264832-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-29T13:36:58Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j4324224-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.28:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4324224/providers/microsoft.monitor/accounts/services-j4324224", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-5cbcfdd97b-8b5k4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j4324224-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-29T13:38:02Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j4324224-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.28:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4324224/providers/microsoft.monitor/accounts/services-j4324224", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-5cbcfdd97b-8b5k4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j4324224-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert KustoLogsDataStale does not fire	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-29 21:49	PR #7170	2105052482278264832
2026-09-29 11:13	PR #7001	2104892283974324224

private ingress should not be reachable from outside the VNet, but connection succeeded

full failure pattern:

private ingress should not be reachable from outside the VNet, but connection succeeded

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:170]: private ingress should not be reachable from outside the VNet, but connection succeeded
Expected an error to have occurred.  Got:
    <nil>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:170]: private ingress should not be reachable from outside the VNet, but connection succeeded
Expected an error to have occurred.  Got:
    <nil>: ...

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-29 21:50	PR #7244	2105052522010906624
2026-09-29 05:35	PR #6746	2104807316128796672

UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded

full failure pattern:

UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:154]: failed to scale up node pool np-update-NODES from 2 to 3 replicas
Unexpected error:
    <*fmt.wrapErrors | 0xc00308adb0>: 
    failed waiting for nodepool="np-update-NODES" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-f5zptcd4xflk" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-NODES in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-f5zptcd4xflk, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:154]: failed to scale up node pool np-update-NODES from 2 to 3 replicas
Unexpected error:
    <*fmt.wrapErrors | 0xc00308adb0>: 
    failed waiting for nodepool="np-update-NODES" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-f5zptcd4xflk" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-NODES in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-f5zptcd4xflk, error: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapErrors | 0xc0019a67e0>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-sbrs7l4xp4mk" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-sbrs7l4xp4mk, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapErrors | 0xc0019a67e0>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-sbrs7l4xp4mk" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-sbrs7l4xp4mk, error: context deadline exceeded
    ...
occurred

Contributing tests (2)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update node pool labels and taints	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update nodepool replicas and autoscaling	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-29 01:37	PR #7221	2104747373145100288
2026-09-29 01:37	PR #7215	2104747333580230656

breakglass session kubeconfig not ready: timeout waiting for session to become ready

full failure pattern:

breakglass session kubeconfig not ready: timeout waiting for session to become ready

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:249]: failed to create breakglass credentials for another user
Unexpected error:
    <*fmt.wrapError | 0xc00095e460>: 
    failed to get ready session kubeconfig from /admin/v1/hcp/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/admin-api-breakglass-rsq2rx94fvrl/providers/microsoft.redhatopenshift/hcpopenshiftclusters/sre-hcp-cluster/breakglass/breakglass-bcnzj/kubeconfig: timeout waiting for session to become ready (last status: {"status":"Session is not ready"})
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:249]: failed to create breakglass credentials for another user
Unexpected error:
    <*fmt.wrapError | 0xc00095e460>: 
    failed to get ready session kubeconfig from /admin/v1/hcp/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/admin-api-breakglass-rsq2rx94fvrl/providers/microsoft.redhatopenshift/hcpopenshiftclusters/sre-hcp-cluster/breakglass/breakglass-bcnzj/kubeconfig: timeout waiting for session to become ready (last status: {"status":"Session is not ready"})
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:249]: failed to create breakglass credentials for another user
Unexpected error:
    <*fmt.wrapError | 0xc00019f000>: 
    failed to get ready session kubeconfig from /admin/v1/hcp/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/admin-api-breakglass-xcdsf8s6jfv2/providers/microsoft.redhatopenshift/hcpopenshiftclusters/sre-hcp-cluster/breakglass/breakglass-94w82/kubeconfig: timeout waiting for session to become ready (last status: {"status":"Session is not ready"})
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:249]: failed to create breakglass credentials for another user
Unexpected error:
    <*fmt.wrapError | 0xc00019f000>: 
    failed to get ready session kubeconfig from /admin/v1/hcp/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/admin-api-breakglass-xcdsf8s6jfv2/providers/microsoft.redhatopenshift/hcpopenshiftclusters/sre-hcp-cluster/breakglass/breakglass-94w82/kubeconfig: timeout waiting for session to become ready (last status: {"status":"Session is not ready"})
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	SRE should be able to log into a cluster via a breakglass session	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-29 21:49	PR #7068	2105052413286158336
2026-09-29 16:51	PR #7170	2104977428710428672

assigning role <uuid> on /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft

full failure pattern:

assigning role <uuid> on /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:248]: Told to stop trying after 15.612s.
DNS Zone Contributor role assignment should succeed once principal propagates
assigning role befefa01-2a29-4197-83a8-272ff33ce314 on /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-c7hffp5rx8qh--managed/providers/Microsoft.Network/dnsZones/aro.ingress-cert-le.8h5b.j0435840.hcp.osadev.cloud to a626cc91-d8ba-4fcd-b448-1c0201b41a1e: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-c7hffp5rx8qh--managed/providers/Microsoft.Network/dnsZones/aro.ingress-cert-le.8h5b.j0435840.hcp.osadev.cloud/providers/Microsoft.Authorization/roleAssignments/b39c004e-40fa-570e-813c-44e71fc436b7
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: RoleAssignmentLimitExceeded
--------------------------------------------------------------------------------
{
  "error": {
    "code": "RoleAssignmentLimitExceeded",
    "message": "No more role assignments can be created."
  }
}
--------------------------------------------------------------------------------
: assigning role befefa01-2a29-4197-83a8-272ff33ce314 on /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-c7hffp5rx8qh--managed/providers/Microsoft.Network/dnsZones/aro.ingress-cert-le.8h5b.j0435840.hcp.osadev.cloud to a626cc91-d8ba-4fcd-b448-1c0201b41a1e: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-c7hffp5rx8qh--managed/providers/Microsoft.Network/dnsZones/aro.ingress-cert-le.8h5b.j0435840.hcp.osadev.cloud/providers/Microsoft.Authorization/roleAssignments/b39c004e-40fa-570e-813c-44e71fc436b7
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: RoleAssignmentLimitExceeded
--------------------------------------------------------------------------------
{
  "error": {
    "code": "RoleAssignmentLimitExceeded",
    "message": "No more role assignments can be created."
  }
}
--------------------------------------------------------------------------------

At one point, however, the function did return successfully.
Yet, Eventually failed because the matcher was not satisfied:
Expected success, but got an error:
    <*fmt.wrapError | 0xc000e58700>: 
    assigning role befefa01-2a29-4197-83a8-272ff33ce314 on /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-c7hffp5rx8qh--managed/providers/Microsoft.Network/dnsZones/aro.ingress-cert-le.8h5b.j0435840.hcp.osadev.cloud to a626cc91-d8ba-4fcd-b448-1c0201b41a1e: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-c7hffp5rx8qh--managed/providers/Microsoft.Network/dnsZones/aro.ingress-cert-le.8h5b.j0435840.hcp.osadev.cloud/providers/Microsoft.Authorization/roleAssignments/b39c004e-40fa-570e-813c-44e71fc436b7
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: PrincipalNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "PrincipalNotFound",
        "message": "Principal a626cc91d8ba4fcdb4481c0201b41a1e does not exist in the directory XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX. Check that you have the correct principal ID. If you are creating this principal and then immediately assigning a role, this error might be related to a replication delay. In this case, set the role assignment principalType property to a value, such as ServicePrincipal, User, or Group.  See https://aka.ms/docs-principaltype"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:248]: Told to stop trying after 15.612s.
DNS Zone Contributor role assignment should succeed once principal propagates
assigning role befefa01-2a29-4197-83a8-272ff33ce314 on /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-c7hffp5rx8qh--managed/providers/Microsoft.Network/dnsZones/aro.ingress-cert-le.8h5b.j0435840.hcp.osadev.cloud to a626cc91-d8ba-4fcd-b448-1c0201b41a1e: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-c7hffp5rx8qh--managed/providers/Microsoft.Network/dnsZones/aro.ingress-cert-le.8h5b.j0435840.hcp.osadev.cloud/providers/Microsoft.Authorization/roleAssignments/b39c004e-40fa-570e-813c-44e71fc436b7
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: RoleAssignmentLimitExceeded
--------------------------------------------------------------------------------
{
  "error": {
    "code": "RoleAssignmentLimitExceeded",
    "message": "No more role assignments can be created."
  }
}
--------------------------------------------------------------------------------
: assigning role befefa01-2a29-4197-83a8-272ff33ce314 on /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-c7hffp5rx8qh--managed/providers/Microsoft.Network/dnsZones/aro.ingress-cert-le.8h5b.j0435840.hcp.osadev.cloud to a626cc91-d8ba-4fcd-b448-1c0201b41a1e: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-c7hffp5rx8qh--managed/providers/Microsoft.Network/dnsZones/aro.ingress-cert-le.8h5b.j0435840.hcp.osadev.cloud/providers/Microsoft.Authorization/roleAssignments/b39c004e-40fa-570e-813c-44e71fc436b7
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: RoleAssignmentLimitExceeded
--------------------------------------------------------------------------------
{
  "error": {
    "code": "RoleAssignmentLimitExceeded",
    "message": "No more role assignments can be created."
  }
}
--------------------------------------------------------------------------------

At one point, however, the function did return successfully.
Yet, Eventually failed because the matcher was not satisfied:
Expected success, but got an error:
    <*fmt.wrapError | 0xc000e58700>: 
    assigning role befefa01-2a29-4197-83a8-272ff33ce314 on /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-c7hffp5rx8qh--managed/providers/Microsoft.Network/dnsZones/aro.ingress-cert-le.8h5b.j0435840.hcp.osadev.cloud to a626cc91-d8ba-4fcd-b448-1c0201b41a1e: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-c7hffp5rx8qh--managed/providers/Microsoft.Network/dnsZones/aro.ingress-cert-le.8h5b.j0435840.hcp.osadev.cloud/providers/Microsoft.Authorization/roleAssignments/b39c004e-40fa-570e-813c-44e71fc436b7
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: PrincipalNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "PrincipalNotFound",
        "message": "Principal a626cc91d8ba4fcdb4481c0201b41a1e does not exist in the directory XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX. Check that you have the correct principal ID. If you are creating this principal and then immediately assigning a role, this error might be related to a replication delay. In this case, set the role assignment principalType property to a value, such as ServicePrincipal, User, or Group.  See https://aka.ms/docs-principaltype"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:248]: Told to stop trying after 15.676s.
DNS Zone Contributor role assignment should succeed once principal propagates
assigning role befefa01-2a29-4197-83a8-272ff33ce314 on /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-rpc5rk6cx9j4--managed/providers/Microsoft.Network/dnsZones/aro.ingress-cert-le.9hzx.j0230656.hcp.osadev.cloud to c1778d09-4e30-43d3-a6e5-c12816d71c41: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-rpc5rk6cx9j4--managed/providers/Microsoft.Network/dnsZones/aro.ingress-cert-le.9hzx.j0230656.hcp.osadev.cloud/providers/Microsoft.Authorization/roleAssignments/18eec22c-5874-5077-bd44-80689d165720
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: RoleAssignmentLimitExceeded
--------------------------------------------------------------------------------
{
  "error": {
    "code": "RoleAssignmentLimitExceeded",
    "message": "No more role assignments can be created."
  }
}
--------------------------------------------------------------------------------
: assigning role befefa01-2a29-4197-83a8-272ff33ce314 on /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-rpc5rk6cx9j4--managed/providers/Microsoft.Network/dnsZones/aro.ingress-cert-le.9hzx.j0230656.hcp.osadev.cloud to c1778d09-4e30-43d3-a6e5-c12816d71c41: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-rpc5rk6cx9j4--managed/providers/Microsoft.Network/dnsZones/aro.ingress-cert-le.9hzx.j0230656.hcp.osadev.cloud/providers/Microsoft.Authorization/roleAssignments/18eec22c-5874-5077-bd44-80689d165720
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: RoleAssignmentLimitExceeded
--------------------------------------------------------------------------------
{
  "error": {
    "code": "RoleAssignmentLimitExceeded",
    "message": "No more role assignments can be created."
  }
}
--------------------------------------------------------------------------------

At one point, however, the function did return successfully.
Yet, Eventually failed because the matcher was not satisfied:
Expected success, but got an error:
    <*fmt.wrapError | 0xc000ebe2a0>: 
    assigning role befefa01-2a29-4197-83a8-272ff33ce314 on /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-rpc5rk6cx9j4--managed/providers/Microsoft.Network/dnsZones/aro.ingress-cert-le.9hzx.j0230656.hcp.osadev.cloud to c1778d09-4e30-43d3-a6e5-c12816d71c41: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-rpc5rk6cx9j4--managed/providers/Microsoft.Network/dnsZones/aro.ingress-cert-le.9hzx.j0230656.hcp.osadev.cloud/providers/Microsoft.Authorization/roleAssignments/18eec22c-5874-5077-bd44-80689d165720
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: PrincipalNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "PrincipalNotFound",
        "message": "Principal c1778d094e3043d3a6e5c12816d71c41 does not exist in the directory XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX. Check that you have the correct principal ID. If you are creating this principal and then immediately assigning a role, this error might be related to a replication delay. In this case, set the role assignment principalType property to a value, such as ServicePrincipal, User, or Group.  See https://aka.ms/docs-principaltype"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:248]: Told to stop trying after 15.676s.
DNS Zone Contributor role assignment should succeed once principal propagates
assigning role befefa01-2a29-4197-83a8-272ff33ce314 on /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-rpc5rk6cx9j4--managed/providers/Microsoft.Network/dnsZones/aro.ingress-cert-le.9hzx.j0230656.hcp.osadev.cloud to c1778d09-4e30-43d3-a6e5-c12816d71c41: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-rpc5rk6cx9j4--managed/providers/Microsoft.Network/dnsZones/aro.ingress-cert-le.9hzx.j0230656.hcp.osadev.cloud/providers/Microsoft.Authorization/roleAssignments/18eec22c-5874-5077-bd44-80689d165720
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: RoleAssignmentLimitExceeded
--------------------------------------------------------------------------------
{
  "error": {
    "code": "RoleAssignmentLimitExceeded",
    "message": "No more role assignments can be created."
  }
}
--------------------------------------------------------------------------------
: assigning role befefa01-2a29-4197-83a8-272ff33ce314 on /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-rpc5rk6cx9j4--managed/providers/Microsoft.Network/dnsZones/aro.ingress-cert-le.9hzx.j0230656.hcp.osadev.cloud to c1778d09-4e30-43d3-a6e5-c12816d71c41: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-rpc5rk6cx9j4--managed/providers/Microsoft.Network/dnsZones/aro.ingress-cert-le.9hzx.j0230656.hcp.osadev.cloud/providers/Microsoft.Authorization/roleAssignments/18eec22c-5874-5077-bd44-80689d165720
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: RoleAssignmentLimitExceeded
--------------------------------------------------------------------------------
{
  "error": {
    "code": "RoleAssignmentLimitExceeded",
    "message": "No more role assignments can be created."
  }
}
--------------------------------------------------------------------------------

At one point, however, the function did return successfully.
Yet, Eventually failed because the matcher was not satisfied:
Expected success, but got an error:
    <*fmt.wrapError | 0xc000ebe2a0>: 
    assigning role befefa01-2a29-4197-83a8-272ff33ce314 on /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-rpc5rk6cx9j4--managed/providers/Microsoft.Network/dnsZones/aro.ingress-cert-le.9hzx.j0230656.hcp.osadev.cloud to c1778d09-4e30-43d3-a6e5-c12816d71c41: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-rpc5rk6cx9j4--managed/providers/Microsoft.Network/dnsZones/aro.ingress-cert-le.9hzx.j0230656.hcp.osadev.cloud/providers/Microsoft.Authorization/roleAssignments/18eec22c-5874-5077-bd44-80689d165720
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: PrincipalNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "PrincipalNotFound",
        "message": "Principal c1778d094e3043d3a6e5c12816d71c41 does not exist in the directory XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX. Check that you have the correct principal ID. If you are creating this principal and then immediately assigning a role, this error might be related to a replication delay. In this case, set the role assignment principalType property to a value, such as ServicePrincipal, User, or Group.  See https://aka.ms/docs-principaltype"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-09-29 01:37	PR #7224	2104747427830435840
2026-09-29 01:37	PR #7215	2104747333580230656

ERROR CODE: InvalidResource; provider Microsoft.ManagedIdentity

full failure pattern:

ERROR CODE: InvalidResource; provider Microsoft.ManagedIdentity

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0xc0010386e0>: 
    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-7vndfwh8dc5z" to finish creating: GET https://rp.j1689472.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c0cfb255-b550-40af-8f39-679f917ffd5f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidResource
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c0cfb255-b550-40af-8f39-679f917ffd5f",
      "name": "c0cfb255-b550-40af-8f39-679f917ffd5f",
      "status": "Failed",
      "startTime": "2026-09-29T10:00:52.42060332Z",
      "endTime": "2026-09-29T10:15:11.349723956Z",
      "error": {
        "code": "InvalidResource",
        "message": "DataPlaneIdentitiesPermissionsValidation: MissingRequiredPermissions: Data plane operators missing required permissions: [{\"resource\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-7vndfwh8dc5z/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\",\"identity\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard1-00-58/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver\",\"decisions\":[{\"actionId\":\"Microsoft.Network/networkSecurityGroups/join/action\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"}]},{\"resource\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-7vndfwh8dc5z/providers/Microsoft.Network/virtualNetworks/customer-vnet-name\",\"identity\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard1-00-58/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver\",\"decisions\":[{\"actionId\":\"Microsoft.Network/virtualNetworks/join/action\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"}]},{\"resource\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-7vndfwh8dc5z/providers/Microsoft.Network/virtualNetworks/customer-vnet-name/subnets/customer-vnet-subnet1\",\"identity\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard1-00-58/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver\",\"decisions\":[{\"actionId\":\"Microsoft.Network/virtualNetworks/subnets/read\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"},{\"actionId\":\"Microsoft.Network/virtualNetworks/subnets/write\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"},{\"actionId\":\"Microsoft.Network/virtualNetworks/subnets/join/action\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"}]}]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0xc0010386e0>: 
    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-7vndfwh8dc5z" to finish creating: GET https://rp.j1689472.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c0cfb255-b550-40af-8f39-679f917ffd5f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidResource
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c0cfb255-b550-40af-8f39-679f917ffd5f",
      "name": "c0cfb255-b550-40af-8f39-679f917ffd5f",
      "status": "Failed",
      "startTime": "2026-09-29T10:00:52.42060332Z",
      "endTime": "2026-09-29T10:15:11.349723956Z",
      "error": {
        "code": "InvalidResource",
        "message": "DataPlaneIdentitiesPermissionsValidation: MissingRequiredPermissions: Data plane operators missing required permissions: [{\"resource\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-7vndfwh8dc5z/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\",\"identity\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard1-00-58/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver\",\"decisions\":[{\"actionId\":\"Microsoft.Network/networkSecurityGroups/join/action\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"}]},{\"resource\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-7vndfwh8dc5z/providers/Microsoft.Network/virtualNetworks/customer-vnet-name\",\"identity\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard1-00-58/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver\",\"decisions\":[{\"actionId\":\"Microsoft.Network/virtualNetworks/join/action\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"}]},{\"resource\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-7vndfwh8dc5z/providers/Microsoft.Network/virtualNetworks/customer-vnet-name/subnets/customer-vnet-subnet1\",\"identity\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard1-00-58/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver\",\"decisions\":[{\"actionId\":\"Microsoft.Network/virtualNetworks/subnets/read\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"},{\"actionId\":\"Microsoft.Network/virtualNetworks/subnets/write\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"},{\"actionId\":\"Microsoft.Network/virtualNetworks/subnets/join/action\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"}]}]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-hzw8jglf845v/cluster-candidate-4-21-779mcz should provision
Unexpected error:
    <*fmt.wrapError | 0xc0008bc800>: 
    failed to create HCP cluster cluster-candidate-4-21-779mcz: failed waiting for cluster="cluster-candidate-4-21-779mcz" in resourcegroup="rg-candidate-4-21-hzw8jglf845v" to finish creating: GET https://rp.j1689472.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/650e9430-8570-4b8f-b095-fa3e61539c0e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidResource
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/650e9430-8570-4b8f-b095-fa3e61539c0e",
      "name": "650e9430-8570-4b8f-b095-fa3e61539c0e",
      "status": "Failed",
      "startTime": "2026-09-29T09:59:50.884256802Z",
      "endTime": "2026-09-29T10:12:21.595783509Z",
      "error": {
        "code": "InvalidResource",
        "message": "DataPlaneIdentitiesPermissionsValidation: MissingRequiredPermissions: Data plane operators missing required permissions: [{\"resource\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-hzw8jglf845v/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-candidate-779mcz\",\"identity\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard1-00-30/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver\",\"decisions\":[{\"actionId\":\"Microsoft.Network/networkSecurityGroups/join/action\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"}]},{\"resource\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-hzw8jglf845v/providers/Microsoft.Network/virtualNetworks/customer-vnet-candidate-779mcz\",\"identity\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard1-00-30/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver\",\"decisions\":[{\"actionId\":\"Microsoft.Network/virtualNetworks/join/action\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"}]},{\"resource\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-hzw8jglf845v/providers/Microsoft.Network/virtualNetworks/customer-vnet-candidate-779mcz/subnets/customer-vnet-subnet-candidate-779mcz\",\"identity\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard1-00-30/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver\",\"decisions\":[{\"actionId\":\"Microsoft.Network/virtualNetworks/subnets/read\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"},{\"actionId\":\"Microsoft.Network/virtualNetworks/subnets/write\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"},{\"actionId\":\"Microsoft.Network/virtualNetworks/subnets/join/action\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"}]}]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-hzw8jglf845v/cluster-candidate-4-21-779mcz should provision
Unexpected error:
    <*fmt.wrapError | 0xc0008bc800>: 
    failed to create HCP cluster cluster-candidate-4-21-779mcz: failed waiting for cluster="cluster-candidate-4-21-779mcz" in resourcegroup="rg-candidate-4-21-hzw8jglf845v" to finish creating: GET https://rp.j1689472.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/650e9430-8570-4b8f-b095-fa3e61539c0e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidResource
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/650e9430-8570-4b8f-b095-fa3e61539c0e",
      "name": "650e9430-8570-4b8f-b095-fa3e61539c0e",
      "status": "Failed",
      "startTime": "2026-09-29T09:59:50.884256802Z",
      "endTime": "2026-09-29T10:12:21.595783509Z",
      "error": {
        "code": "InvalidResource",
        "message": "DataPlaneIdentitiesPermissionsValidation: MissingRequiredPermissions: Data plane operators missing required permissions: [{\"resource\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-hzw8jglf845v/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-candidate-779mcz\",\"identity\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard1-00-30/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver\",\"decisions\":[{\"actionId\":\"Microsoft.Network/networkSecurityGroups/join/action\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"}]},{\"resource\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-hzw8jglf845v/providers/Microsoft.Network/virtualNetworks/customer-vnet-candidate-779mcz\",\"identity\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard1-00-30/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver\",\"decisions\":[{\"actionId\":\"Microsoft.Network/virtualNetworks/join/action\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"}]},{\"resource\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-hzw8jglf845v/providers/Microsoft.Network/virtualNetworks/customer-vnet-candidate-779mcz/subnets/customer-vnet-subnet-candidate-779mcz\",\"identity\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard1-00-30/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver\",\"decisions\":[{\"actionId\":\"Microsoft.Network/virtualNetworks/subnets/read\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"},{\"actionId\":\"Microsoft.Network/virtualNetworks/subnets/write\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"},{\"actionId\":\"Microsoft.Network/virtualNetworks/subnets/join/action\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"}]}]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:303]: failed to create HCP cluster np-noedge-cxd8l4
Unexpected error:
    <*fmt.wrapError | 0xc00123e500>: 
    failed to create HCP cluster np-noedge-cxd8l4: failed waiting for cluster="np-noedge-cxd8l4" in resourcegroup="rg-np-noedge-cxd8l4-h9t6xzrtxdvv" to finish creating: GET https://rp.j1689472.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3b33b6ae-0be6-4497-a86f-67ea2a4de210
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidResource
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3b33b6ae-0be6-4497-a86f-67ea2a4de210",
      "name": "3b33b6ae-0be6-4497-a86f-67ea2a4de210",
      "status": "Failed",
      "startTime": "2026-09-29T10:26:32.054517719Z",
      "endTime": "2026-09-29T10:37:32.806465773Z",
      "error": {
        "code": "InvalidResource",
        "message": "DataPlaneIdentitiesPermissionsValidation: MissingRequiredPermissions: Data plane operators missing required permissions: [{\"resource\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-cxd8l4-h9t6xzrtxdvv/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-np-noedge-cxd8l4\",\"identity\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard1-00-37/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver\",\"decisions\":[{\"actionId\":\"Microsoft.Network/networkSecurityGroups/join/action\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"}]},{\"resource\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-cxd8l4-h9t6xzrtxdvv/providers/Microsoft.Network/virtualNetworks/customer-vnet-np-noedge-cxd8l4\",\"identity\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard1-00-37/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver\",\"decisions\":[{\"actionId\":\"Microsoft.Network/virtualNetworks/join/action\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"}]},{\"resource\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-cxd8l4-h9t6xzrtxdvv/providers/Microsoft.Network/virtualNetworks/customer-vnet-np-noedge-cxd8l4/subnets/customer-vnet-subnet-np-noedge-cxd8l4\",\"identity\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard1-00-37/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver\",\"decisions\":[{\"actionId\":\"Microsoft.Network/virtualNetworks/subnets/join/action\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"},{\"actionId\":\"Microsoft.Network/virtualNetworks/subnets/read\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"},{\"actionId\":\"Microsoft.Network/virtualNetworks/subnets/write\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"}]}]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:303]: failed to create HCP cluster np-noedge-cxd8l4
Unexpected error:
    <*fmt.wrapError | 0xc00123e500>: 
    failed to create HCP cluster np-noedge-cxd8l4: failed waiting for cluster="np-noedge-cxd8l4" in resourcegroup="rg-np-noedge-cxd8l4-h9t6xzrtxdvv" to finish creating: GET https://rp.j1689472.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3b33b6ae-0be6-4497-a86f-67ea2a4de210
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidResource
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3b33b6ae-0be6-4497-a86f-67ea2a4de210",
      "name": "3b33b6ae-0be6-4497-a86f-67ea2a4de210",
      "status": "Failed",
      "startTime": "2026-09-29T10:26:32.054517719Z",
      "endTime": "2026-09-29T10:37:32.806465773Z",
      "error": {
        "code": "InvalidResource",
        "message": "DataPlaneIdentitiesPermissionsValidation: MissingRequiredPermissions: Data plane operators missing required permissions: [{\"resource\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-cxd8l4-h9t6xzrtxdvv/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-np-noedge-cxd8l4\",\"identity\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard1-00-37/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver\",\"decisions\":[{\"actionId\":\"Microsoft.Network/networkSecurityGroups/join/action\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"}]},{\"resource\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-cxd8l4-h9t6xzrtxdvv/providers/Microsoft.Network/virtualNetworks/customer-vnet-np-noedge-cxd8l4\",\"identity\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard1-00-37/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver\",\"decisions\":[{\"actionId\":\"Microsoft.Network/virtualNetworks/join/action\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"}]},{\"resource\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-cxd8l4-h9t6xzrtxdvv/providers/Microsoft.Network/virtualNetworks/customer-vnet-np-noedge-cxd8l4/subnets/customer-vnet-subnet-np-noedge-cxd8l4\",\"identity\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard1-00-37/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver\",\"decisions\":[{\"actionId\":\"Microsoft.Network/virtualNetworks/subnets/join/action\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"},{\"actionId\":\"Microsoft.Network/virtualNetworks/subnets/read\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"},{\"actionId\":\"Microsoft.Network/virtualNetworks/subnets/write\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"}]}]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (63)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.1	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to cordon, drain, and uncordon a node in an HCP cluster	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a HCP cluster and use cilium CNI plugin	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with ACR pull via managed identity and pull from a private ACR	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and custom node pool osDisk size	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage ImageDigestMirrors	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with Image Registry not present	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create node pools with ARM64-based VMs	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to delete a nodepool	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to lifecycle and confirm external auth on a cluster	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to retrieve cluster and nodepool status active versions	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to rotate KMS key for a cluster with version >= 4.22	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update node pool labels and taints	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update nodepool replicas and autoscaling	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to use workload identity via the cluster OIDC issuer URL	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster using v20260901preview API and verify cluster health	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should not be able to perform various invalid operations on cluster resources	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should not be able to reuse subnets and NSGs between clusters	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should provision Azure Disk and Azure File CSI volumes and run a pod against each	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should respect cluster-wide node limits with nodepool autoscaling	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering want metrics from HCP to be present in Azure Monitor Workspace	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	SRE should be able to log into a cluster via a breakglass session	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	SRE should be able to retrieve serial console logs for a VM	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	SRE should return 409 when boot diagnostics is disabled on a VM	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.23	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.0	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-29 09:05	PR #7073	2104860015251689472

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000bfa500>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-s6psdk49m28r" to finish creating: GET https://rp.j5787776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/edddc05b-21c8-4d1e-832c-40b942f545e9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/edddc05b-21c8-4d1e-832c-40b942f545e9",
      "name": "edddc05b-21c8-4d1e-832c-40b942f545e9",
      "status": "Failed",
      "startTime": "2026-09-29T02:50:21.626128144Z",
      "endTime": "2026-09-29T03:19:23.597463388Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-node-tuning-operator, ingress-operator, redhat-operators-catalog, redhat-marketplace-catalog, hosted-cluster-config-operator, csi-snapshot-controller-operator, packageserver, cluster-version-operator, openshift-route-controller-manager, cluster-image-registry-operator, openshift-controller-manager, cluster-storage-operator, olm-operator, certified-operators-catalog, dns-operator, cluster-policy-controller, catalog-operator, community-operators-catalog, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000bfa500>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-s6psdk49m28r" to finish creating: GET https://rp.j5787776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/edddc05b-21c8-4d1e-832c-40b942f545e9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/edddc05b-21c8-4d1e-832c-40b942f545e9",
      "name": "edddc05b-21c8-4d1e-832c-40b942f545e9",
      "status": "Failed",
      "startTime": "2026-09-29T02:50:21.626128144Z",
      "endTime": "2026-09-29T03:19:23.597463388Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-node-tuning-operator, ingress-operator, redhat-operators-catalog, redhat-marketplace-catalog, hosted-cluster-config-operator, csi-snapshot-controller-operator, packageserver, cluster-version-operator, openshift-route-controller-manager, cluster-image-registry-operator, openshift-controller-manager, cluster-storage-operator, olm-operator, certified-operators-catalog, dns-operator, cluster-policy-controller, catalog-operator, community-operators-catalog, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-lw74fqklhgk4/cluster-candidate-4-21-g2bwxp should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f986a0>: 
    failed to create HCP cluster cluster-candidate-4-21-g2bwxp: failed waiting for cluster="cluster-candidate-4-21-g2bwxp" in resourcegroup="rg-candidate-4-21-lw74fqklhgk4" to finish creating: GET https://rp.j5787776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/5608e9f9-828b-4064-a2f9-7ed4a7817028
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/5608e9f9-828b-4064-a2f9-7ed4a7817028",
      "name": "5608e9f9-828b-4064-a2f9-7ed4a7817028",
      "status": "Failed",
      "startTime": "2026-09-29T02:49:24.101243473Z",
      "endTime": "2026-09-29T03:18:33.772930492Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: packageserver, cluster-version-operator, openshift-controller-manager, cluster-node-tuning-operator, catalog-operator, cluster-network-operator, dns-operator, csi-snapshot-controller-operator, cluster-image-registry-operator, olm-operator, cluster-policy-controller, redhat-operators-catalog, redhat-marketplace-catalog, certified-operators-catalog, openshift-route-controller-manager, ingress-operator, hosted-cluster-config-operator, community-operators-catalog, cluster-storage-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 2 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-lw74fqklhgk4/cluster-candidate-4-21-g2bwxp should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f986a0>: 
    failed to create HCP cluster cluster-candidate-4-21-g2bwxp: failed waiting for cluster="cluster-candidate-4-21-g2bwxp" in resourcegroup="rg-candidate-4-21-lw74fqklhgk4" to finish creating: GET https://rp.j5787776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/5608e9f9-828b-4064-a2f9-7ed4a7817028
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/5608e9f9-828b-4064-a2f9-7ed4a7817028",
      "name": "5608e9f9-828b-4064-a2f9-7ed4a7817028",
      "status": "Failed",
      "startTime": "2026-09-29T02:49:24.101243473Z",
      "endTime": "2026-09-29T03:18:33.772930492Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: packageserver, cluster-version-operator, openshift-controller-manager, cluster-node-tuning-operator, catalog-operator, cluster-network-operator, dns-operator, csi-snapshot-controller-operator, cluster-image-registry-operator, olm-operator, cluster-policy-controller, redhat-operators-catalog, redhat-marketplace-catalog, certified-operators-catalog, openshift-route-controller-manager, ingress-operator, hosted-cluster-config-operator, community-operators-catalog, cluster-storage-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 2 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (2)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-29 01:37	PR #7169	2104747271315787776

ERROR CODE: ResourceGroupNotFound; provider Microsoft.AlertsManagement

full failure pattern:

ERROR CODE: ResourceGroupNotFound; provider Microsoft.AlertsManagement

Signal:

alert

none

Full failure examples (2)

alert collection incomplete
failed to fetch hcp alert rules: failed to list prometheus rule groups: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-yt/providers/Microsoft.AlertsManagement/prometheusRuleGroups
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceGroupNotFound",
    "message": "Resource group 'hcp-underlay-ci01-yt' could not be found."
  }
}
--------------------------------------------------------------------------------

alert collection incomplete
failed to fetch svc alert rules: failed to list prometheus rule groups: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-yt/providers/Microsoft.AlertsManagement/prometheusRuleGroups
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceGroupNotFound",
    "message": "Resource group 'hcp-underlay-ci01-yt' could not be found."
  }
}
--------------------------------------------------------------------------------

Contributing tests (2)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [hcp] alert collection is complete	1
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert collection is complete	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-29 14:09	PR #6756	2104936554697854976

Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed

full failure pattern:

Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-29 09:05	PR #7073	2104860015251689472

ERROR CODE: 429; provider Microsoft.ManagedIdentity

full failure pattern:

ERROR CODE: 429; provider Microsoft.ManagedIdentity

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:219]: failed to look up CAPZ principal ID
Unexpected error:
    <*fmt.wrapError | 0xc000fccf00>: 
    failed to get CAPZ MI: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard1-01-18/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure
    --------------------------------------------------------------------------------
    RESPONSE 429: 429 Too Many Requests
    ERROR CODE: 429
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "429",
        "message": "Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 2 seconds."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:219]: failed to look up CAPZ principal ID
Unexpected error:
    <*fmt.wrapError | 0xc000fccf00>: 
    failed to get CAPZ MI: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard1-01-18/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure
    --------------------------------------------------------------------------------
    RESPONSE 429: 429 Too Many Requests
    ERROR CODE: 429
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "429",
        "message": "Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 2 seconds."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with ACR pull via managed identity and pull from a private ACR	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-29 14:11	PR #7054	2104937201925099520

VerifySimpleWebApp failed: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial...

full failure pattern:

VerifySimpleWebApp failed: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:203]: failed to verify cluster workloads and pods after autoscaling update
Unexpected error:
    <*errors.joinError | 0xc0013f83f0>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-47km4.apps.aro.z8j4o9t8b9u3i6w.b8n7.j9659648.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-47km4.apps.aro.z8j4o9t8b9u3i6w.b8n7.j9659648.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-47km4.apps.aro.z8j4o9t8b9u3i6w.b8n7.j9659648.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:203]: failed to verify cluster workloads and pods after autoscaling update
Unexpected error:
    <*errors.joinError | 0xc0013f83f0>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-47km4.apps.aro.z8j4o9t8b9u3i6w.b8n7.j9659648.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-47km4.apps.aro.z8j4o9t8b9u3i6w.b8n7.j9659648.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-47km4.apps.aro.z8j4o9t8b9u3i6w.b8n7.j9659648.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-29 01:37	PR #7020	2104747357269659648

DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline...

full failure pattern:

DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline exceeded

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-x756tn"
Unexpected error:
    <*fmt.wrapError | 0xc000b82d40>: 
    DNS for route host agnhost-e2e-sample-app-z4rfg.apps.aro.x8b7b2l9a5c4m4j.y2zb.j3483392.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-z4rfg.apps.aro.x8b7b2l9a5c4m4j.y2zb.j3483392.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-z4rfg.apps.aro.x8b7b2l9a5c4m4j.y2zb.j3483392.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-x756tn"
Unexpected error:
    <*fmt.wrapError | 0xc000b82d40>: 
    DNS for route host agnhost-e2e-sample-app-z4rfg.apps.aro.x8b7b2l9a5c4m4j.y2zb.j3483392.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-z4rfg.apps.aro.x8b7b2l9a5c4m4j.y2zb.j3483392.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-z4rfg.apps.aro.x8b7b2l9a5c4m4j.y2zb.j3483392.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-29 01:37	PR #7170	2104747285383483392

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:138]: failed to wait for second cluster "basic-hcp-cluster2" to complete creation (timeout '30.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0xc0006be1b0>: 
    GET https://rp.j5787776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c185b878-6f2e-4a60-86e4-2feff70756de
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c185b878-6f2e-4a60-86e4-2feff70756de",
      "name": "c185b878-6f2e-4a60-86e4-2feff70756de",
      "status": "Failed",
      "startTime": "2026-09-29T02:50:57.370798886Z",
      "endTime": "2026-09-29T03:20:03.878832155Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:138]: failed to wait for second cluster "basic-hcp-cluster2" to complete creation (timeout '30.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0xc0006be1b0>: 
    GET https://rp.j5787776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c185b878-6f2e-4a60-86e4-2feff70756de
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c185b878-6f2e-4a60-86e4-2feff70756de",
      "name": "c185b878-6f2e-4a60-86e4-2feff70756de",
      "status": "Failed",
      "startTime": "2026-09-29T02:50:57.370798886Z",
      "endTime": "2026-09-29T03:20:03.878832155Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-29 01:37	PR #7169	2104747271315787776

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, olm-operator, packageserver]; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0xc000c18980>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-vspgr4ltfkks" to finish creating: GET https://rp.j5787776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/81a89006-e9e6-43e0-a945-6223e2371587
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/81a89006-e9e6-43e0-a945-6223e2371587",
      "name": "81a89006-e9e6-43e0-a945-6223e2371587",
      "status": "Failed",
      "startTime": "2026-09-29T02:49:48.446308819Z",
      "endTime": "2026-09-29T03:18:53.683107891Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, ingress-operator, cluster-image-registry-operator, cluster-version-operator, hosted-cluster-config-operator, certified-operators-catalog, cluster-storage-operator, cluster-node-tuning-operator, cluster-policy-controller, catalog-operator, packageserver, redhat-operators-catalog, csi-snapshot-controller-operator, olm-operator, openshift-controller-manager, cluster-network-operator, dns-operator, community-operators-catalog, redhat-marketplace-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0xc000c18980>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-vspgr4ltfkks" to finish creating: GET https://rp.j5787776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/81a89006-e9e6-43e0-a945-6223e2371587
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/81a89006-e9e6-43e0-a945-6223e2371587",
      "name": "81a89006-e9e6-43e0-a945-6223e2371587",
      "status": "Failed",
      "startTime": "2026-09-29T02:49:48.446308819Z",
      "endTime": "2026-09-29T03:18:53.683107891Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, ingress-operator, cluster-image-registry-operator, cluster-version-operator, hosted-cluster-config-operator, certified-operators-catalog, cluster-storage-operator, cluster-node-tuning-operator, cluster-policy-controller, catalog-operator, packageserver, redhat-operators-catalog, csi-snapshot-controller-operator, olm-operator, openshift-controller-manager, cluster-network-operator, dns-operator, community-operators-catalog, redhat-marketplace-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-29 01:37	PR #7169	2104747271315787776

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator, cluster-network-operator, cluster-policy-controller, csi-snapshot-controller-operator, ingress-operator]; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-pxxjsb9mbpw2/cluster-candidate-4-22-vchxk6 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000c28520>: 
    failed to create HCP cluster cluster-candidate-4-22-vchxk6: failed waiting for cluster="cluster-candidate-4-22-vchxk6" in resourcegroup="rg-candidate-4-22-pxxjsb9mbpw2" to finish creating: GET https://rp.j5530752.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dcd43e84-3bac-4e4c-a704-8ad21694f9d5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dcd43e84-3bac-4e4c-a704-8ad21694f9d5",
      "name": "dcd43e84-3bac-4e4c-a704-8ad21694f9d5",
      "status": "Failed",
      "startTime": "2026-09-29T02:51:39.168217674Z",
      "endTime": "2026-09-29T03:20:45.495027988Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator, ingress-operator, cluster-network-operator, csi-snapshot-controller-operator; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-pxxjsb9mbpw2/cluster-candidate-4-22-vchxk6 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000c28520>: 
    failed to create HCP cluster cluster-candidate-4-22-vchxk6: failed waiting for cluster="cluster-candidate-4-22-vchxk6" in resourcegroup="rg-candidate-4-22-pxxjsb9mbpw2" to finish creating: GET https://rp.j5530752.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dcd43e84-3bac-4e4c-a704-8ad21694f9d5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dcd43e84-3bac-4e4c-a704-8ad21694f9d5",
      "name": "dcd43e84-3bac-4e4c-a704-8ad21694f9d5",
      "status": "Failed",
      "startTime": "2026-09-29T02:51:39.168217674Z",
      "endTime": "2026-09-29T03:20:45.495027988Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator, ingress-operator, cluster-network-operator, csi-snapshot-controller-operator; hosted cluster degraded: UnavailableReplicas: [cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-29 01:37	PR #6922	2104747348675530752

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-23-pjfds7" with version 4.23.0-ec.0 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc00060a5c0>: 
    failed to create HCP cluster cluster-zstream-4-23-pjfds7: failed waiting for cluster="cluster-zstream-4-23-pjfds7" in resourcegroup="rg-zstream-upgrade-4-23-2ph8ks4m6rn4" to finish creating: GET https://rp.j5787776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e4d243b0-a8bd-4697-acb0-9ef4486ad1e1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e4d243b0-a8bd-4697-acb0-9ef4486ad1e1",
      "name": "e4d243b0-a8bd-4697-acb0-9ef4486ad1e1",
      "status": "Failed",
      "startTime": "2026-09-29T02:49:21.332416072Z",
      "endTime": "2026-09-29T03:18:23.455609493Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-storage-version-migrator, catalog-operator, olm-collect-profiles, redhat-operators-catalog, openshift-route-controller-manager, cluster-policy-controller, cluster-version-operator, hosted-cluster-config-operator, dns-operator, csi-snapshot-controller-operator, control-plane-pki-operator, cluster-storage-operator, cluster-image-registry-operator, kube-scheduler, packageserver, certified-operators-catalog, kube-controller-manager, cluster-node-tuning-operator, cluster-autoscaler, openshift-controller-manager, olm-operator, community-operators-catalog, konnectivity-agent, azure-cloud-controller-manager, cluster-network-operator, machine-approver, openshift-apiserver, ingress-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-23-pjfds7" with version 4.23.0-ec.0 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc00060a5c0>: 
    failed to create HCP cluster cluster-zstream-4-23-pjfds7: failed waiting for cluster="cluster-zstream-4-23-pjfds7" in resourcegroup="rg-zstream-upgrade-4-23-2ph8ks4m6rn4" to finish creating: GET https://rp.j5787776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e4d243b0-a8bd-4697-acb0-9ef4486ad1e1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e4d243b0-a8bd-4697-acb0-9ef4486ad1e1",
      "name": "e4d243b0-a8bd-4697-acb0-9ef4486ad1e1",
      "status": "Failed",
      "startTime": "2026-09-29T02:49:21.332416072Z",
      "endTime": "2026-09-29T03:18:23.455609493Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-storage-version-migrator, catalog-operator, olm-collect-profiles, redhat-operators-catalog, openshift-route-controller-manager, cluster-policy-controller, cluster-version-operator, hosted-cluster-config-operator, dns-operator, csi-snapshot-controller-operator, control-plane-pki-operator, cluster-storage-operator, cluster-image-registry-operator, kube-scheduler, packageserver, certified-operators-catalog, kube-controller-manager, cluster-node-tuning-operator, cluster-autoscaler, openshift-controller-manager, olm-operator, community-operators-catalog, konnectivity-agent, azure-cloud-controller-manager, cluster-network-operator, machine-approver, openshift-apiserver, ingress-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.23	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-29 01:37	PR #7169	2104747271315787776

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [cluster-version-operator]; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc00105c340>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-469757bfrb2w" to finish creating: GET https://rp.j5787776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e8d80fe2-11a1-436f-9808-5ee06d64084d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e8d80fe2-11a1-436f-9808-5ee06d64084d",
      "name": "e8d80fe2-11a1-436f-9808-5ee06d64084d",
      "status": "Failed",
      "startTime": "2026-09-29T02:51:03.349640727Z",
      "endTime": "2026-09-29T03:20:03.842186988Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 48s ago; version 4.21.35 is Partial (want Completed), started 7m33s ago; version 4.21.35 is Partial (want Completed), started 11m51s ago; version 4.21.35 is Partial (want Completed), started 22m57s ago; hosted cluster degraded: UnavailableReplicas: cluster-version-operator deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc00105c340>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-469757bfrb2w" to finish creating: GET https://rp.j5787776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e8d80fe2-11a1-436f-9808-5ee06d64084d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e8d80fe2-11a1-436f-9808-5ee06d64084d",
      "name": "e8d80fe2-11a1-436f-9808-5ee06d64084d",
      "status": "Failed",
      "startTime": "2026-09-29T02:51:03.349640727Z",
      "endTime": "2026-09-29T03:20:03.842186988Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 48s ago; version 4.21.35 is Partial (want Completed), started 7m33s ago; version 4.21.35 is Partial (want Completed), started 11m51s ago; version 4.21.35 is Partial (want Completed), started 22m57s ago; hosted cluster degraded: UnavailableReplicas: cluster-version-operator deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster and node pools with aggregated advanced features	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-29 01:37	PR #7169	2104747271315787776

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [kube-controller-manager]; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:89]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc00075e620>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-lzq55sh9jmq4" to finish creating: GET https://rp.j5530752.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/772d74f8-28ec-4061-981a-203cb9d2341a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/772d74f8-28ec-4061-981a-203cb9d2341a",
      "name": "772d74f8-28ec-4061-981a-203cb9d2341a",
      "status": "Failed",
      "startTime": "2026-09-29T02:53:30.135245133Z",
      "endTime": "2026-09-29T03:22:32.693389266Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 5m0s ago; hosted cluster degraded: UnavailableReplicas: kube-controller-manager deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:89]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc00075e620>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-lzq55sh9jmq4" to finish creating: GET https://rp.j5530752.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/772d74f8-28ec-4061-981a-203cb9d2341a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/772d74f8-28ec-4061-981a-203cb9d2341a",
      "name": "772d74f8-28ec-4061-981a-203cb9d2341a",
      "status": "Failed",
      "startTime": "2026-09-29T02:53:30.135245133Z",
      "endTime": "2026-09-29T03:22:32.693389266Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 5m0s ago; hosted cluster degraded: UnavailableReplicas: kube-controller-manager deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to retrieve cluster and nodepool status active versions	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-29 01:37	PR #6922	2104747348675530752

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...

full failure pattern:

ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-23-gsz2fzdqj9cx/cluster-candidate-4-23-t8dbk2 should provision
Unexpected error:
    <*fmt.wrapError | 0xc0012024a0>: 
    failed to create HCP cluster cluster-candidate-4-23-t8dbk2: failed waiting for cluster="cluster-candidate-4-23-t8dbk2" in resourcegroup="rg-candidate-4-23-gsz2fzdqj9cx" to finish creating: GET https://rp.j5787776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f4f996b9-ba98-44c4-95fb-3d754f91c4bd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f4f996b9-ba98-44c4-95fb-3d754f91c4bd",
      "name": "f4f996b9-ba98-44c4-95fb-3d754f91c4bd",
      "status": "Failed",
      "startTime": "2026-09-29T02:56:27.709492276Z",
      "endTime": "2026-09-29T03:25:33.644090026Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.23.0-ec.1 is Partial (want Completed), started 24m4s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-23-gsz2fzdqj9cx/cluster-candidate-4-23-t8dbk2 should provision
Unexpected error:
    <*fmt.wrapError | 0xc0012024a0>: 
    failed to create HCP cluster cluster-candidate-4-23-t8dbk2: failed waiting for cluster="cluster-candidate-4-23-t8dbk2" in resourcegroup="rg-candidate-4-23-gsz2fzdqj9cx" to finish creating: GET https://rp.j5787776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f4f996b9-ba98-44c4-95fb-3d754f91c4bd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f4f996b9-ba98-44c4-95fb-3d754f91c4bd",
      "name": "f4f996b9-ba98-44c4-95fb-3d754f91c4bd",
      "status": "Failed",
      "startTime": "2026-09-29T02:56:27.709492276Z",
      "endTime": "2026-09-29T03:25:33.644090026Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.23.0-ec.1 is Partial (want Completed), started 24m4s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-29 01:37	PR #7169	2104747271315787776

ERROR CODE: ResourceGroupNotFound; provider Microsoft.Insights

full failure pattern:

ERROR CODE: ResourceGroupNotFound; provider Microsoft.Insights

Signal:

alert

none

Full failure examples (1)

alert collection incomplete
failed to fetch metric alert rules for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-yt: failed to list metric alert rules: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-yt/providers/Microsoft.Insights/metricAlerts
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceGroupNotFound",
    "message": "Resource group 'hcp-underlay-ci01-yt' could not be found."
  }
}
--------------------------------------------------------------------------------

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [infra] alert collection is complete	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-29 14:09	PR #6756	2104936554697854976

VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5

full failure pattern:

VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.004s.
Expected success, but got an error:
    <*errors.joinError | 0xc00180d170>: 
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.004s.
Expected success, but got an error:
    <*errors.joinError | 0xc00180d170>: 
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    ...

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-29 15:16	PR #7244	2104953578396323840

alert [svc] userJourneyClusterProvisionErrors6h30m fired

full failure pattern:

alert [svc] userJourneyClusterProvisionErrors6h30m fired

Signal:

alert

none

Full failure examples (1)

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-29T10:44:18Z
  Severity:    Sev3
  Labels:      alertname="userJourneyClusterProvisionErrors6h30m", cluster="ci01-j1689472-svc", component="slo", long_window="6h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1689472/providers/microsoft.monitor/accounts/services-j1689472", region="westus3", severity="3", short_window="30m", slo="cluster-provision-errors"
  Description: More than 30% of completed cluster create (install) operations on ci01-j1689472-svc failed over the last 6 hours (at least 5 completions), a 6x burn of the 95% SLO budget.

Contributing tests (1)

Failed At	Job	Test	Support
alert	pull-ci-Azure-ARO-HCP-main-e2e-parallel	[aro-hcp-observability] [svc] alert userJourneyClusterProvisionErrors6h30m does not fire	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-29 09:05	PR #7073	2104860015251689472

failed to roll out Helm release: resource Deployment/monitoring/kube-events not ready. status: InProgress, message: Available: 0/1 context deadline exceeded

full failure pattern:

failed to roll out Helm release: resource Deployment/monitoring/kube-events not ready. status: InProgress, message: Available: 0/1 context deadline exceeded

Signal:

provision

none

Full failure examples (1)

time=2026-09-29T20:43:19.784Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=kube-events description="Step kube-events\n Kind: Helm\n"
time=2026-09-29T20:43:20.131Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=kube-events values="map[enabled:true image:map[digest:sha256:54b14654077ec8c910ff0eeeda191bb45421c6496b2211751fbfd13b38779a46 registry:kubernetesshared.azurecr.io repository:shared/kube-events] resources:map[limits:map[memory:unlimited] requests:map[cpu:200m memory:512Mi]]]"
time=2026-09-29T20:43:20.131Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=kube-events
time=2026-09-29T20:43:20.307Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=kube-events namespace=monitoring
time=2026-09-29T20:43:20.477Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=kube-events dryRun=false
time=2026-09-29T20:43:20.477Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=kube-events
time=2026-09-29T20:43:20.513Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=kube-events
time=2026-09-29T20:48:23.298Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=kube-events err="resource Deployment/monitoring/kube-events not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded"
time=2026-09-29T20:48:23.298Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=kube-events err="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource Deployment/monitoring/kube-events not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/kube-events	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-29 20:00	PR #7231	2105029025452789760

failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: context deadline exceeded

full failure pattern:

failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: context deadline exceeded

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:198]: failed to create node pool version 4.20.32 for back-level cluster
Unexpected error:
    <*fmt.wrapError | 0xc00090ce00>: 
    failed waiting for nodepool="np-ver-4-20-32" for cluster "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-26c6r9w89qkm" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:198]: failed to create node pool version 4.20.32 for back-level cluster
Unexpected error:
    <*fmt.wrapError | 0xc00090ce00>: 
    failed waiting for nodepool="np-ver-4-20-32" for cluster "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-26c6r9w89qkm" to finish creating: context deadline exceeded
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with back-level version 4.20	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-29 11:13	PR #7001	2104892283974324224

Environment: INT

Failure patterns: 2 · Failures matched: 7 · Total runs: 6 · Runs affected: 4

Failure pattern

Failed at

Trend

Also in

ERROR CODE: DeadlineExceeded; provider Microsoft.Compute

full failure pattern:

ERROR CODE: DeadlineExceeded; provider Microsoft.Compute

Signal:

e2e

DEV, PROD, STG

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc00134a240>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-lf58x6mrt8sn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6899b842-6be7-4752-9378-45ead2f46f38
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6899b842-6be7-4752-9378-45ead2f46f38",
      "name": "6899b842-6be7-4752-9378-45ead2f46f38",
      "status": "Failed",
      "startTime": "2026-09-29T22:48:49.695468888Z",
      "endTime": "2026-09-29T23:07:57.514241775Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine j1d5g5c7r6t3f3c-autoscale-noaz-hllff-7t674: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-lf58x6mrt8sn--managed/providers/Microsoft.Compute/virtualMachines/j1d5g5c7r6t3f3c-autoscale-noaz-hllff-7t674 to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine j1d5g5c7r6t3f3c-autoscale-noaz-hllff-7t674: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-lf58x6mrt8sn--managed/providers/Microsoft.Compute/virtualMachines/j1d5g5c7r6t3f3c-autoscale-noaz-hllff-7t674 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc00134a240>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-lf58x6mrt8sn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6899b842-6be7-4752-9378-45ead2f46f38
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6899b842-6be7-4752-9378-45ead2f46f38",
      "name": "6899b842-6be7-4752-9378-45ead2f46f38",
      "status": "Failed",
      "startTime": "2026-09-29T22:48:49.695468888Z",
      "endTime": "2026-09-29T23:07:57.514241775Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine j1d5g5c7r6t3f3c-autoscale-noaz-hllff-7t674: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-lf58x6mrt8sn--managed/providers/Microsoft.Compute/virtualMachines/j1d5g5c7r6t3f3c-autoscale-noaz-hllff-7t674 to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine j1d5g5c7r6t3f3c-autoscale-noaz-hllff-7t674: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-lf58x6mrt8sn--managed/providers/Microsoft.Compute/virtualMachines/j1d5g5c7r6t3f3c-autoscale-noaz-hllff-7t674 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:139]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0010311c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-zbbkqjx7fwrv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/15a80b6b-fe14-4fa6-a826-e605868e189f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/15a80b6b-fe14-4fa6-a826-e605868e189f",
      "name": "15a80b6b-fe14-4fa6-a826-e605868e189f",
      "status": "Failed",
      "startTime": "2026-09-29T21:44:03.296557Z",
      "endTime": "2026-09-29T22:03:11.679917905Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine e3b7f3p1b7r7k1s-np-1-djfk4-kclwz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-zbbkqjx7fwrv--managed/providers/Microsoft.Compute/virtualMachines/e3b7f3p1b7r7k1s-np-1-djfk4-kclwz to exist\nMachine e3b7f3p1b7r7k1s-np-1-djfk4-t2vnm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-zbbkqjx7fwrv--managed/providers/Microsoft.Compute/virtualMachines/e3b7f3p1b7r7k1s-np-1-djfk4-t2vnm to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine e3b7f3p1b7r7k1s-np-1-djfk4-kclwz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-zbbkqjx7fwrv--managed/providers/Microsoft.Compute/virtualMachines/e3b7f3p1b7r7k1s-np-1-djfk4-kclwz to exist\nMachine e3b7f3p1b7r7k1s-np-1-djfk4-t2vnm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-zbbkqjx7fwrv--managed/providers/Microsoft.Compute/virtualMachines/e3b7f3p1b7r7k1s-np-1-djfk4-t2vnm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:139]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0010311c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-zbbkqjx7fwrv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/15a80b6b-fe14-4fa6-a826-e605868e189f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/15a80b6b-fe14-4fa6-a826-e605868e189f",
      "name": "15a80b6b-fe14-4fa6-a826-e605868e189f",
      "status": "Failed",
      "startTime": "2026-09-29T21:44:03.296557Z",
      "endTime": "2026-09-29T22:03:11.679917905Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine e3b7f3p1b7r7k1s-np-1-djfk4-kclwz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-zbbkqjx7fwrv--managed/providers/Microsoft.Compute/virtualMachines/e3b7f3p1b7r7k1s-np-1-djfk4-kclwz to exist\nMachine e3b7f3p1b7r7k1s-np-1-djfk4-t2vnm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-zbbkqjx7fwrv--managed/providers/Microsoft.Compute/virtualMachines/e3b7f3p1b7r7k1s-np-1-djfk4-t2vnm to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine e3b7f3p1b7r7k1s-np-1-djfk4-kclwz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-zbbkqjx7fwrv--managed/providers/Microsoft.Compute/virtualMachines/e3b7f3p1b7r7k1s-np-1-djfk4-kclwz to exist\nMachine e3b7f3p1b7r7k1s-np-1-djfk4-t2vnm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-zbbkqjx7fwrv--managed/providers/Microsoft.Compute/virtualMachines/e3b7f3p1b7r7k1s-np-1-djfk4-t2vnm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:222]: failed to create nodepool "np-1" for authorized CIDRs cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f32080>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-thvwrhnx9f2p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa62eeea-3561-4f5f-872a-af727c8e3ec4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa62eeea-3561-4f5f-872a-af727c8e3ec4",
      "name": "aa62eeea-3561-4f5f-872a-af727c8e3ec4",
      "status": "Failed",
      "startTime": "2026-09-29T16:37:13.215287875Z",
      "endTime": "2026-09-29T16:56:16.225730373Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine e2v2e3f8a5f0f0j-np-1-pksnh-jmfl5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-thvwrhnx9f2p--managed/providers/Microsoft.Compute/virtualMachines/e2v2e3f8a5f0f0j-np-1-pksnh-jmfl5 to exist\nMachine e2v2e3f8a5f0f0j-np-1-pksnh-vpvtc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-thvwrhnx9f2p--managed/providers/Microsoft.Compute/virtualMachines/e2v2e3f8a5f0f0j-np-1-pksnh-vpvtc to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine e2v2e3f8a5f0f0j-np-1-pksnh-jmfl5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-thvwrhnx9f2p--managed/providers/Microsoft.Compute/virtualMachines/e2v2e3f8a5f0f0j-np-1-pksnh-jmfl5 to exist\nMachine e2v2e3f8a5f0f0j-np-1-pksnh-vpvtc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-thvwrhnx9f2p--managed/providers/Microsoft.Compute/virtualMachines/e2v2e3f8a5f0f0j-np-1-pksnh-vpvtc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:222]: failed to create nodepool "np-1" for authorized CIDRs cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f32080>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-thvwrhnx9f2p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa62eeea-3561-4f5f-872a-af727c8e3ec4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa62eeea-3561-4f5f-872a-af727c8e3ec4",
      "name": "aa62eeea-3561-4f5f-872a-af727c8e3ec4",
      "status": "Failed",
      "startTime": "2026-09-29T16:37:13.215287875Z",
      "endTime": "2026-09-29T16:56:16.225730373Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine e2v2e3f8a5f0f0j-np-1-pksnh-jmfl5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-thvwrhnx9f2p--managed/providers/Microsoft.Compute/virtualMachines/e2v2e3f8a5f0f0j-np-1-pksnh-jmfl5 to exist\nMachine e2v2e3f8a5f0f0j-np-1-pksnh-vpvtc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-thvwrhnx9f2p--managed/providers/Microsoft.Compute/virtualMachines/e2v2e3f8a5f0f0j-np-1-pksnh-vpvtc to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine e2v2e3f8a5f0f0j-np-1-pksnh-jmfl5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-thvwrhnx9f2p--managed/providers/Microsoft.Compute/virtualMachines/e2v2e3f8a5f0f0j-np-1-pksnh-jmfl5 to exist\nMachine e2v2e3f8a5f0f0j-np-1-pksnh-vpvtc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-thvwrhnx9f2p--managed/providers/Microsoft.Compute/virtualMachines/e2v2e3f8a5f0f0j-np-1-pksnh-vpvtc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (4)

Failed At	Job	Test	Support
e2e	branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel	Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel	Customer should be able to create an HCP cluster then delete it by deleting the customer resource group	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel	Customer should create an HCP cluster and validate TLS certificates	1

Affected runs (3)

Date (UTC)	Associated PR	Prow job
2026-09-29 21:03	n/a	2105040832624070656
2026-09-29 16:09	n/a	2104966711995273216
2026-09-29 13:34	n/a	2104927790917750784

VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)

full failure pattern:

VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)

Signal:

e2e

DEV, PROD, STG

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.002s.
Expected success, but got an error:
    <*errors.joinError | 0xc00143c978>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.002s.
Expected success, but got an error:
    <*errors.joinError | 0xc00143c978>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.006s.
Expected success, but got an error:
    <*errors.joinError | 0xc00090a4c8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.006s.
Expected success, but got an error:
    <*errors.joinError | 0xc00090a4c8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.000s.
Expected success, but got an error:
    <*errors.joinError | 0xc00061eb40>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.000s.
Expected success, but got an error:
    <*errors.joinError | 0xc00061eb40>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...

Contributing tests (1)

Failed At	Job	Test	Support
e2e	branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor	3

Affected runs (3)

Date (UTC)	Associated PR	Prow job
2026-09-29 16:09	n/a	2104966711995273216
2026-09-29 13:34	n/a	2104927790917750784
2026-09-29 10:13	n/a	2104877329665757184

Environment: PROD

Failure patterns: 8 · Failures matched: 28 · Total runs: 7 · Runs affected: 6

Failure pattern

Failed at

Trend

Also in

VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)

full failure pattern:

VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)

Signal:

e2e

DEV, INT, STG

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.001s.
Expected success, but got an error:
    <*errors.joinError | 0xc0016fb248>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-29-110018, targetMinor=5.1.0-0.nightly-multi-2026-09-29-141926) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-09-29-141926"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.001s.
Expected success, but got an error:
    <*errors.joinError | 0xc0016fb248>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-29-110018, targetMinor=5.1.0-0.nightly-multi-2026-09-29-141926) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-09-29-141926"
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.001s.
Expected success, but got an error:
    <*errors.joinError | 0xc000deacc0>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.001s.
Expected success, but got an error:
    <*errors.joinError | 0xc000deacc0>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.003s.
Expected success, but got an error:
    <*errors.joinError | 0xc0014784b0>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.003s.
Expected success, but got an error:
    <*errors.joinError | 0xc0014784b0>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...

Contributing tests (2)

Failed At	Job	Test	Support
e2e	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor	1

Affected runs (3)

Date (UTC)	Associated PR	Prow job
2026-09-29 23:00	n/a	2105070229963935744
2026-09-29 16:04	n/a	2104965516199202816
2026-09-29 12:23	n/a	2104910007010070528

ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when c...

full failure pattern:

ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift

Signal:

e2e

STG

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0xc002153e30>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-ndd4ps9rz69q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0xc002153e30>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-ndd4ps9rz69q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc0015ed110>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-scbxjjvdnhlm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc0015ed110>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-scbxjjvdnhlm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc0025a1e90>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-g9r75fj7b84l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc0025a1e90>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-g9r75fj7b84l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (5)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should be able to create a cluster and node pools with aggregated advanced features	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should be able to create an HCP cluster and manage ImageDigestMirrors	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-29 23:00	n/a	2105070229963935744

ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift

full failure pattern:

ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift

Signal:

e2e

STG

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc000fb6040>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-dzh29w" in resourcegroup="rg-np-skip-minor-dzh29w-lnc5s2hc2m99": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-dzh29w-lnc5s2hc2m99/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-dzh29w/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc000fb6040>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-dzh29w" in resourcegroup="rg-np-skip-minor-dzh29w-lnc5s2hc2m99": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-dzh29w-lnc5s2hc2m99/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-dzh29w/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc000800580>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-v4bb8q" in resourcegroup="rg-np-version-upgrade-v4bb8q-6wwsb7bm4cjc": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-v4bb8q-6wwsb7bm4cjc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-v4bb8q/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc000800580>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-v4bb8q" in resourcegroup="rg-np-version-upgrade-v4bb8q-6wwsb7bm4cjc": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-v4bb8q-6wwsb7bm4cjc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-v4bb8q/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc0012dbc40>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-hz8wft" resourcegroup="rg-np-dg-minor-hz8wft-2td2rb8jcw6z": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-hz8wft-2td2rb8jcw6z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-hz8wft/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc0012dbc40>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-hz8wft" resourcegroup="rg-np-dg-minor-hz8wft-2td2rb8jcw6z": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-hz8wft-2td2rb8jcw6z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-hz8wft/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (4)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.20	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-29 23:00	n/a	2105070229963935744

ERROR CODE: DeadlineExceeded; provider Microsoft.Compute

full failure pattern:

ERROR CODE: DeadlineExceeded; provider Microsoft.Compute

Signal:

e2e

DEV, INT, STG

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:139]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000ec1320>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-cbk98wj7cx2h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a41762b8-220b-4e53-b3be-168dfb531e91
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a41762b8-220b-4e53-b3be-168dfb531e91",
      "name": "a41762b8-220b-4e53-b3be-168dfb531e91",
      "status": "Failed",
      "startTime": "2026-09-29T23:10:58.443702839Z",
      "endTime": "2026-09-29T23:30:01.681918109Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine k2r6f6w3v3g4c8f-np-1-psrbp-v6w6d: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-cbk98wj7cx2h--managed/providers/Microsoft.Compute/virtualMachines/k2r6f6w3v3g4c8f-np-1-psrbp-v6w6d to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine k2r6f6w3v3g4c8f-np-1-psrbp-v6w6d: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-cbk98wj7cx2h--managed/providers/Microsoft.Compute/virtualMachines/k2r6f6w3v3g4c8f-np-1-psrbp-v6w6d to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:139]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000ec1320>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-cbk98wj7cx2h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a41762b8-220b-4e53-b3be-168dfb531e91
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a41762b8-220b-4e53-b3be-168dfb531e91",
      "name": "a41762b8-220b-4e53-b3be-168dfb531e91",
      "status": "Failed",
      "startTime": "2026-09-29T23:10:58.443702839Z",
      "endTime": "2026-09-29T23:30:01.681918109Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine k2r6f6w3v3g4c8f-np-1-psrbp-v6w6d: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-cbk98wj7cx2h--managed/providers/Microsoft.Compute/virtualMachines/k2r6f6w3v3g4c8f-np-1-psrbp-v6w6d to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine k2r6f6w3v3g4c8f-np-1-psrbp-v6w6d: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-cbk98wj7cx2h--managed/providers/Microsoft.Compute/virtualMachines/k2r6f6w3v3g4c8f-np-1-psrbp-v6w6d to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:740]: failed to create nodepool npdg-422-420
Unexpected error:
    <*fmt.wrapError | 0xc001022120>: 
    failed to create NodePool npdg-422-420: failed waiting for nodepool="npdg-422-420" for cluster "np-dg-minor-kvm5ll" in resourcegroup="rg-np-dg-minor-kvm5ll-6sf2vr9vw4v6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/42494991-e3af-48e6-9cf3-a4fc4d279c44
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/42494991-e3af-48e6-9cf3-a4fc4d279c44",
      "name": "42494991-e3af-48e6-9cf3-a4fc4d279c44",
      "status": "Failed",
      "startTime": "2026-09-29T17:48:38.482509384Z",
      "endTime": "2026-09-29T18:07:42.157683685Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine z0x4w0e5g3t9x5d-npdg-422-420-slkn4-nbjwr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-kvm5ll-6sf2vr9vw4v6-np-dgm-kvm5ll--managed/providers/Microsoft.Compute/virtualMachines/z0x4w0e5g3t9x5d-npdg-422-420-slkn4-nbjwr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine z0x4w0e5g3t9x5d-npdg-422-420-slkn4-nbjwr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-kvm5ll-6sf2vr9vw4v6-np-dgm-kvm5ll--managed/providers/Microsoft.Compute/virtualMachines/z0x4w0e5g3t9x5d-npdg-422-420-slkn4-nbjwr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:740]: failed to create nodepool npdg-422-420
Unexpected error:
    <*fmt.wrapError | 0xc001022120>: 
    failed to create NodePool npdg-422-420: failed waiting for nodepool="npdg-422-420" for cluster "np-dg-minor-kvm5ll" in resourcegroup="rg-np-dg-minor-kvm5ll-6sf2vr9vw4v6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/42494991-e3af-48e6-9cf3-a4fc4d279c44
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/42494991-e3af-48e6-9cf3-a4fc4d279c44",
      "name": "42494991-e3af-48e6-9cf3-a4fc4d279c44",
      "status": "Failed",
      "startTime": "2026-09-29T17:48:38.482509384Z",
      "endTime": "2026-09-29T18:07:42.157683685Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine z0x4w0e5g3t9x5d-npdg-422-420-slkn4-nbjwr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-kvm5ll-6sf2vr9vw4v6-np-dgm-kvm5ll--managed/providers/Microsoft.Compute/virtualMachines/z0x4w0e5g3t9x5d-npdg-422-420-slkn4-nbjwr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine z0x4w0e5g3t9x5d-npdg-422-420-slkn4-nbjwr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-kvm5ll-6sf2vr9vw4v6-np-dgm-kvm5ll--managed/providers/Microsoft.Compute/virtualMachines/z0x4w0e5g3t9x5d-npdg-422-420-slkn4-nbjwr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:123]: failed to create node pool "np-1" for private ingress cluster "private-ingress"
Unexpected error:
    <*fmt.wrapError | 0xc000f3a060>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-ingress" in resourcegroup="private-ingress-fld425vx7n2r" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/84d85252-8c39-447d-8d5d-a497997a37f5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/84d85252-8c39-447d-8d5d-a497997a37f5",
      "name": "84d85252-8c39-447d-8d5d-a497997a37f5",
      "status": "Failed",
      "startTime": "2026-09-29T16:34:49.001481538Z",
      "endTime": "2026-09-29T16:53:58.101757577Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine private-ingress-np-1-l7x62-h42h7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-fld425vx7n2r--managed/providers/Microsoft.Compute/virtualMachines/private-ingress-np-1-l7x62-h42h7 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine private-ingress-np-1-l7x62-h42h7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-fld425vx7n2r--managed/providers/Microsoft.Compute/virtualMachines/private-ingress-np-1-l7x62-h42h7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:123]: failed to create node pool "np-1" for private ingress cluster "private-ingress"
Unexpected error:
    <*fmt.wrapError | 0xc000f3a060>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-ingress" in resourcegroup="private-ingress-fld425vx7n2r" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/84d85252-8c39-447d-8d5d-a497997a37f5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/84d85252-8c39-447d-8d5d-a497997a37f5",
      "name": "84d85252-8c39-447d-8d5d-a497997a37f5",
      "status": "Failed",
      "startTime": "2026-09-29T16:34:49.001481538Z",
      "endTime": "2026-09-29T16:53:58.101757577Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine private-ingress-np-1-l7x62-h42h7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-fld425vx7n2r--managed/providers/Microsoft.Compute/virtualMachines/private-ingress-np-1-l7x62-h42h7 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine private-ingress-np-1-l7x62-h42h7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-fld425vx7n2r--managed/providers/Microsoft.Compute/virtualMachines/private-ingress-np-1-l7x62-h42h7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (10)

Failed At	Job	Test	Support
e2e	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel	Customer should be able to create an HCP cluster and custom node pool osDisk size	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel	Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel	Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest	1
e2e	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel	Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should be able to create an HCP cluster then delete it by deleting the customer resource group	1

Affected runs (5)

Date (UTC)	Associated PR	Prow job
2026-09-29 23:00	n/a	2105070229963935744
2026-09-29 16:04	n/a	2104965516199202816
2026-09-29 12:23	n/a	2104910007010070528
2026-09-29 02:00	n/a	2104753137003794432
2026-09-29 01:52	n/a	2104751055710457856

failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>

full failure pattern:

failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>

Signal:

e2e

DEV

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc000d87400>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-cg4srqjqkk9n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/49815947-d474-46af-9a2a-1b7e7d2030b2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/49815947-d474-46af-9a2a-1b7e7d2030b2",
      "name": "49815947-d474-46af-9a2a-1b7e7d2030b2",
      "status": "Failed",
      "startTime": "2026-09-29T23:56:01.394801153Z",
      "endTime": "2026-09-30T00:15:03.380581382Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l3i7y0e6s8b1j1r-np-delete-main-5wq22-8rjb7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-cg4srqjqkk9n--managed/providers/Microsoft.Compute/virtualMachines/l3i7y0e6s8b1j1r-np-delete-main-5wq22-8rjb7 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l3i7y0e6s8b1j1r-np-delete-main-5wq22-8rjb7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-cg4srqjqkk9n--managed/providers/Microsoft.Compute/virtualMachines/l3i7y0e6s8b1j1r-np-delete-main-5wq22-8rjb7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc000d87400>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-cg4srqjqkk9n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/49815947-d474-46af-9a2a-1b7e7d2030b2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/49815947-d474-46af-9a2a-1b7e7d2030b2",
      "name": "49815947-d474-46af-9a2a-1b7e7d2030b2",
      "status": "Failed",
      "startTime": "2026-09-29T23:56:01.394801153Z",
      "endTime": "2026-09-30T00:15:03.380581382Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l3i7y0e6s8b1j1r-np-delete-main-5wq22-8rjb7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-cg4srqjqkk9n--managed/providers/Microsoft.Compute/virtualMachines/l3i7y0e6s8b1j1r-np-delete-main-5wq22-8rjb7 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l3i7y0e6s8b1j1r-np-delete-main-5wq22-8rjb7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-cg4srqjqkk9n--managed/providers/Microsoft.Compute/virtualMachines/l3i7y0e6s8b1j1r-np-delete-main-5wq22-8rjb7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc00079ab80>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-lwltp7hhv9xn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ade802f3-fc4a-4348-8b7b-a05cecc41cb9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ade802f3-fc4a-4348-8b7b-a05cecc41cb9",
      "name": "ade802f3-fc4a-4348-8b7b-a05cecc41cb9",
      "status": "Failed",
      "startTime": "2026-09-29T17:40:59.55523927Z",
      "endTime": "2026-09-29T18:00:02.471778007Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine w8d5r4n4c6l8f1n-np-deletable-5mg5s-5v87h: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-lwltp7hhv9xn--managed/providers/Microsoft.Compute/virtualMachines/w8d5r4n4c6l8f1n-np-deletable-5mg5s-5v87h to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine w8d5r4n4c6l8f1n-np-deletable-5mg5s-5v87h: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-lwltp7hhv9xn--managed/providers/Microsoft.Compute/virtualMachines/w8d5r4n4c6l8f1n-np-deletable-5mg5s-5v87h to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc00079ab80>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-lwltp7hhv9xn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ade802f3-fc4a-4348-8b7b-a05cecc41cb9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ade802f3-fc4a-4348-8b7b-a05cecc41cb9",
      "name": "ade802f3-fc4a-4348-8b7b-a05cecc41cb9",
      "status": "Failed",
      "startTime": "2026-09-29T17:40:59.55523927Z",
      "endTime": "2026-09-29T18:00:02.471778007Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine w8d5r4n4c6l8f1n-np-deletable-5mg5s-5v87h: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-lwltp7hhv9xn--managed/providers/Microsoft.Compute/virtualMachines/w8d5r4n4c6l8f1n-np-deletable-5mg5s-5v87h to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine w8d5r4n4c6l8f1n-np-deletable-5mg5s-5v87h: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-lwltp7hhv9xn--managed/providers/Microsoft.Compute/virtualMachines/w8d5r4n4c6l8f1n-np-deletable-5mg5s-5v87h to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc000dfcbc0>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-r2grpqzmpxnk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e2b31ee5-5378-49f4-b4f9-8c82f9bace93
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e2b31ee5-5378-49f4-b4f9-8c82f9bace93",
      "name": "e2b31ee5-5378-49f4-b4f9-8c82f9bace93",
      "status": "Failed",
      "startTime": "2026-09-29T13:59:14.617140907Z",
      "endTime": "2026-09-29T14:18:21.033986186Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine r0n8s9h5n0s8u3c-np-delete-main-5vq47-vdkbs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-r2grpqzmpxnk--managed/providers/Microsoft.Compute/virtualMachines/r0n8s9h5n0s8u3c-np-delete-main-5vq47-vdkbs to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine r0n8s9h5n0s8u3c-np-delete-main-5vq47-vdkbs: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-r2grpqzmpxnk--managed/providers/Microsoft.Compute/virtualMachines/r0n8s9h5n0s8u3c-np-delete-main-5vq47-vdkbs to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc000dfcbc0>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-r2grpqzmpxnk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e2b31ee5-5378-49f4-b4f9-8c82f9bace93
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e2b31ee5-5378-49f4-b4f9-8c82f9bace93",
      "name": "e2b31ee5-5378-49f4-b4f9-8c82f9bace93",
      "status": "Failed",
      "startTime": "2026-09-29T13:59:14.617140907Z",
      "endTime": "2026-09-29T14:18:21.033986186Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine r0n8s9h5n0s8u3c-np-delete-main-5vq47-vdkbs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-r2grpqzmpxnk--managed/providers/Microsoft.Compute/virtualMachines/r0n8s9h5n0s8u3c-np-delete-main-5vq47-vdkbs to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine r0n8s9h5n0s8u3c-np-delete-main-5vq47-vdkbs: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-r2grpqzmpxnk--managed/providers/Microsoft.Compute/virtualMachines/r0n8s9h5n0s8u3c-np-delete-main-5vq47-vdkbs to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

Contributing tests (2)

Failed At	Job	Test	Support
e2e	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel	Customer should be able to delete a nodepool	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	Customer should be able to delete a nodepool	1

Affected runs (3)

Date (UTC)	Associated PR	Prow job
2026-09-29 23:00	n/a	2105070229963935744
2026-09-29 16:04	n/a	2104965516199202816
2026-09-29 12:23	n/a	2104910007010070528

selector "<selector>" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or q...

full failure pattern:

selector "<selector>" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:88]: failed to select a VM size with ephemeral OS disk support; this typically indicates a SKU restriction or quota issue in the test subscription/region
Unexpected error:
    <*fmt.wrapError | 0xc001352fa0>: 
    selector "ephemeral-osdisk-worker" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:88]: failed to select a VM size with ephemeral OS disk support; this typically indicates a SKU restriction or quota issue in the test subscription/region
Unexpected error:
    <*fmt.wrapError | 0xc001352fa0>: 
    selector "ephemeral-osdisk-worker" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel	Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-29 06:27	n/a	2104820270236700672

GET <url>

full failure pattern:

GET <url>

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:258]: failed to install cert-manager manifests
Expected success, but got an error:
    <*errors.errorString | 0xc000b64160>: 
    GET https://github.com/cert-manager/cert-manager/releases/download/v1.20.2/cert-manager.yaml: HTTP 500
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:258]: failed to install cert-manager manifests
Expected success, but got an error:
    <*errors.errorString | 0xc000b64160>: 
    GET https://github.com/cert-manager/cert-manager/releases/download/v1.20.2/cert-manager.yaml: HTTP 500
    ...

Contributing tests (1)

Failed At	Job	Test	Support
e2e	branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel	Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-29 06:27	n/a	2104820270236700672

VerifyBasicAccess failed: failed to list services: Unauthorized

full failure pattern:

VerifyBasicAccess failed: failed to list services: Unauthorized

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:135]: failed to verify HCP cluster "cluster-nightly-4-21-lhp5d5" is viable
Unexpected error:
    <*errors.joinError | 0xc001347770>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    VerifyBasicAccess failed: failed to list services: Unauthorized
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:135]: failed to verify HCP cluster "cluster-nightly-4-21-lhp5d5" is viable
Unexpected error:
    <*errors.joinError | 0xc001347770>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    VerifyBasicAccess failed: failed to list services: Unauthorized
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly	ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.21	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-29 23:00	n/a	2105070229963935744

Environment: STG

Failure patterns: 4 · Failures matched: 11 · Total runs: 3 · Runs affected: 2

Failure pattern

Failed at

Trend

Also in

ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when c...

full failure pattern:

ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift

Signal:

e2e

PROD

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc0008e90e0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-gfr8dlt6lb7m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc0008e90e0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-gfr8dlt6lb7m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc00150c9c0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-2s6gqzlqnmd9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc00150c9c0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-2s6gqzlqnmd9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc001684150>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-qccnxb6p8bvl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc001684150>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-qccnxb6p8bvl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (5)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create a cluster and node pools with aggregated advanced features	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create an HCP cluster and manage ImageDigestMirrors	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-29 23:00	n/a	2105070230005878784

ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift

full failure pattern:

ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift

Signal:

e2e

PROD

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc00097e440>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-kqcc66" in resourcegroup="rg-np-version-upgrade-kqcc66-8mp5bjlqdlxf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-kqcc66-8mp5bjlqdlxf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-kqcc66/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc00097e440>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-kqcc66" in resourcegroup="rg-np-version-upgrade-kqcc66-8mp5bjlqdlxf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-kqcc66-8mp5bjlqdlxf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-kqcc66/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-pvgxzh" for cluster "cluster-nightly-4-20-pvgxzh"
Unexpected error:
    <*fmt.wrapError | 0xc0010d80e0>: 
    failed to create NodePool np-pvgxzh: failed starting nodepool creation "np-pvgxzh" for cluster "cluster-nightly-4-20-pvgxzh" in resourcegroup="rg-nightly-4-20-p6d2fjzl4q89": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-p6d2fjzl4q89/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-pvgxzh/nodePools/np-pvgxzh
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-pvgxzh" for cluster "cluster-nightly-4-20-pvgxzh"
Unexpected error:
    <*fmt.wrapError | 0xc0010d80e0>: 
    failed to create NodePool np-pvgxzh: failed starting nodepool creation "np-pvgxzh" for cluster "cluster-nightly-4-20-pvgxzh" in resourcegroup="rg-nightly-4-20-p6d2fjzl4q89": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-p6d2fjzl4q89/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-pvgxzh/nodePools/np-pvgxzh
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc000895700>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-6nhc5q" resourcegroup="rg-np-dg-minor-6nhc5q-8fsxplbkq2t5": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-6nhc5q-8fsxplbkq2t5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-6nhc5q/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc000895700>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-6nhc5q" resourcegroup="rg-np-dg-minor-6nhc5q-8fsxplbkq2t5": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-6nhc5q-8fsxplbkq2t5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-6nhc5q/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (4)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.20	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-29 23:00	n/a	2105070230005878784

VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)

full failure pattern:

VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)

Signal:

e2e

DEV, INT, PROD

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.004s.
Expected success, but got an error:
    <*errors.joinError | 0xc00131c360>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-29-110018, targetMinor=5.1.0-0.nightly-multi-2026-09-29-141926) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-09-29-141926"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.004s.
Expected success, but got an error:
    <*errors.joinError | 0xc00131c360>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-29-110018, targetMinor=5.1.0-0.nightly-multi-2026-09-29-141926) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-09-29-141926"
    ...

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly	Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-29 23:00	n/a	2105070230005878784

ERROR CODE: DeadlineExceeded; provider Microsoft.Compute

full failure pattern:

ERROR CODE: DeadlineExceeded; provider Microsoft.Compute

Signal:

e2e

DEV, INT, PROD

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:139]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0010098a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-2x9ww8xf75tr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e2ca4ae6-8c7b-4757-944a-a1ada0f17a94
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e2ca4ae6-8c7b-4757-944a-a1ada0f17a94",
      "name": "e2ca4ae6-8c7b-4757-944a-a1ada0f17a94",
      "status": "Failed",
      "startTime": "2026-09-29T02:12:36.53948617Z",
      "endTime": "2026-09-29T02:31:39.690266696Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine d5a6t5l1x4b3i2w-np-1-wqsmx-cskns: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-2x9ww8xf75tr--managed/providers/Microsoft.Compute/virtualMachines/d5a6t5l1x4b3i2w-np-1-wqsmx-cskns to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine d5a6t5l1x4b3i2w-np-1-wqsmx-cskns: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-2x9ww8xf75tr--managed/providers/Microsoft.Compute/virtualMachines/d5a6t5l1x4b3i2w-np-1-wqsmx-cskns to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:139]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0010098a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-2x9ww8xf75tr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e2ca4ae6-8c7b-4757-944a-a1ada0f17a94
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e2ca4ae6-8c7b-4757-944a-a1ada0f17a94",
      "name": "e2ca4ae6-8c7b-4757-944a-a1ada0f17a94",
      "status": "Failed",
      "startTime": "2026-09-29T02:12:36.53948617Z",
      "endTime": "2026-09-29T02:31:39.690266696Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine d5a6t5l1x4b3i2w-np-1-wqsmx-cskns: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-2x9ww8xf75tr--managed/providers/Microsoft.Compute/virtualMachines/d5a6t5l1x4b3i2w-np-1-wqsmx-cskns to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine d5a6t5l1x4b3i2w-np-1-wqsmx-cskns: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-2x9ww8xf75tr--managed/providers/Microsoft.Compute/virtualMachines/d5a6t5l1x4b3i2w-np-1-wqsmx-cskns to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create an HCP cluster then delete it by deleting the customer resource group	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-09-29 02:00	n/a	2104753137033154560