fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--rrwf84jdrq9t/cluster-hs-pre-dlzcc5 should provision
Unexpected error:
    <*fmt.wrapError | 0xc0007ce420>: 
    failed to create HCP cluster cluster-hs-pre-dlzcc5: failed waiting for cluster="cluster-hs-pre-dlzcc5" in resourcegroup="rg-hs-pre--rrwf84jdrq9t" to finish creating: GET https://rp.j1442176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7d6063b8-a9f0-4a41-b701-82a552707172
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/7d6063b8-a9f0-4a41-b701-82a552707172' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7d6063b8-a9f0-4a41-b701-82a552707172"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--rrwf84jdrq9t/cluster-hs-pre-dlzcc5 should provision
Unexpected error:
    <*fmt.wrapError | 0xc0007ce420>: 
    failed to create HCP cluster cluster-hs-pre-dlzcc5: failed waiting for cluster="cluster-hs-pre-dlzcc5" in resourcegroup="rg-hs-pre--rrwf84jdrq9t" to finish creating: GET https://rp.j1442176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7d6063b8-a9f0-4a41-b701-82a552707172
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/7d6063b8-a9f0-4a41-b701-82a552707172' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7d6063b8-a9f0-4a41-b701-82a552707172"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:403]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc001d94018>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-g7cwbl5k67j4" to finish creating: GET https://rp.j8532992.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fbeee8b5-27cd-473e-a9c2-df2114c56f2b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fbeee8b5-27cd-473e-a9c2-df2114c56f2b",
      "name": "fbeee8b5-27cd-473e-a9c2-df2114c56f2b",
      "status": "Failed",
      "startTime": "2026-09-18T00:05:39.3471997Z",
      "endTime": "2026-09-18T00:06:19.6272248Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:403]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc001d94018>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-g7cwbl5k67j4" to finish creating: GET https://rp.j8532992.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fbeee8b5-27cd-473e-a9c2-df2114c56f2b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fbeee8b5-27cd-473e-a9c2-df2114c56f2b",
      "name": "fbeee8b5-27cd-473e-a9c2-df2114c56f2b",
      "status": "Failed",
      "startTime": "2026-09-18T00:05:39.3471997Z",
      "endTime": "2026-09-18T00:06:19.6272248Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000f703e0>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-lxp82nsl4llc" to finish creating: GET https://rp.j9381376.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f9411175-a2f7-4b2d-9324-66fd2acc379f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f9411175-a2f7-4b2d-9324-66fd2acc379f",
      "name": "f9411175-a2f7-4b2d-9324-66fd2acc379f",
      "status": "Failed",
      "startTime": "2026-09-17T23:50:07.442410011Z",
      "endTime": "2026-09-18T00:19:13.547086739Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 24m11s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000f703e0>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-lxp82nsl4llc" to finish creating: GET https://rp.j9381376.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f9411175-a2f7-4b2d-9324-66fd2acc379f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f9411175-a2f7-4b2d-9324-66fd2acc379f",
      "name": "f9411175-a2f7-4b2d-9324-66fd2acc379f",
      "status": "Failed",
      "startTime": "2026-09-17T23:50:07.442410011Z",
      "endTime": "2026-09-18T00:19:13.547086739Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 24m11s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T00:02:16Z
  Ended:       2026-09-18T00:27:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9381376-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2st2tc78t6cvdchmjv242i9bdc8rpm9q-k2h4w6t5x7l3o9z", pod="router-586c95bb7f-lzcz4", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2st2tc78t6cvdchmjv242i9bdc8rpm9q-k2h4w6t5x7l3o9z/router-586c95bb7f-lzcz4 has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-18T01:12:37Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j6258560-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-m8n67", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-m8n67 (namespace arobit) on cluster ci01-j6258560-svc has averaged 125.1% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc0007926a0>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-vsg6jbtlvgqx" to finish creating: GET https://rp.j5944704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ced041bf-9cd3-4dc0-8816-ef44d4b87c6b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ced041bf-9cd3-4dc0-8816-ef44d4b87c6b",
      "name": "ced041bf-9cd3-4dc0-8816-ef44d4b87c6b",
      "status": "Failed",
      "startTime": "2026-09-17T23:20:00.021017752Z",
      "endTime": "2026-09-17T23:49:08.836539572Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m14s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc0007926a0>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-vsg6jbtlvgqx" to finish creating: GET https://rp.j5944704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ced041bf-9cd3-4dc0-8816-ef44d4b87c6b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ced041bf-9cd3-4dc0-8816-ef44d4b87c6b",
      "name": "ced041bf-9cd3-4dc0-8816-ef44d4b87c6b",
      "status": "Failed",
      "startTime": "2026-09-17T23:20:00.021017752Z",
      "endTime": "2026-09-17T23:49:08.836539572Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m14s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc0006f07e0>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-6f9wkzp5v4jd" to finish creating: GET https://rp.j0002944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/97dac56f-020e-4301-ba21-2ca0fd4e2f6d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/97dac56f-020e-4301-ba21-2ca0fd4e2f6d",
      "name": "97dac56f-020e-4301-ba21-2ca0fd4e2f6d",
      "status": "Failed",
      "startTime": "2026-09-17T23:29:07.249708545Z",
      "endTime": "2026-09-17T23:58:12.85966363Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m18s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc0006f07e0>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-6f9wkzp5v4jd" to finish creating: GET https://rp.j0002944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/97dac56f-020e-4301-ba21-2ca0fd4e2f6d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/97dac56f-020e-4301-ba21-2ca0fd4e2f6d",
      "name": "97dac56f-020e-4301-ba21-2ca0fd4e2f6d",
      "status": "Failed",
      "startTime": "2026-09-17T23:29:07.249708545Z",
      "endTime": "2026-09-17T23:58:12.85966363Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m18s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000fe8980>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-hgg4wl6sg8sc" to finish creating: GET https://rp.j0002944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2f65c2fa-1e9a-4309-96ea-66a270395b00
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2f65c2fa-1e9a-4309-96ea-66a270395b00",
      "name": "2f65c2fa-1e9a-4309-96ea-66a270395b00",
      "status": "Failed",
      "startTime": "2026-09-17T23:30:18.236986606Z",
      "endTime": "2026-09-17T23:59:22.774158112Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 24m14s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000fe8980>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-hgg4wl6sg8sc" to finish creating: GET https://rp.j0002944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2f65c2fa-1e9a-4309-96ea-66a270395b00
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2f65c2fa-1e9a-4309-96ea-66a270395b00",
      "name": "2f65c2fa-1e9a-4309-96ea-66a270395b00",
      "status": "Failed",
      "startTime": "2026-09-17T23:30:18.236986606Z",
      "endTime": "2026-09-17T23:59:22.774158112Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 24m14s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T23:37:25Z
  Ended:       2026-09-17T23:53:21Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j9695872-svc", component="backend", name="clusterresources", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 2:
  State:       Resolved
  Started:     2026-09-17T23:46:22Z
  Ended:       2026-09-17T23:56:22Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j9695872-svc", component="backend", name="csstatedump", region="canadacentral", severity="warning"
  Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc000648280>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-qghsblcj4pgk" to finish creating: GET https://rp.j4009472.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0b8c439-b7ef-4e73-b372-5a8c55bbf10e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0b8c439-b7ef-4e73-b372-5a8c55bbf10e",
      "name": "f0b8c439-b7ef-4e73-b372-5a8c55bbf10e",
      "status": "Failed",
      "startTime": "2026-09-17T23:29:11.075687263Z",
      "endTime": "2026-09-17T23:58:14.591633856Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc000648280>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-qghsblcj4pgk" to finish creating: GET https://rp.j4009472.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0b8c439-b7ef-4e73-b372-5a8c55bbf10e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f0b8c439-b7ef-4e73-b372-5a8c55bbf10e",
      "name": "f0b8c439-b7ef-4e73-b372-5a8c55bbf10e",
      "status": "Failed",
      "startTime": "2026-09-17T23:29:11.075687263Z",
      "endTime": "2026-09-17T23:58:14.591633856Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-22-5xnxlfwdsctw/cluster-candidate-4-22-9lfv7s should provision
Unexpected error:
    <*fmt.wrapError | 0xc000edc880>: 
    failed to create HCP cluster cluster-candidate-4-22-9lfv7s: failed waiting for cluster="cluster-candidate-4-22-9lfv7s" in resourcegroup="rg-candidate-4-22-5xnxlfwdsctw" to finish creating: GET https://rp.j4009472.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c0b39836-fc96-480c-9689-6141753bae0a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c0b39836-fc96-480c-9689-6141753bae0a",
      "name": "c0b39836-fc96-480c-9689-6141753bae0a",
      "status": "Failed",
      "startTime": "2026-09-17T23:29:11.159787793Z",
      "endTime": "2026-09-17T23:58:14.737814238Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 21m52s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-22-5xnxlfwdsctw/cluster-candidate-4-22-9lfv7s should provision
Unexpected error:
    <*fmt.wrapError | 0xc000edc880>: 
    failed to create HCP cluster cluster-candidate-4-22-9lfv7s: failed waiting for cluster="cluster-candidate-4-22-9lfv7s" in resourcegroup="rg-candidate-4-22-5xnxlfwdsctw" to finish creating: GET https://rp.j4009472.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c0b39836-fc96-480c-9689-6141753bae0a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c0b39836-fc96-480c-9689-6141753bae0a",
      "name": "c0b39836-fc96-480c-9689-6141753bae0a",
      "status": "Failed",
      "startTime": "2026-09-17T23:29:11.159787793Z",
      "endTime": "2026-09-17T23:58:14.737814238Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 21m52s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-6pj6568ghzb2/cluster-candidate-4-21-k6bndh should provision
Unexpected error:
    <*fmt.wrapError | 0xc000fbc220>: 
    failed to create HCP cluster cluster-candidate-4-21-k6bndh: failed waiting for cluster="cluster-candidate-4-21-k6bndh" in resourcegroup="rg-candidate-4-21-6pj6568ghzb2" to finish creating: GET https://rp.j4009472.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f3d3840e-8750-44f9-b1f2-00d9d2c11a0e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f3d3840e-8750-44f9-b1f2-00d9d2c11a0e",
      "name": "f3d3840e-8750-44f9-b1f2-00d9d2c11a0e",
      "status": "Failed",
      "startTime": "2026-09-17T23:29:11.265897654Z",
      "endTime": "2026-09-17T23:58:14.739219867Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m6s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-6pj6568ghzb2/cluster-candidate-4-21-k6bndh should provision
Unexpected error:
    <*fmt.wrapError | 0xc000fbc220>: 
    failed to create HCP cluster cluster-candidate-4-21-k6bndh: failed waiting for cluster="cluster-candidate-4-21-k6bndh" in resourcegroup="rg-candidate-4-21-6pj6568ghzb2" to finish creating: GET https://rp.j4009472.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f3d3840e-8750-44f9-b1f2-00d9d2c11a0e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f3d3840e-8750-44f9-b1f2-00d9d2c11a0e",
      "name": "f3d3840e-8750-44f9-b1f2-00d9d2c11a0e",
      "status": "Failed",
      "startTime": "2026-09-17T23:29:11.265897654Z",
      "endTime": "2026-09-17T23:58:14.739219867Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m6s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-20-k8zkqfwswxpz/cluster-candidate-4-20-z4ljdv should provision
Unexpected error:
    <*fmt.wrapError | 0xc000efc7c0>: 
    failed to create HCP cluster cluster-candidate-4-20-z4ljdv: failed waiting for cluster="cluster-candidate-4-20-z4ljdv" in resourcegroup="rg-candidate-4-20-k8zkqfwswxpz" to finish creating: GET https://rp.j4009472.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/290c11b0-aade-4173-8aad-dda5c860485f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/290c11b0-aade-4173-8aad-dda5c860485f",
      "name": "290c11b0-aade-4173-8aad-dda5c860485f",
      "status": "Failed",
      "startTime": "2026-09-17T23:29:11.979295423Z",
      "endTime": "2026-09-17T23:58:14.730283695Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.38 is Partial (want Completed), started 22m5s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-20-k8zkqfwswxpz/cluster-candidate-4-20-z4ljdv should provision
Unexpected error:
    <*fmt.wrapError | 0xc000efc7c0>: 
    failed to create HCP cluster cluster-candidate-4-20-z4ljdv: failed waiting for cluster="cluster-candidate-4-20-z4ljdv" in resourcegroup="rg-candidate-4-20-k8zkqfwswxpz" to finish creating: GET https://rp.j4009472.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/290c11b0-aade-4173-8aad-dda5c860485f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/290c11b0-aade-4173-8aad-dda5c860485f",
      "name": "290c11b0-aade-4173-8aad-dda5c860485f",
      "status": "Failed",
      "startTime": "2026-09-17T23:29:11.979295423Z",
      "endTime": "2026-09-17T23:58:14.730283695Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.38 is Partial (want Completed), started 22m5s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000f70c60>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-wkvscvwv244k" to finish creating: GET https://rp.j4009472.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4d69226f-9a2e-4afc-ae7f-94f222fc3a40
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4d69226f-9a2e-4afc-ae7f-94f222fc3a40",
      "name": "4d69226f-9a2e-4afc-ae7f-94f222fc3a40",
      "status": "Failed",
      "startTime": "2026-09-17T23:28:52.213252998Z",
      "endTime": "2026-09-17T23:57:54.693039246Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m38s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000f70c60>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-wkvscvwv244k" to finish creating: GET https://rp.j4009472.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4d69226f-9a2e-4afc-ae7f-94f222fc3a40
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4d69226f-9a2e-4afc-ae7f-94f222fc3a40",
      "name": "4d69226f-9a2e-4afc-ae7f-94f222fc3a40",
      "status": "Failed",
      "startTime": "2026-09-17T23:28:52.213252998Z",
      "endTime": "2026-09-17T23:57:54.693039246Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m38s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 9 time(s) (3 unknown, 6 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T23:40:23Z
  Ended:       2026-09-18T00:06:22Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4009472-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2st2jdcitjtn6r1jl4q34p61lesqvar0-l2u3i7e5p7i0m7j", pod="router-dd88b46c7-ckx8f", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2st2jdcitjtn6r1jl4q34p61lesqvar0-l2u3i7e5p7i0m7j/router-dd88b46c7-ckx8f has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-17T23:42:25Z
  Ended:       2026-09-18T00:11:23Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4009472-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2st2ji98etmkm3fuvovie18d64b0rt6v-v20260901", pod="router-7fd9f9b4c6-gn2vk", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2st2ji98etmkm3fuvovie18d64b0rt6v-v20260901/router-7fd9f9b4c6-gn2vk has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-17T23:42:25Z
  Ended:       2026-09-18T00:06:22Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4009472-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2st2ji98etmkm3fuvovie18d64b0rt6v-v20260901", pod="router-7fd9f9b4c6-q4q5l", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2st2ji98etmkm3fuvovie18d64b0rt6v-v20260901/router-7fd9f9b4c6-q4q5l has been in a non-ready state for longer than 5 minutes.

time=2026-09-17T22:08:23.042Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 description="Step cluster\n  Kind: ARM\n  Template: templates/mgmt-cluster.bicep\n  Parameters: configurations/mgmt-cluster.tmpl.bicepparam"
time=2026-09-17T22:08:24.010Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2
time=2026-09-17T22:08:24.819Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 deployment=949ef12d6ab429b901225672cf02dc8c2e8f5344e8e4f364f694e8721bc1ed4e portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j0316160-mgmt-2%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F949ef12d6ab429b901225672cf02dc8c2e8f5344e8e4f364f694e8721bc1ed4e
time=2026-09-17T22:38:22.805Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 err="stamp 2: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded"

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T22:37:27Z
  Ended:       2026-09-17T22:43:26Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5842688-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2st1lu1gnkusrr9aaeo128agqit9e6jo-a8y8c4i0m5j1k4s", pod="router-6f4d998d46-d2xxt", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2st1lu1gnkusrr9aaeo128agqit9e6jo-a8y8c4i0m5j1k4s/router-6f4d998d46-d2xxt has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-17T22:37:27Z
  Ended:       2026-09-17T22:43:25Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5842688-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2st1lcku7i9kn16epu26j9jmha4t9219-t5q0q7d1s0b0b2h", pod="router-5b9f46c564-vq9rt", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2st1lcku7i9kn16epu26j9jmha4t9219-t5q0q7d1s0b0b2h/router-5b9f46c564-vq9rt has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:123]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc0007b82c0>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-pd9rhzzfgcc5" to finish creating: GET https://rp.j5800576.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/aec160c6-5a35-495d-bd6d-5268450f3305
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/aec160c6-5a35-495d-bd6d-5268450f3305",
      "name": "aec160c6-5a35-495d-bd6d-5268450f3305",
      "status": "Failed",
      "startTime": "2026-09-17T22:12:49.625987517Z",
      "endTime": "2026-09-17T22:41:55.499178233Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m25s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:123]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc0007b82c0>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-pd9rhzzfgcc5" to finish creating: GET https://rp.j5800576.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/aec160c6-5a35-495d-bd6d-5268450f3305
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/aec160c6-5a35-495d-bd6d-5268450f3305",
      "name": "aec160c6-5a35-495d-bd6d-5268450f3305",
      "status": "Failed",
      "startTime": "2026-09-17T22:12:49.625987517Z",
      "endTime": "2026-09-17T22:41:55.499178233Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m25s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000f68560>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-6t2khml5tznz" to finish creating: GET https://rp.j5800576.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/137a17eb-a935-4a81-8849-b3928c62acad
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/137a17eb-a935-4a81-8849-b3928c62acad",
      "name": "137a17eb-a935-4a81-8849-b3928c62acad",
      "status": "Failed",
      "startTime": "2026-09-17T22:14:35.599602035Z",
      "endTime": "2026-09-17T22:43:38.546147Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m7s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000f68560>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-6t2khml5tznz" to finish creating: GET https://rp.j5800576.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/137a17eb-a935-4a81-8849-b3928c62acad
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/137a17eb-a935-4a81-8849-b3928c62acad",
      "name": "137a17eb-a935-4a81-8849-b3928c62acad",
      "status": "Failed",
      "startTime": "2026-09-17T22:14:35.599602035Z",
      "endTime": "2026-09-17T22:43:38.546147Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m7s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:98]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc000c1a2a0>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-x6272vqbqkct" to finish creating: GET https://rp.j5800576.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/84a3014a-22f5-4e28-9c7d-08c6235f1490
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/84a3014a-22f5-4e28-9c7d-08c6235f1490",
      "name": "84a3014a-22f5-4e28-9c7d-08c6235f1490",
      "status": "Failed",
      "startTime": "2026-09-17T22:13:03.512574477Z",
      "endTime": "2026-09-17T22:42:05.645433373Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m36s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:98]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc000c1a2a0>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-x6272vqbqkct" to finish creating: GET https://rp.j5800576.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/84a3014a-22f5-4e28-9c7d-08c6235f1490
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/84a3014a-22f5-4e28-9c7d-08c6235f1490",
      "name": "84a3014a-22f5-4e28-9c7d-08c6235f1490",
      "status": "Failed",
      "startTime": "2026-09-17T22:13:03.512574477Z",
      "endTime": "2026-09-17T22:42:05.645433373Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m36s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc0010ba240>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-6jfrxnr5w7hl" to finish creating: GET https://rp.j5800576.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b6f5865b-82eb-499b-9d49-57742af03c1b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b6f5865b-82eb-499b-9d49-57742af03c1b",
      "name": "b6f5865b-82eb-499b-9d49-57742af03c1b",
      "status": "Failed",
      "startTime": "2026-09-17T22:12:49.653858883Z",
      "endTime": "2026-09-17T22:41:55.67204599Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m58s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc0010ba240>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-6jfrxnr5w7hl" to finish creating: GET https://rp.j5800576.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b6f5865b-82eb-499b-9d49-57742af03c1b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b6f5865b-82eb-499b-9d49-57742af03c1b",
      "name": "b6f5865b-82eb-499b-9d49-57742af03c1b",
      "status": "Failed",
      "startTime": "2026-09-17T22:12:49.653858883Z",
      "endTime": "2026-09-17T22:41:55.67204599Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m58s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:91]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc0009a84e0>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-tdqjfrqdmsbh" to finish creating: GET https://rp.j5800576.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/241526e5-f5bb-4ce5-9863-b91ecf53191b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/241526e5-f5bb-4ce5-9863-b91ecf53191b",
      "name": "241526e5-f5bb-4ce5-9863-b91ecf53191b",
      "status": "Failed",
      "startTime": "2026-09-17T22:13:08.316752119Z",
      "endTime": "2026-09-17T22:42:15.586114195Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 22m46s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:91]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc0009a84e0>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-tdqjfrqdmsbh" to finish creating: GET https://rp.j5800576.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/241526e5-f5bb-4ce5-9863-b91ecf53191b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/241526e5-f5bb-4ce5-9863-b91ecf53191b",
      "name": "241526e5-f5bb-4ce5-9863-b91ecf53191b",
      "status": "Failed",
      "startTime": "2026-09-17T22:13:08.316752119Z",
      "endTime": "2026-09-17T22:42:15.586114195Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 22m46s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 5 time(s) (4 unknown, 1 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T22:25:10Z
  Ended:       2026-09-17T22:54:10Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5800576-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2st1fogv0fkmdfbch8spar202qsf3fdi-v20260901", pod="router-56bd6ddfc5-bmvj7", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2st1fogv0fkmdfbch8spar202qsf3fdi-v20260901/router-56bd6ddfc5-bmvj7 has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-17T22:26:09Z
  Ended:       2026-09-17T22:53:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5800576-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2st1fsfr24pblv7ln1lp377gfrc09jmf-v20261001", pod="router-68476f767d-wdknb", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2st1fsfr24pblv7ln1lp377gfrc09jmf-v20261001/router-68476f767d-wdknb has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-17T22:26:09Z
  Ended:       2026-09-17T22:36:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5800576-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2st1fov1spppgn6o9h5ptch95gtec7bo-ingress-cert-le", pod="router-7f74c869f6-n6zvj", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2st1fov1spppgn6o9h5ptch95gtec7bo-ingress-cert-le/router-7f74c869f6-n6zvj has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-17T22:26:09Z
  Ended:       2026-09-17T22:53:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5800576-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2st1fteggef19rbhrbirc9jgdq6conlt-w0k7b1n5e9p9u0h", pod="router-7f679584bc-qz5ts", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2st1fteggef19rbhrbirc9jgdq6conlt-w0k7b1n5e9p9u0h/router-7f679584bc-qz5ts has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:124]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0xc000c52c40>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-s2zdnnlcz4wt" to finish creating: GET https://rp.j5719424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1a80008e-4b66-48be-b60f-ca94f871de32
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1a80008e-4b66-48be-b60f-ca94f871de32",
      "name": "1a80008e-4b66-48be-b60f-ca94f871de32",
      "status": "Failed",
      "startTime": "2026-09-17T21:56:32.579892703Z",
      "endTime": "2026-09-17T22:15:39.575240678Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:124]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0xc000c52c40>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-s2zdnnlcz4wt" to finish creating: GET https://rp.j5719424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1a80008e-4b66-48be-b60f-ca94f871de32
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1a80008e-4b66-48be-b60f-ca94f871de32",
      "name": "1a80008e-4b66-48be-b60f-ca94f871de32",
      "status": "Failed",
      "startTime": "2026-09-17T21:56:32.579892703Z",
      "endTime": "2026-09-17T22:15:39.575240678Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

time=2026-09-17T21:21:29.249Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 description="Step cluster\n  Kind: ARM\n  Template: templates/mgmt-cluster.bicep\n  Parameters: configurations/mgmt-cluster.tmpl.bicepparam"
time=2026-09-17T21:21:29.969Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2
time=2026-09-17T21:21:30.723Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 deployment=63b12b492d5d28b0b0b18408fa469c00c31993867db99322c5926f55f6db447a portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j6654848-mgmt-2%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F63b12b492d5d28b0b0b18408fa469c00c31993867db99322c5926f55f6db447a
time=2026-09-17T21:51:28.943Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 err="stamp 2: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded"

time=2026-09-17T21:19:38.472Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 description="Step cluster\n  Kind: ARM\n  Template: templates/mgmt-cluster.bicep\n  Parameters: configurations/mgmt-cluster.tmpl.bicepparam"
time=2026-09-17T21:19:39.145Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1
time=2026-09-17T21:19:39.871Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 deployment=f518dca3cc0829098231fe8a127f84b76b966486264714a9e41a0d1fd09997db portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j0025600-mgmt-1%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Ff518dca3cc0829098231fe8a127f84b76b966486264714a9e41a0d1fd09997db
time=2026-09-17T21:49:38.148Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 err="stamp 1: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded"

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-17T21:40:23Z
  Severity:    Sev3
  Labels:      alertname="MaestroAgentMetricsProxyDown", cluster="ci01-j0025600-mgmt-2", component="monitoring-infrastructure", region="canadacentral", severity="3"
  Description: Prometheus has not scraped maestro-agent /metrics through the metrics-proxy sidecar (namespace=maestro, endpoint=metrics) for 10 minutes, while a kube-state-metrics scrape series still exists for this management cluster. Maestro agent metrics and any alerts that depend on them may be missing.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:98]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc0011081a0>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-wrb88wtb5d8f" to finish creating: GET https://rp.j8535296.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ff4b63f0-45c8-411e-81ab-a9132fbbad49
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ff4b63f0-45c8-411e-81ab-a9132fbbad49",
      "name": "ff4b63f0-45c8-411e-81ab-a9132fbbad49",
      "status": "Failed",
      "startTime": "2026-09-17T21:37:11.907762823Z",
      "endTime": "2026-09-17T22:06:19.7850947Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m20s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:98]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc0011081a0>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-wrb88wtb5d8f" to finish creating: GET https://rp.j8535296.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ff4b63f0-45c8-411e-81ab-a9132fbbad49
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ff4b63f0-45c8-411e-81ab-a9132fbbad49",
      "name": "ff4b63f0-45c8-411e-81ab-a9132fbbad49",
      "status": "Failed",
      "startTime": "2026-09-17T21:37:11.907762823Z",
      "endTime": "2026-09-17T22:06:19.7850947Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m20s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T21:50:34Z
  Ended:       2026-09-17T22:18:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8535296-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2st0v24nqnshekebv9kf1to2rdotufnh-v20261001", pod="router-b886fb777-8chsd", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2st0v24nqnshekebv9kf1to2rdotufnh-v20261001/router-b886fb777-8chsd has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-20-whlxpq8kpbjp/cluster-candidate-4-20-fzx8w5 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000fa2260>: 
    failed to create HCP cluster cluster-candidate-4-20-fzx8w5: failed waiting for cluster="cluster-candidate-4-20-fzx8w5" in resourcegroup="rg-candidate-4-20-whlxpq8kpbjp" to finish creating: GET https://rp.j6073984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/15ce6538-14f5-43b6-95c3-8e840903f713
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/15ce6538-14f5-43b6-95c3-8e840903f713",
      "name": "15ce6538-14f5-43b6-95c3-8e840903f713",
      "status": "Failed",
      "startTime": "2026-09-17T21:28:54.798993309Z",
      "endTime": "2026-09-17T21:58:02.391238314Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.38 is Partial (want Completed), started 24m21s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-20-whlxpq8kpbjp/cluster-candidate-4-20-fzx8w5 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000fa2260>: 
    failed to create HCP cluster cluster-candidate-4-20-fzx8w5: failed waiting for cluster="cluster-candidate-4-20-fzx8w5" in resourcegroup="rg-candidate-4-20-whlxpq8kpbjp" to finish creating: GET https://rp.j6073984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/15ce6538-14f5-43b6-95c3-8e840903f713
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/15ce6538-14f5-43b6-95c3-8e840903f713",
      "name": "15ce6538-14f5-43b6-95c3-8e840903f713",
      "status": "Failed",
      "startTime": "2026-09-17T21:28:54.798993309Z",
      "endTime": "2026-09-17T21:58:02.391238314Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.38 is Partial (want Completed), started 24m21s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0004ec1e0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-hdsrgp87vhp6" to finish deleting: GET https://rp.j6073984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/140fc1a6-e41e-4f25-9908-25d174a04842
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/140fc1a6-e41e-4f25-9908-25d174a04842",
      "name": "140fc1a6-e41e-4f25-9908-25d174a04842",
      "status": "Failed",
      "startTime": "2026-09-17T21:51:08.271574522Z",
      "endTime": "2026-09-17T22:15:12.38485138Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2st0r3e7gsp00p1v6je2veqpisj6tv8b still exists (deletion dispatched at 2026-09-17T21:51:09Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0004ec1e0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-hdsrgp87vhp6" to finish deleting: GET https://rp.j6073984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/140fc1a6-e41e-4f25-9908-25d174a04842
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/140fc1a6-e41e-4f25-9908-25d174a04842",
      "name": "140fc1a6-e41e-4f25-9908-25d174a04842",
      "status": "Failed",
      "startTime": "2026-09-17T21:51:08.271574522Z",
      "endTime": "2026-09-17T22:15:12.38485138Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2st0r3e7gsp00p1v6je2veqpisj6tv8b still exists (deletion dispatched at 2026-09-17T21:51:09Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-17T22:35:22Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6073984-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.6:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6073984/providers/microsoft.monitor/accounts/hcps-j6073984", namespace="ocm-arohcpci01-2st0r3e7gsp00p1v6je2veqpisj6tv8b-a8q3g3x9d3e6x7g", pod="arohcp-monitor-kube-state-metrics-574f657d96-7mb8q", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2st0r3e7gsp00p1v6je2veqpisj6tv8b-a8q3g3x9d3e6x7g/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-17T22:35:22Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6073984-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.6:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6073984/providers/microsoft.monitor/accounts/hcps-j6073984", namespace="ocm-arohcpci01-2st0r3e7gsp00p1v6je2veqpisj6tv8b-a8q3g3x9d3e6x7g", pod="arohcp-monitor-kube-state-metrics-574f657d96-7mb8q", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2st0r3e7gsp00p1v6je2veqpisj6tv8b-a8q3g3x9d3e6x7g/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:276]: failed to create external auth config "ext-auth-cidr"
Unexpected error:
    <*fmt.wrapErrors | 0xc002990240>: 
    failed waiting for external auth "ext-auth-cidr" in resourcegroup="e2e-cidr-connectivity-j7tx89s2rwk5" for cluster="cidr-connectivity-test" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth ext-auth-cidr in cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-j7tx89s2rwk5, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:276]: failed to create external auth config "ext-auth-cidr"
Unexpected error:
    <*fmt.wrapErrors | 0xc002990240>: 
    failed waiting for external auth "ext-auth-cidr" in resourcegroup="e2e-cidr-connectivity-j7tx89s2rwk5" for cluster="cidr-connectivity-test" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth ext-auth-cidr in cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-j7tx89s2rwk5, error: context deadline exceeded
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T21:40:09Z
  Ended:       2026-09-17T21:58:09Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerRetryHotLoop", cluster="ci01-j1728768-svc", component="backend", name="externalauthclusterservicecreate", region="canadacentral", severity="warning"
  Description: Backend controller workqueue externalauthclusterservicecreate has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T21:49:48Z
  Ended:       2026-09-17T22:07:50Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j1728768-svc", component="frontend", method="get", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters", severity="warning"
  Description: The median (p50) of frontend request latency for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters has exceeded 250ms over the past 30 minutes (cluster ci01-j1728768-svc).

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:123]: failed to create node pool ea-np-1
Unexpected error:
    <*fmt.wrapError | 0xc000c8c920>: 
    failed to create NodePool ea-np-1: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-z9b9d4ng6qd2" to finish creating: GET https://rp.j2375680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ad30d90d-8aef-4edc-88c0-22bb42dbb8f8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ad30d90d-8aef-4edc-88c0-22bb42dbb8f8",
      "name": "ad30d90d-8aef-4edc-88c0-22bb42dbb8f8",
      "status": "Failed",
      "startTime": "2026-09-17T21:14:44.291403118Z",
      "endTime": "2026-09-17T21:33:45.719372752Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:123]: failed to create node pool ea-np-1
Unexpected error:
    <*fmt.wrapError | 0xc000c8c920>: 
    failed to create NodePool ea-np-1: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-z9b9d4ng6qd2" to finish creating: GET https://rp.j2375680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ad30d90d-8aef-4edc-88c0-22bb42dbb8f8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ad30d90d-8aef-4edc-88c0-22bb42dbb8f8",
      "name": "ad30d90d-8aef-4edc-88c0-22bb42dbb8f8",
      "status": "Failed",
      "startTime": "2026-09-17T21:14:44.291403118Z",
      "endTime": "2026-09-17T21:33:45.719372752Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:89]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc000eb8ee0>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-cjljbxb6clv6" to finish creating: GET https://rp.j6143232.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/631b27eb-736d-4b48-8f88-11668c9ed067
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/631b27eb-736d-4b48-8f88-11668c9ed067",
      "name": "631b27eb-736d-4b48-8f88-11668c9ed067",
      "status": "Failed",
      "startTime": "2026-09-17T14:02:04.639808723Z",
      "endTime": "2026-09-17T14:31:12.3306894Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 24m33s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:89]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc000eb8ee0>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-cjljbxb6clv6" to finish creating: GET https://rp.j6143232.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/631b27eb-736d-4b48-8f88-11668c9ed067
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/631b27eb-736d-4b48-8f88-11668c9ed067",
      "name": "631b27eb-736d-4b48-8f88-11668c9ed067",
      "status": "Failed",
      "startTime": "2026-09-17T14:02:04.639808723Z",
      "endTime": "2026-09-17T14:31:12.3306894Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 24m33s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:80]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc001092960>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-n6bjtxs9xn9r" to finish creating: GET https://rp.j6143232.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eda415e4-abae-48ea-8897-a619811ef2d8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eda415e4-abae-48ea-8897-a619811ef2d8",
      "name": "eda415e4-abae-48ea-8897-a619811ef2d8",
      "status": "Failed",
      "startTime": "2026-09-17T14:01:55.971807004Z",
      "endTime": "2026-09-17T14:31:02.419340074Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 24m49s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:80]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc001092960>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-n6bjtxs9xn9r" to finish creating: GET https://rp.j6143232.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eda415e4-abae-48ea-8897-a619811ef2d8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eda415e4-abae-48ea-8897-a619811ef2d8",
      "name": "eda415e4-abae-48ea-8897-a619811ef2d8",
      "status": "Failed",
      "startTime": "2026-09-17T14:01:55.971807004Z",
      "endTime": "2026-09-17T14:31:02.419340074Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 24m49s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000889420>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-nwn99xtp6np4" to finish creating: GET https://rp.j6143232.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7d8268f9-6e03-421e-aa07-d0427e5d6f85
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7d8268f9-6e03-421e-aa07-d0427e5d6f85",
      "name": "7d8268f9-6e03-421e-aa07-d0427e5d6f85",
      "status": "Failed",
      "startTime": "2026-09-17T14:00:30.716316995Z",
      "endTime": "2026-09-17T14:29:32.422264706Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 24m24s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000889420>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-nwn99xtp6np4" to finish creating: GET https://rp.j6143232.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7d8268f9-6e03-421e-aa07-d0427e5d6f85
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7d8268f9-6e03-421e-aa07-d0427e5d6f85",
      "name": "7d8268f9-6e03-421e-aa07-d0427e5d6f85",
      "status": "Failed",
      "startTime": "2026-09-17T14:00:30.716316995Z",
      "endTime": "2026-09-17T14:29:32.422264706Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 24m24s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:123]: failed to verify FIPS is enabled on cluster fips-enabled-cluster
Unexpected error:
    <*errors.joinError | 0xc0014ee660>: 
    VerifyFIPSEnabled failed: failed to check FIPS mode on node h8s4t1i2l4b8s8g-np-1-tjxxr-g4lsv: failed to get pod logs for pod fips-check-rxfv5 in namespace e2e-fips-test-k5rlj on node h8s4t1i2l4b8s8g-np-1-tjxxr-g4lsv after retries: an error on the server ("unknown") has prevented the request from succeeding (get pods fips-check-rxfv5)
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:123]: failed to verify FIPS is enabled on cluster fips-enabled-cluster
Unexpected error:
    <*errors.joinError | 0xc0014ee660>: 
    VerifyFIPSEnabled failed: failed to check FIPS mode on node h8s4t1i2l4b8s8g-np-1-tjxxr-g4lsv: failed to get pod logs for pod fips-check-rxfv5 in namespace e2e-fips-test-k5rlj on node h8s4t1i2l4b8s8g-np-1-tjxxr-g4lsv after retries: an error on the server ("unknown") has prevented the request from succeeding (get pods fips-check-rxfv5)
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc001102400>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-fq99lts22lbf" to finish creating: GET https://rp.j5074560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/efdd5ddb-35ed-4306-8558-b11e4c1ba83c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/efdd5ddb-35ed-4306-8558-b11e4c1ba83c",
      "name": "efdd5ddb-35ed-4306-8558-b11e4c1ba83c",
      "status": "Failed",
      "startTime": "2026-09-17T14:03:49.544259109Z",
      "endTime": "2026-09-17T14:32:53.332127754Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m48s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc001102400>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-fq99lts22lbf" to finish creating: GET https://rp.j5074560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/efdd5ddb-35ed-4306-8558-b11e4c1ba83c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/efdd5ddb-35ed-4306-8558-b11e4c1ba83c",
      "name": "efdd5ddb-35ed-4306-8558-b11e4c1ba83c",
      "status": "Failed",
      "startTime": "2026-09-17T14:03:49.544259109Z",
      "endTime": "2026-09-17T14:32:53.332127754Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m48s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 7 time(s) (1 unknown, 6 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T14:14:45Z
  Ended:       2026-09-17T14:44:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5074560-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssqahfphn079fl5988maka6l79507m4-private-ingress", pod="router-6d6447d77d-xlpbk", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssqahfphn079fl5988maka6l79507m4-private-ingress/router-6d6447d77d-xlpbk has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000d270f8>: 
    failed to cleanup resource group: failed listing hcp clusters in resourcegroup="rg-np-version-upgrade-pr4qr2-pmcbs2hlnrf8": Get "https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-np-version-upgrade-pr4qr2-pmcbs2hlnrf8?api-version=2021-04-01": stream error: stream ID 13; CANCEL; received from peer
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000d270f8>: 
    failed to cleanup resource group: failed listing hcp clusters in resourcegroup="rg-np-version-upgrade-pr4qr2-pmcbs2hlnrf8": Get "https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-np-version-upgrade-pr4qr2-pmcbs2hlnrf8?api-version=2021-04-01": stream error: stream ID 13; CANCEL; received from peer
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:89]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc000c1d120>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-qmzbhsdszqnv" to finish creating: GET https://rp.j8995200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dbbd6da6-c17e-49e2-af9e-2a3933f43a85
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dbbd6da6-c17e-49e2-af9e-2a3933f43a85",
      "name": "dbbd6da6-c17e-49e2-af9e-2a3933f43a85",
      "status": "Failed",
      "startTime": "2026-09-17T13:28:36.437510186Z",
      "endTime": "2026-09-17T13:57:39.205388869Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 20m57s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:89]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc000c1d120>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-qmzbhsdszqnv" to finish creating: GET https://rp.j8995200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dbbd6da6-c17e-49e2-af9e-2a3933f43a85
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dbbd6da6-c17e-49e2-af9e-2a3933f43a85",
      "name": "dbbd6da6-c17e-49e2-af9e-2a3933f43a85",
      "status": "Failed",
      "startTime": "2026-09-17T13:28:36.437510186Z",
      "endTime": "2026-09-17T13:57:39.205388869Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 20m57s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc000846b80>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-l4rcq7cd4xkr" to finish creating: GET https://rp.j8995200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fb12334b-a512-43ea-824e-be2e5c1b7445
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fb12334b-a512-43ea-824e-be2e5c1b7445",
      "name": "fb12334b-a512-43ea-824e-be2e5c1b7445",
      "status": "Failed",
      "startTime": "2026-09-17T13:27:46.68038337Z",
      "endTime": "2026-09-17T13:56:49.088605231Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 19m27s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc000846b80>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-l4rcq7cd4xkr" to finish creating: GET https://rp.j8995200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fb12334b-a512-43ea-824e-be2e5c1b7445
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fb12334b-a512-43ea-824e-be2e5c1b7445",
      "name": "fb12334b-a512-43ea-824e-be2e5c1b7445",
      "status": "Failed",
      "startTime": "2026-09-17T13:27:46.68038337Z",
      "endTime": "2026-09-17T13:56:49.088605231Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 19m27s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc0010622c0>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-2k9hnrc7s8nw" to finish creating: GET https://rp.j8995200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/13dbceb4-6c92-4d49-97f3-f75c8277895b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/13dbceb4-6c92-4d49-97f3-f75c8277895b",
      "name": "13dbceb4-6c92-4d49-97f3-f75c8277895b",
      "status": "Failed",
      "startTime": "2026-09-17T13:28:50.480676293Z",
      "endTime": "2026-09-17T13:57:59.187242674Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 20m7s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc0010622c0>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-2k9hnrc7s8nw" to finish creating: GET https://rp.j8995200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/13dbceb4-6c92-4d49-97f3-f75c8277895b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/13dbceb4-6c92-4d49-97f3-f75c8277895b",
      "name": "13dbceb4-6c92-4d49-97f3-f75c8277895b",
      "status": "Failed",
      "startTime": "2026-09-17T13:28:50.480676293Z",
      "endTime": "2026-09-17T13:57:59.187242674Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 20m7s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T13:43:06Z
  Ended:       2026-09-17T14:08:08Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8995200-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspq1c5fekjjfpb2eogshu5ujqg9qiq-z1f6t4a0j5i6b3o", pod="router-58bb4b4c4b-qx4x4", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspq1c5fekjjfpb2eogshu5ujqg9qiq-z1f6t4a0j5i6b3o/router-58bb4b4c4b-qx4x4 has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-20-pxkzk5k6tj2b/cluster-candidate-4-20-rdptdx should provision
Unexpected error:
    <*fmt.wrapError | 0xc001252760>: 
    failed to create HCP cluster cluster-candidate-4-20-rdptdx: failed waiting for cluster="cluster-candidate-4-20-rdptdx" in resourcegroup="rg-candidate-4-20-pxkzk5k6tj2b" to finish creating: GET https://rp.j2532096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ce05291f-4663-43c2-95b1-34c4d5d56d3c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ce05291f-4663-43c2-95b1-34c4d5d56d3c",
      "name": "ce05291f-4663-43c2-95b1-34c4d5d56d3c",
      "status": "Failed",
      "startTime": "2026-09-17T13:10:50.219697768Z",
      "endTime": "2026-09-17T13:39:57.0065588Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.38 is Partial (want Completed), started 23m32s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-20-pxkzk5k6tj2b/cluster-candidate-4-20-rdptdx should provision
Unexpected error:
    <*fmt.wrapError | 0xc001252760>: 
    failed to create HCP cluster cluster-candidate-4-20-rdptdx: failed waiting for cluster="cluster-candidate-4-20-rdptdx" in resourcegroup="rg-candidate-4-20-pxkzk5k6tj2b" to finish creating: GET https://rp.j2532096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ce05291f-4663-43c2-95b1-34c4d5d56d3c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ce05291f-4663-43c2-95b1-34c4d5d56d3c",
      "name": "ce05291f-4663-43c2-95b1-34c4d5d56d3c",
      "status": "Failed",
      "startTime": "2026-09-17T13:10:50.219697768Z",
      "endTime": "2026-09-17T13:39:57.0065588Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.38 is Partial (want Completed), started 23m32s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 7 time(s) (1 unknown, 6 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T13:22:10Z
  Ended:       2026-09-17T13:38:09Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2532096-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssphmmqqlnjk70h22r3i4bauan32tpt-e5j6t5h1p6n1t9u", pod="router-669df879db-lfkh8", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssphmmqqlnjk70h22r3i4bauan32tpt-e5j6t5h1p6n1t9u/router-669df879db-lfkh8 has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T13:42:29Z
  Ended:       2026-09-17T13:48:30Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j2532096-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j2532096-svc).

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--xw728ddtqcc2/cluster-hs-pre-bht88r should provision
Unexpected error:
    <*fmt.wrapError | 0xc000fe0440>: 
    failed to create HCP cluster cluster-hs-pre-bht88r: failed waiting for cluster="cluster-hs-pre-bht88r" in resourcegroup="rg-hs-pre--xw728ddtqcc2" to finish creating: GET https://rp.j6691200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/99f56462-8a68-42f5-b39b-3881c0af36e7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/99f56462-8a68-42f5-b39b-3881c0af36e7",
      "name": "99f56462-8a68-42f5-b39b-3881c0af36e7",
      "status": "Failed",
      "startTime": "2026-09-17T13:14:13.9724219Z",
      "endTime": "2026-09-17T13:43:14.263460227Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 19m36s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--xw728ddtqcc2/cluster-hs-pre-bht88r should provision
Unexpected error:
    <*fmt.wrapError | 0xc000fe0440>: 
    failed to create HCP cluster cluster-hs-pre-bht88r: failed waiting for cluster="cluster-hs-pre-bht88r" in resourcegroup="rg-hs-pre--xw728ddtqcc2" to finish creating: GET https://rp.j6691200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/99f56462-8a68-42f5-b39b-3881c0af36e7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/99f56462-8a68-42f5-b39b-3881c0af36e7",
      "name": "99f56462-8a68-42f5-b39b-3881c0af36e7",
      "status": "Failed",
      "startTime": "2026-09-17T13:14:13.9724219Z",
      "endTime": "2026-09-17T13:43:14.263460227Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 19m36s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc0006f47e0>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-rqblsgkhsl67" to finish creating: GET https://rp.j6691200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ffb5ae05-311d-47a9-81e8-a78c76770695
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ffb5ae05-311d-47a9-81e8-a78c76770695",
      "name": "ffb5ae05-311d-47a9-81e8-a78c76770695",
      "status": "Failed",
      "startTime": "2026-09-17T13:14:59.825714146Z",
      "endTime": "2026-09-17T13:44:04.017647611Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 19m0s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc0006f47e0>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-rqblsgkhsl67" to finish creating: GET https://rp.j6691200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ffb5ae05-311d-47a9-81e8-a78c76770695
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ffb5ae05-311d-47a9-81e8-a78c76770695",
      "name": "ffb5ae05-311d-47a9-81e8-a78c76770695",
      "status": "Failed",
      "startTime": "2026-09-17T13:14:59.825714146Z",
      "endTime": "2026-09-17T13:44:04.017647611Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 19m0s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000c0a6a0>: 
    failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-n5scb8djfw2f" to finish creating: GET https://rp.j6691200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a13212a1-a469-4833-ad96-5cf68dc1002e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a13212a1-a469-4833-ad96-5cf68dc1002e",
      "name": "a13212a1-a469-4833-ad96-5cf68dc1002e",
      "status": "Failed",
      "startTime": "2026-09-17T13:14:27.329387989Z",
      "endTime": "2026-09-17T13:43:34.147081491Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 20m52s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000c0a6a0>: 
    failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-n5scb8djfw2f" to finish creating: GET https://rp.j6691200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a13212a1-a469-4833-ad96-5cf68dc1002e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a13212a1-a469-4833-ad96-5cf68dc1002e",
      "name": "a13212a1-a469-4833-ad96-5cf68dc1002e",
      "status": "Failed",
      "startTime": "2026-09-17T13:14:27.329387989Z",
      "endTime": "2026-09-17T13:43:34.147081491Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 20m52s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:356]: Timed out after 300.007s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0xc000c05990>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:53121->172.30.0.10:53: read: connection refused
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:356]: Timed out after 300.007s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0xc000c05990>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:53121->172.30.0.10:53: read: connection refused
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-tsxbd6f5jxgs/cluster-candidate-4-21-6rv67k should provision
Unexpected error:
    <*fmt.wrapError | 0xc0017be420>: 
    failed to create HCP cluster cluster-candidate-4-21-6rv67k: failed waiting for cluster="cluster-candidate-4-21-6rv67k" in resourcegroup="rg-candidate-4-21-tsxbd6f5jxgs" to finish creating: GET https://rp.j6691200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bc291a9b-26c3-4ca5-924b-ef4256d35e27
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bc291a9b-26c3-4ca5-924b-ef4256d35e27",
      "name": "bc291a9b-26c3-4ca5-924b-ef4256d35e27",
      "status": "Failed",
      "startTime": "2026-09-17T13:14:50.324424123Z",
      "endTime": "2026-09-17T13:43:54.118510486Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 20m16s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-tsxbd6f5jxgs/cluster-candidate-4-21-6rv67k should provision
Unexpected error:
    <*fmt.wrapError | 0xc0017be420>: 
    failed to create HCP cluster cluster-candidate-4-21-6rv67k: failed waiting for cluster="cluster-candidate-4-21-6rv67k" in resourcegroup="rg-candidate-4-21-tsxbd6f5jxgs" to finish creating: GET https://rp.j6691200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bc291a9b-26c3-4ca5-924b-ef4256d35e27
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bc291a9b-26c3-4ca5-924b-ef4256d35e27",
      "name": "bc291a9b-26c3-4ca5-924b-ef4256d35e27",
      "status": "Failed",
      "startTime": "2026-09-17T13:14:50.324424123Z",
      "endTime": "2026-09-17T13:43:54.118510486Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 20m16s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000884200>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-pqfbdsfxgpwf" to finish creating: GET https://rp.j6691200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5b7c6859-32e0-4c13-b03c-200c651d5ba9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5b7c6859-32e0-4c13-b03c-200c651d5ba9",
      "name": "5b7c6859-32e0-4c13-b03c-200c651d5ba9",
      "status": "Failed",
      "startTime": "2026-09-17T13:15:12.801180509Z",
      "endTime": "2026-09-17T13:44:14.063627342Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 21m30s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000884200>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-pqfbdsfxgpwf" to finish creating: GET https://rp.j6691200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5b7c6859-32e0-4c13-b03c-200c651d5ba9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5b7c6859-32e0-4c13-b03c-200c651d5ba9",
      "name": "5b7c6859-32e0-4c13-b03c-200c651d5ba9",
      "status": "Failed",
      "startTime": "2026-09-17T13:15:12.801180509Z",
      "endTime": "2026-09-17T13:44:14.063627342Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 21m30s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T13:26:50Z
  Ended:       2026-09-17T13:43:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6691200-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspiucgjqnoq5us70um13m98tr1nrls-g3c5i5x9u6t6v1r", pod="router-69779865cd-8hd8n", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspiucgjqnoq5us70um13m98tr1nrls-g3c5i5x9u6t6v1r/router-69779865cd-8hd8n has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc0016a45a0>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-jvkrg2m7vz86" to finish creating: GET https://rp.j2652928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1daf1f66-7aa9-4611-bffb-22454b3ff87d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1daf1f66-7aa9-4611-bffb-22454b3ff87d",
      "name": "1daf1f66-7aa9-4611-bffb-22454b3ff87d",
      "status": "Failed",
      "startTime": "2026-09-17T13:09:14.976943123Z",
      "endTime": "2026-09-17T13:38:20.215505179Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 21m59s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc0016a45a0>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-jvkrg2m7vz86" to finish creating: GET https://rp.j2652928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1daf1f66-7aa9-4611-bffb-22454b3ff87d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1daf1f66-7aa9-4611-bffb-22454b3ff87d",
      "name": "1daf1f66-7aa9-4611-bffb-22454b3ff87d",
      "status": "Failed",
      "startTime": "2026-09-17T13:09:14.976943123Z",
      "endTime": "2026-09-17T13:38:20.215505179Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 21m59s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc00061ecc0>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-9z5zhn4wcxq7" to finish creating: GET https://rp.j2652928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/864bbcb9-28d5-49bd-9ab9-4a5ea088a8f3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/864bbcb9-28d5-49bd-9ab9-4a5ea088a8f3",
      "name": "864bbcb9-28d5-49bd-9ab9-4a5ea088a8f3",
      "status": "Failed",
      "startTime": "2026-09-17T13:10:00.288784653Z",
      "endTime": "2026-09-17T13:39:01.074596069Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 22m44s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc00061ecc0>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-9z5zhn4wcxq7" to finish creating: GET https://rp.j2652928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/864bbcb9-28d5-49bd-9ab9-4a5ea088a8f3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/864bbcb9-28d5-49bd-9ab9-4a5ea088a8f3",
      "name": "864bbcb9-28d5-49bd-9ab9-4a5ea088a8f3",
      "status": "Failed",
      "startTime": "2026-09-17T13:10:00.288784653Z",
      "endTime": "2026-09-17T13:39:01.074596069Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 22m44s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:80]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000df7160>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-2mtbkjvnnzmc" to finish creating: GET https://rp.j2652928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a0607f3e-d3c6-4002-afc3-b216aabd4af7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a0607f3e-d3c6-4002-afc3-b216aabd4af7",
      "name": "a0607f3e-d3c6-4002-afc3-b216aabd4af7",
      "status": "Failed",
      "startTime": "2026-09-17T13:09:52.721401557Z",
      "endTime": "2026-09-17T13:39:01.109176495Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m50s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:80]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000df7160>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-2mtbkjvnnzmc" to finish creating: GET https://rp.j2652928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a0607f3e-d3c6-4002-afc3-b216aabd4af7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a0607f3e-d3c6-4002-afc3-b216aabd4af7",
      "name": "a0607f3e-d3c6-4002-afc3-b216aabd4af7",
      "status": "Failed",
      "startTime": "2026-09-17T13:09:52.721401557Z",
      "endTime": "2026-09-17T13:39:01.109176495Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m50s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc000f6ade0>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-txp6prl987ll" to finish creating: GET https://rp.j2652928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c77e7129-6fb9-48eb-b5a7-009ab48c6967
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c77e7129-6fb9-48eb-b5a7-009ab48c6967",
      "name": "c77e7129-6fb9-48eb-b5a7-009ab48c6967",
      "status": "Failed",
      "startTime": "2026-09-17T13:08:46.266706852Z",
      "endTime": "2026-09-17T13:37:50.130353042Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m8s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc000f6ade0>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-txp6prl987ll" to finish creating: GET https://rp.j2652928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c77e7129-6fb9-48eb-b5a7-009ab48c6967
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c77e7129-6fb9-48eb-b5a7-009ab48c6967",
      "name": "c77e7129-6fb9-48eb-b5a7-009ab48c6967",
      "status": "Failed",
      "startTime": "2026-09-17T13:08:46.266706852Z",
      "endTime": "2026-09-17T13:37:50.130353042Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m8s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000db2f00>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-jtbcrf6fbph7" to finish creating: GET https://rp.j2652928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b5818a0a-e8dd-420e-9ceb-3bf8f83ab02f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b5818a0a-e8dd-420e-9ceb-3bf8f83ab02f",
      "name": "b5818a0a-e8dd-420e-9ceb-3bf8f83ab02f",
      "status": "Failed",
      "startTime": "2026-09-17T13:08:56.52781692Z",
      "endTime": "2026-09-17T13:38:00.207851281Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m7s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000db2f00>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-jtbcrf6fbph7" to finish creating: GET https://rp.j2652928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b5818a0a-e8dd-420e-9ceb-3bf8f83ab02f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b5818a0a-e8dd-420e-9ceb-3bf8f83ab02f",
      "name": "b5818a0a-e8dd-420e-9ceb-3bf8f83ab02f",
      "status": "Failed",
      "startTime": "2026-09-17T13:08:56.52781692Z",
      "endTime": "2026-09-17T13:38:00.207851281Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m7s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-20-b96ppk6wk4k9/cluster-candidate-4-20-7tcz5g should provision
Unexpected error:
    <*fmt.wrapError | 0xc00140a620>: 
    failed to create HCP cluster cluster-candidate-4-20-7tcz5g: failed waiting for cluster="cluster-candidate-4-20-7tcz5g" in resourcegroup="rg-candidate-4-20-b96ppk6wk4k9" to finish creating: GET https://rp.j2652928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bc41842e-dec4-4955-9bfc-4e2ff8b8270f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bc41842e-dec4-4955-9bfc-4e2ff8b8270f",
      "name": "bc41842e-dec4-4955-9bfc-4e2ff8b8270f",
      "status": "Failed",
      "startTime": "2026-09-17T13:08:46.930094659Z",
      "endTime": "2026-09-17T13:37:50.186246427Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.38 is Partial (want Completed), started 22m48s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-20-b96ppk6wk4k9/cluster-candidate-4-20-7tcz5g should provision
Unexpected error:
    <*fmt.wrapError | 0xc00140a620>: 
    failed to create HCP cluster cluster-candidate-4-20-7tcz5g: failed waiting for cluster="cluster-candidate-4-20-7tcz5g" in resourcegroup="rg-candidate-4-20-b96ppk6wk4k9" to finish creating: GET https://rp.j2652928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bc41842e-dec4-4955-9bfc-4e2ff8b8270f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bc41842e-dec4-4955-9bfc-4e2ff8b8270f",
      "name": "bc41842e-dec4-4955-9bfc-4e2ff8b8270f",
      "status": "Failed",
      "startTime": "2026-09-17T13:08:46.930094659Z",
      "endTime": "2026-09-17T13:37:50.186246427Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.38 is Partial (want Completed), started 22m48s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 6 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T13:23:01Z
  Ended:       2026-09-17T13:48:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2652928-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssph8fr577g8dbjaqg0tloglfhip8ib-y7i4q7g4c7s7i5z", pod="router-894997d67-7c5ck", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssph8fr577g8dbjaqg0tloglfhip8ib-y7i4q7g4c7s7i5z/router-894997d67-7c5ck has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-17T13:23:01Z
  Ended:       2026-09-17T13:48:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2652928-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspgnuu0iphnirf8pnuijesopcagti4-m2g9d8c5h8m2v1p", pod="router-954856bf9-78nmw", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspgnuu0iphnirf8pnuijesopcagti4-m2g9d8c5h8m2v1p/router-954856bf9-78nmw has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-17T13:23:01Z
  Ended:       2026-09-17T13:48:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2652928-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspgnp46398o3uuu74c13ce8b0gj7i2-v20260901", pod="router-bc9c99475-44czs", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspgnp46398o3uuu74c13ce8b0gj7i2-v20260901/router-bc9c99475-44czs has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-17T13:23:01Z
  Ended:       2026-09-17T13:46:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2652928-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspguqf18fp17m9t6tqvviau001p1ol-r7d2n7a9f0o0l3p", pod="router-69689b7c68-b85dk", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspguqf18fp17m9t6tqvviau001p1ol-r7d2n7a9f0o0l3p/router-69689b7c68-b85dk has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-09-17T13:23:01Z
  Ended:       2026-09-17T13:48:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2652928-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspgqdlc1rn1r1tm8lpqprm3tucttvj-private-ingress", pod="router-666f48c4b5-ksh9p", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspgqdlc1rn1r1tm8lpqprm3tucttvj-private-ingress/router-666f48c4b5-ksh9p has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-09-17T13:23:01Z
  Ended:       2026-09-17T13:48:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2652928-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspha5kdni12tmq452o97bfrlna4nmg-private-kas", pod="router-5cff7b7494-tl542", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspha5kdni12tmq452o97bfrlna4nmg-private-kas/router-5cff7b7494-tl542 has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-20-srrxr479d7ch/cluster-candidate-4-20-rh4pb8 should provision
Unexpected error:
    <*fmt.wrapError | 0xc0009e3940>: 
    failed to create HCP cluster cluster-candidate-4-20-rh4pb8: failed waiting for cluster="cluster-candidate-4-20-rh4pb8" in resourcegroup="rg-candidate-4-20-srrxr479d7ch" to finish creating: GET https://rp.j5507968.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/38bd640e-c367-4ef7-a27c-0a98287d687e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/38bd640e-c367-4ef7-a27c-0a98287d687e",
      "name": "38bd640e-c367-4ef7-a27c-0a98287d687e",
      "status": "Failed",
      "startTime": "2026-09-17T13:10:10.863639821Z",
      "endTime": "2026-09-17T13:39:18.855692604Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.38 is Partial (want Completed), started 23m14s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-20-srrxr479d7ch/cluster-candidate-4-20-rh4pb8 should provision
Unexpected error:
    <*fmt.wrapError | 0xc0009e3940>: 
    failed to create HCP cluster cluster-candidate-4-20-rh4pb8: failed waiting for cluster="cluster-candidate-4-20-rh4pb8" in resourcegroup="rg-candidate-4-20-srrxr479d7ch" to finish creating: GET https://rp.j5507968.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/38bd640e-c367-4ef7-a27c-0a98287d687e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/38bd640e-c367-4ef7-a27c-0a98287d687e",
      "name": "38bd640e-c367-4ef7-a27c-0a98287d687e",
      "status": "Failed",
      "startTime": "2026-09-17T13:10:10.863639821Z",
      "endTime": "2026-09-17T13:39:18.855692604Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.38 is Partial (want Completed), started 23m14s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T13:24:17Z
  Ended:       2026-09-17T13:48:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5507968-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssphd4ggk7ujcv9c3mpfi14jorh70dh-a6v8w8w3u5q9y3a", pod="router-7b5888dd5c-hmrr8", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssphd4ggk7ujcv9c3mpfi14jorh70dh-a6v8w8w3u5q9y3a/router-7b5888dd5c-hmrr8 has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/ksm_hcp_metrics.go:85]: Timed out after 900.000s.
kube_node_info metrics never appeared in Azure Monitor for any hostedcontrolplane
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/ksm_hcp_metrics.go:83 with:
expected kube_node_info metrics for at least one hostedcontrolplane but got no results
Expected
    <[]prometheus.Result | len:0, cap:0>: ...
not to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/ksm_hcp_metrics.go:85]: Timed out after 900.000s.
kube_node_info metrics never appeared in Azure Monitor for any hostedcontrolplane
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/ksm_hcp_metrics.go:83 with:
expected kube_node_info metrics for at least one hostedcontrolplane but got no results
Expected
    <[]prometheus.Result | len:0, cap:0>: ...
not to be empty

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:101]: Timed out after 600.007s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc000aac060>: 
    failed to get kusto cluster name from config: configuration: key kusto not found
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:101]: Timed out after 600.007s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc000aac060>: 
    failed to get kusto cluster name from config: configuration: key kusto not found
    ...

alert fired 36 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T13:08:59Z
  Ended:       2026-09-17T13:33:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5128576-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspcb7sqsldk7bi5dnp0oa4r7mbmdgt-k9m1g1q6v1l7j4e", pod="kube-state-metrics-hcp-8654bffc85-sw4zr", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspcb7sqsldk7bi5dnp0oa4r7mbmdgt-k9m1g1q6v1l7j4e/kube-state-metrics-hcp-8654bffc85-sw4zr has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-17T13:08:59Z
  Ended:       2026-09-17T13:29:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5128576-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspc83v54i8b9ktds0mno1jqu66puk8-a3i1c3s3v5b7y2b", pod="kube-state-metrics-hcp-8654bffc85-vj9bl", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspc83v54i8b9ktds0mno1jqu66puk8-a3i1c3s3v5b7y2b/kube-state-metrics-hcp-8654bffc85-vj9bl has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-17T13:08:59Z
  Ended:       2026-09-17T13:28:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5128576-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspcab6g26b61ugvq1qa8er5s26ukr5-n8g9h1m4z5h0y5j", pod="kube-state-metrics-hcp-8654bffc85-84h9q", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspcab6g26b61ugvq1qa8er5s26ukr5-n8g9h1m4z5h0y5j/kube-state-metrics-hcp-8654bffc85-84h9q has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-17T13:08:59Z
  Ended:       2026-09-17T13:34:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5128576-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspcd1ibd9bsldcjivfv1pv891lupgb-g3q4b0s4i5m4w8a", pod="kube-state-metrics-hcp-8654bffc85-74h7f", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspcd1ibd9bsldcjivfv1pv891lupgb-g3q4b0s4i5m4w8a/kube-state-metrics-hcp-8654bffc85-74h7f has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-09-17T13:08:59Z
  Ended:       2026-09-17T13:30:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5128576-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspcd5pitmepf4tthfr105bf0u6qjkj-l6d4u3q2l8k1p4w", pod="kube-state-metrics-hcp-8654bffc85-gl5hc", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspcd5pitmepf4tthfr105bf0u6qjkj-l6d4u3q2l8k1p4w/kube-state-metrics-hcp-8654bffc85-gl5hc has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-09-17T13:10:00Z
  Ended:       2026-09-17T13:49:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5128576-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspch2tl2fng9je6t8uouikhvml9pr5-y6x5d3t0a0o9h1v", pod="kube-state-metrics-hcp-8654bffc85-w544w", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspch2tl2fng9je6t8uouikhvml9pr5-y6x5d3t0a0o9h1v/kube-state-metrics-hcp-8654bffc85-w544w has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-09-17T13:10:00Z
  Ended:       2026-09-17T13:30:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5128576-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspca9memkr10lu9ab1oaph4b2tmsjc-v3o4o5x5g9j2j1f", pod="kube-state-metrics-hcp-8654bffc85-qxmgb", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspca9memkr10lu9ab1oaph4b2tmsjc-v3o4o5x5g9j2j1f/kube-state-metrics-hcp-8654bffc85-qxmgb has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-09-17T13:10:56Z
  Ended:       2026-09-17T13:28:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5128576-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspcd48a8v6hh0ifl98g9db1ua89n14-sre-hcp-cluster", pod="kube-state-metrics-hcp-8654bffc85-fjj8q", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspcd48a8v6hh0ifl98g9db1ua89n14-sre-hcp-cluster/kube-state-metrics-hcp-8654bffc85-fjj8q has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-09-17T13:10:56Z
  Ended:       2026-09-17T13:37:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5128576-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspcfk8ph0gcjmvrcofvhgjqg35bstr-q8r0w4i7h4v4z5u", pod="kube-state-metrics-hcp-8654bffc85-2c7r8", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspcfk8ph0gcjmvrcofvhgjqg35bstr-q8r0w4i7h4v4z5u/kube-state-metrics-hcp-8654bffc85-2c7r8 has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-09-17T13:10:56Z
  Ended:       2026-09-17T13:18:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5128576-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspcf7ksaqkttkkr2cgq4nadfq3tol1-a5h0f7c3a0r6a1o", pod="kube-state-metrics-hcp-8654bffc85-lrbq6", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspcf7ksaqkttkkr2cgq4nadfq3tol1-a5h0f7c3a0r6a1o/kube-state-metrics-hcp-8654bffc85-lrbq6 has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-09-17T13:10:56Z
  Ended:       2026-09-17T13:39:57Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5128576-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspcaho1pdo9bnsb9380imd1q0cgsht-n6s0o9o5z7w5l1x", pod="kube-state-metrics-hcp-8654bffc85-68k4l", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspcaho1pdo9bnsb9380imd1q0cgsht-n6s0o9o5z7w5l1x/kube-state-metrics-hcp-8654bffc85-68k4l has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-09-17T13:10:56Z
  Ended:       2026-09-17T13:48:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5128576-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspcm0fc6pbiglveskvt7o8lqpfe1ni-q7t6j5c5r6b6e3x", pod="kube-state-metrics-hcp-8654bffc85-25xjz", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspcm0fc6pbiglveskvt7o8lqpfe1ni-q7t6j5c5r6b6e3x/kube-state-metrics-hcp-8654bffc85-25xjz has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-09-17T13:10:56Z
  Ended:       2026-09-17T13:50:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5128576-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspceeeteqt56ehbc42dgm3btdci716-x9h9l3q6w6d2q0m", pod="kube-state-metrics-hcp-8654bffc85-v45vb", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspceeeteqt56ehbc42dgm3btdci716-x9h9l3q6w6d2q0m/kube-state-metrics-hcp-8654bffc85-v45vb has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-09-17T13:10:56Z
  Ended:       2026-09-17T13:47:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5128576-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspcem0sa8hpfqifjev3vv7bold7e92-a5w3q3w8f1o3x9b", pod="kube-state-metrics-hcp-8654bffc85-dxx9w", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspcem0sa8hpfqifjev3vv7bold7e92-a5w3q3w8f1o3x9b/kube-state-metrics-hcp-8654bffc85-dxx9w has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-09-17T13:10:56Z
  Ended:       2026-09-17T13:34:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5128576-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspcge206hudf057d4rapvtvmgma2mo-oidc-wi-cluster", pod="kube-state-metrics-hcp-8654bffc85-hckmt", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspcge206hudf057d4rapvtvmgma2mo-oidc-wi-cluster/kube-state-metrics-hcp-8654bffc85-hckmt has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-09-17T13:10:56Z
  Ended:       2026-09-17T13:31:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5128576-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspccnril3e7pdaev0lngoai5csi2ku-ea-cluster", pod="kube-state-metrics-hcp-8654bffc85-pbjqj", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspccnril3e7pdaev0lngoai5csi2ku-ea-cluster/kube-state-metrics-hcp-8654bffc85-pbjqj has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Resolved
  Started:     2026-09-17T13:10:56Z
  Ended:       2026-09-17T13:19:57Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5128576-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspcm4o5ijtniiue45eh50id70dfp6a-f9t9u0r3u1b9n0r", pod="kube-state-metrics-hcp-8654bffc85-w8w27", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspcm4o5ijtniiue45eh50id70dfp6a-f9t9u0r3u1b9n0r/kube-state-metrics-hcp-8654bffc85-w8w27 has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Resolved
  Started:     2026-09-17T13:10:56Z
  Ended:       2026-09-17T13:18:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5128576-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspca8036n0fivuel444dlkbothfifs-basic-cluster", pod="kube-state-metrics-hcp-8654bffc85-47zwq", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspca8036n0fivuel444dlkbothfifs-basic-cluster/kube-state-metrics-hcp-8654bffc85-47zwq has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Resolved
  Started:     2026-09-17T13:10:56Z
  Ended:       2026-09-17T13:37:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5128576-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspcabjfnqjhlbhj0oghprl8hrp1oio-c6z8w3p7h3e9r9m", pod="kube-state-metrics-hcp-8654bffc85-pwt7b", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspcabjfnqjhlbhj0oghprl8hrp1oio-c6z8w3p7h3e9r9m/kube-state-metrics-hcp-8654bffc85-pwt7b has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Resolved
  Started:     2026-09-17T13:10:56Z
  Ended:       2026-09-17T13:37:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5128576-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspccnmknrpuh0747ob6ahb5jenvu9k-r4g1h9p4c5j0q6i", pod="kube-state-metrics-hcp-8654bffc85-5rw4j", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspccnmknrpuh0747ob6ahb5jenvu9k-r4g1h9p4c5j0q6i/kube-state-metrics-hcp-8654bffc85-5rw4j has been in a non-ready state for longer than 5 minutes.
Firing 21:
  State:       Resolved
  Started:     2026-09-17T13:10:57Z
  Ended:       2026-09-17T13:19:57Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5128576-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspcmkp48ol9sgagi11b09j67pb9ul7-l0x7x1b4p1n8t9g", pod="kube-state-metrics-hcp-8654bffc85-nwbg8", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspcmkp48ol9sgagi11b09j67pb9ul7-l0x7x1b4p1n8t9g/kube-state-metrics-hcp-8654bffc85-nwbg8 has been in a non-ready state for longer than 5 minutes.
Firing 22:
  State:       Resolved
  Started:     2026-09-17T13:10:57Z
  Ended:       2026-09-17T13:26:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5128576-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspccfhe1egj635kk9b5u2sg6m1iqmv-ea-listwgh4rg", pod="kube-state-metrics-hcp-8654bffc85-mldrv", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspccfhe1egj635kk9b5u2sg6m1iqmv-ea-listwgh4rg/kube-state-metrics-hcp-8654bffc85-mldrv has been in a non-ready state for longer than 5 minutes.
Firing 23:
  State:       Resolved
  Started:     2026-09-17T13:10:57Z
  Ended:       2026-09-17T13:19:57Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5128576-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspcmrjnf8qshgsel2n8pu5vcrqo8ib-p2l2b5d6x5j0p7f", pod="kube-state-metrics-hcp-8654bffc85-29p94", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspcmrjnf8qshgsel2n8pu5vcrqo8ib-p2l2b5d6x5j0p7f/kube-state-metrics-hcp-8654bffc85-29p94 has been in a non-ready state for longer than 5 minutes.
Firing 24:
  State:       Resolved
  Started:     2026-09-17T13:10:57Z
  Ended:       2026-09-17T13:41:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5128576-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspcgs41mednuce8v8juoarvre0o7f1-y1p7a8c3l5z0p6s", pod="kube-state-metrics-hcp-8654bffc85-fdbbb", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspcgs41mednuce8v8juoarvre0o7f1-y1p7a8c3l5z0p6s/kube-state-metrics-hcp-8654bffc85-fdbbb has been in a non-ready state for longer than 5 minutes.
Firing 25:
  State:       Resolved
  Started:     2026-09-17T13:12:00Z
  Ended:       2026-09-17T13:46:56Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5128576-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspcee5col9vs3ec1hc99u0e0cg6rod-l5t0c4o8b7t1m6k", pod="kube-state-metrics-hcp-8654bffc85-cht7p", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspcee5col9vs3ec1hc99u0e0cg6rod-l5t0c4o8b7t1m6k/kube-state-metrics-hcp-8654bffc85-cht7p has been in a non-ready state for longer than 5 minutes.
Firing 26:
  State:       Resolved
  Started:     2026-09-17T13:12:00Z
  Ended:       2026-09-17T13:42:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5128576-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspdfg78lat8k87fj5b9k4jmbs8qicd-x6v7h0p9g2z4c9r", pod="kube-state-metrics-hcp-8654bffc85-sg492", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspdfg78lat8k87fj5b9k4jmbs8qicd-x6v7h0p9g2z4c9r/kube-state-metrics-hcp-8654bffc85-sg492 has been in a non-ready state for longer than 5 minutes.
Firing 27:
  State:       Resolved
  Started:     2026-09-17T13:12:00Z
  Ended:       2026-09-17T13:49:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5128576-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspcen9oo9lcu99o5iueg61nfrsive0-v5c8o3g3i0h7h3b", pod="kube-state-metrics-hcp-8654bffc85-rrg4m", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspcen9oo9lcu99o5iueg61nfrsive0-v5c8o3g3i0h7h3b/kube-state-metrics-hcp-8654bffc85-rrg4m has been in a non-ready state for longer than 5 minutes.
Firing 28:
  State:       Resolved
  Started:     2026-09-17T13:12:00Z
  Ended:       2026-09-17T13:34:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5128576-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspccsnbr6b8rqfb4lsjsi86jn0doel-cilium-cl", pod="kube-state-metrics-hcp-8654bffc85-5twhb", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspccsnbr6b8rqfb4lsjsi86jn0doel-cilium-cl/kube-state-metrics-hcp-8654bffc85-5twhb has been in a non-ready state for longer than 5 minutes.
Firing 29:
  State:       Resolved
  Started:     2026-09-17T13:12:00Z
  Ended:       2026-09-17T13:41:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5128576-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspccdb2mcj9vfuqhqc0eshri0b51gk-c1n4i1l5h3v3w5y", pod="kube-state-metrics-hcp-8654bffc85-8tkwq", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspccdb2mcj9vfuqhqc0eshri0b51gk-c1n4i1l5h3v3w5y/kube-state-metrics-hcp-8654bffc85-8tkwq has been in a non-ready state for longer than 5 minutes.
Firing 30:
  State:       Resolved
  Started:     2026-09-17T13:12:00Z
  Ended:       2026-09-17T13:48:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5128576-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspcm0k5ga93uqot5b5roh4hu0pno7l-b3e0v4h4q2l7f4x", pod="kube-state-metrics-hcp-8654bffc85-mbg2n", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspcm0k5ga93uqot5b5roh4hu0pno7l-b3e0v4h4q2l7f4x/kube-state-metrics-hcp-8654bffc85-mbg2n has been in a non-ready state for longer than 5 minutes.
Firing 31:
  State:       Resolved
  Started:     2026-09-17T13:12:00Z
  Ended:       2026-09-17T13:39:57Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5128576-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspcf3m7huj2ui7lb82bh163g9kpqpp-s6b5r7u6f2z2f3z", pod="kube-state-metrics-hcp-8654bffc85-twgnb", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspcf3m7huj2ui7lb82bh163g9kpqpp-s6b5r7u6f2z2f3z/kube-state-metrics-hcp-8654bffc85-twgnb has been in a non-ready state for longer than 5 minutes.
Firing 32:
  State:       Resolved
  Started:     2026-09-17T13:12:00Z
  Ended:       2026-09-17T13:25:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5128576-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspccu0f6eg3g5158smlj1ia6m6j77b-t1u8x2j7n4m3n7v", pod="kube-state-metrics-hcp-8654bffc85-smqcz", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspccu0f6eg3g5158smlj1ia6m6j77b-t1u8x2j7n4m3n7v/kube-state-metrics-hcp-8654bffc85-smqcz has been in a non-ready state for longer than 5 minutes.
Firing 33:
  State:       Resolved
  Started:     2026-09-17T13:12:00Z
  Ended:       2026-09-17T13:29:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5128576-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspccsal4ptn0kdvo8ga8voqcluuqn5-g1i8q2w1x1z3q4o", pod="kube-state-metrics-hcp-8654bffc85-7pnz5", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspccsal4ptn0kdvo8ga8voqcluuqn5-g1i8q2w1x1z3q4o/kube-state-metrics-hcp-8654bffc85-7pnz5 has been in a non-ready state for longer than 5 minutes.
Firing 34:
  State:       Resolved
  Started:     2026-09-17T13:12:00Z
  Ended:       2026-09-17T13:41:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5128576-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspcekf3ud0i2iqs827rrhieggpvun6-t3z2j3v2v1y5a6t", pod="kube-state-metrics-hcp-8654bffc85-df2sj", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspcekf3ud0i2iqs827rrhieggpvun6-t3z2j3v2v1y5a6t/kube-state-metrics-hcp-8654bffc85-df2sj has been in a non-ready state for longer than 5 minutes.
Firing 35:
  State:       Resolved
  Started:     2026-09-17T13:12:00Z
  Ended:       2026-09-17T13:31:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5128576-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspclmnibunjprapdm7j3gamrhiduda-d2i3i0b5r0s4j5j", pod="kube-state-metrics-hcp-8654bffc85-prqn6", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspclmnibunjprapdm7j3gamrhiduda-d2i3i0b5r0s4j5j/kube-state-metrics-hcp-8654bffc85-prqn6 has been in a non-ready state for longer than 5 minutes.
Firing 36:
  State:       Resolved
  Started:     2026-09-17T13:12:00Z
  Ended:       2026-09-17T13:36:56Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5128576-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspcfko5q3bri1mmij2t32jidfdi6ri-t5a3k3q0r9w8x3a", pod="kube-state-metrics-hcp-8654bffc85-bpdfw", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspcfko5q3bri1mmij2t32jidfdi6ri-t5a3k3q0r9w8x3a/kube-state-metrics-hcp-8654bffc85-bpdfw has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001790120>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-8928t8c2w987" to finish deleting: GET https://rp.j4930944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7f319ad2-39f3-41e7-b4bb-2445dadbddeb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7f319ad2-39f3-41e7-b4bb-2445dadbddeb",
      "name": "7f319ad2-39f3-41e7-b4bb-2445dadbddeb",
      "status": "Failed",
      "startTime": "2026-09-17T13:04:21.545838494Z",
      "endTime": "2026-09-17T13:28:30.638150402Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssp95g2hh8pms8h7312ekq6dl4tr17v still exists (deletion dispatched at 2026-09-17T13:04:23Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001790120>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-8928t8c2w987" to finish deleting: GET https://rp.j4930944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7f319ad2-39f3-41e7-b4bb-2445dadbddeb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7f319ad2-39f3-41e7-b4bb-2445dadbddeb",
      "name": "7f319ad2-39f3-41e7-b4bb-2445dadbddeb",
      "status": "Failed",
      "startTime": "2026-09-17T13:04:21.545838494Z",
      "endTime": "2026-09-17T13:28:30.638150402Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssp95g2hh8pms8h7312ekq6dl4tr17v still exists (deletion dispatched at 2026-09-17T13:04:23Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0011b4600>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-8sx86dmkm86k" to finish deleting: GET https://rp.j4930944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/13066ffb-8ec0-49b0-91b1-8f95cc778712
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/13066ffb-8ec0-49b0-91b1-8f95cc778712",
      "name": "13066ffb-8ec0-49b0-91b1-8f95cc778712",
      "status": "Failed",
      "startTime": "2026-09-17T13:13:55.616199714Z",
      "endTime": "2026-09-17T13:38:00.670848783Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssp9797225e4pc4eg93kcbplgmr3noo still exists (deletion dispatched at 2026-09-17T13:14:07Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0011b4600>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-8sx86dmkm86k" to finish deleting: GET https://rp.j4930944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/13066ffb-8ec0-49b0-91b1-8f95cc778712
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/13066ffb-8ec0-49b0-91b1-8f95cc778712",
      "name": "13066ffb-8ec0-49b0-91b1-8f95cc778712",
      "status": "Failed",
      "startTime": "2026-09-17T13:13:55.616199714Z",
      "endTime": "2026-09-17T13:38:00.670848783Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssp9797225e4pc4eg93kcbplgmr3noo still exists (deletion dispatched at 2026-09-17T13:14:07Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0027021e0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-fvq6xfsmv5rt" to finish deleting: GET https://rp.j4930944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/06520f2e-e19c-4091-a068-c1fb58d73111
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/06520f2e-e19c-4091-a068-c1fb58d73111",
      "name": "06520f2e-e19c-4091-a068-c1fb58d73111",
      "status": "Failed",
      "startTime": "2026-09-17T13:03:13.698113353Z",
      "endTime": "2026-09-17T13:27:20.680352254Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssp7jvtggrghuov8k46cvi5s0t83kgc still exists (deletion dispatched at 2026-09-17T13:03:16Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0027021e0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-fvq6xfsmv5rt" to finish deleting: GET https://rp.j4930944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/06520f2e-e19c-4091-a068-c1fb58d73111
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/06520f2e-e19c-4091-a068-c1fb58d73111",
      "name": "06520f2e-e19c-4091-a068-c1fb58d73111",
      "status": "Failed",
      "startTime": "2026-09-17T13:03:13.698113353Z",
      "endTime": "2026-09-17T13:27:20.680352254Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssp7jvtggrghuov8k46cvi5s0t83kgc still exists (deletion dispatched at 2026-09-17T13:03:16Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000012e40>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cl" in resourcegroup="cni-cilium-67hdvl9w4j6q" to finish deleting: GET https://rp.j4930944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4599e3d2-e4d0-4e28-8c3a-e6618ada2d97
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4599e3d2-e4d0-4e28-8c3a-e6618ada2d97",
      "name": "4599e3d2-e4d0-4e28-8c3a-e6618ada2d97",
      "status": "Failed",
      "startTime": "2026-09-17T13:11:15.532571894Z",
      "endTime": "2026-09-17T13:35:20.695248589Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssp8bghu2kio5ia0opa50hpqiea2oa0 still exists (deletion dispatched at 2026-09-17T13:11:20Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000012e40>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cl" in resourcegroup="cni-cilium-67hdvl9w4j6q" to finish deleting: GET https://rp.j4930944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4599e3d2-e4d0-4e28-8c3a-e6618ada2d97
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4599e3d2-e4d0-4e28-8c3a-e6618ada2d97",
      "name": "4599e3d2-e4d0-4e28-8c3a-e6618ada2d97",
      "status": "Failed",
      "startTime": "2026-09-17T13:11:15.532571894Z",
      "endTime": "2026-09-17T13:35:20.695248589Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssp8bghu2kio5ia0opa50hpqiea2oa0 still exists (deletion dispatched at 2026-09-17T13:11:20Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001226768>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-22-jwdh6h" in resourcegroup="rg-candidate-4-22-9rtkhgwpwkdp" to finish deleting: GET https://rp.j4930944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/497b563d-6a90-4107-8ea2-18e801b64cd8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/497b563d-6a90-4107-8ea2-18e801b64cd8",
      "name": "497b563d-6a90-4107-8ea2-18e801b64cd8",
      "status": "Failed",
      "startTime": "2026-09-17T13:09:44.365139487Z",
      "endTime": "2026-09-17T13:33:50.717397424Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssp7gelb6kbkk61h7dh9rg6bgds1mn2 still exists (deletion dispatched at 2026-09-17T13:09:47Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001226768>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-22-jwdh6h" in resourcegroup="rg-candidate-4-22-9rtkhgwpwkdp" to finish deleting: GET https://rp.j4930944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/497b563d-6a90-4107-8ea2-18e801b64cd8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/497b563d-6a90-4107-8ea2-18e801b64cd8",
      "name": "497b563d-6a90-4107-8ea2-18e801b64cd8",
      "status": "Failed",
      "startTime": "2026-09-17T13:09:44.365139487Z",
      "endTime": "2026-09-17T13:33:50.717397424Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssp7gelb6kbkk61h7dh9rg6bgds1mn2 still exists (deletion dispatched at 2026-09-17T13:09:47Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00123e3a8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-20-qzhgv4" in resourcegroup="rg-candidate-4-20-w4h7vw7ct2fr" to finish deleting: GET https://rp.j4930944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6fd48b8c-0627-4555-8c0e-d37ed82eb815
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6fd48b8c-0627-4555-8c0e-d37ed82eb815",
      "name": "6fd48b8c-0627-4555-8c0e-d37ed82eb815",
      "status": "Failed",
      "startTime": "2026-09-17T13:04:19.914202757Z",
      "endTime": "2026-09-17T13:28:20.692107951Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssp7811gchpng82erpcrdlvqh636u2b still exists (deletion dispatched at 2026-09-17T13:04:21Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00123e3a8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-20-qzhgv4" in resourcegroup="rg-candidate-4-20-w4h7vw7ct2fr" to finish deleting: GET https://rp.j4930944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6fd48b8c-0627-4555-8c0e-d37ed82eb815
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6fd48b8c-0627-4555-8c0e-d37ed82eb815",
      "name": "6fd48b8c-0627-4555-8c0e-d37ed82eb815",
      "status": "Failed",
      "startTime": "2026-09-17T13:04:19.914202757Z",
      "endTime": "2026-09-17T13:28:20.692107951Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssp7811gchpng82erpcrdlvqh636u2b still exists (deletion dispatched at 2026-09-17T13:04:21Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0014980d8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="delayed-rbac-cluster" in resourcegroup="delayed-rbac-j826q67vvj52" to finish deleting: GET https://rp.j4930944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/280170cf-c530-4d72-9037-9f04f2905802
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/280170cf-c530-4d72-9037-9f04f2905802",
      "name": "280170cf-c530-4d72-9037-9f04f2905802",
      "status": "Failed",
      "startTime": "2026-09-17T13:00:52.342655089Z",
      "endTime": "2026-09-17T13:25:00.74293906Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssp73alrk9qecl63aq268esm2m41s6e still exists (deletion dispatched at 2026-09-17T13:00:54Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0014980d8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="delayed-rbac-cluster" in resourcegroup="delayed-rbac-j826q67vvj52" to finish deleting: GET https://rp.j4930944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/280170cf-c530-4d72-9037-9f04f2905802
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/280170cf-c530-4d72-9037-9f04f2905802",
      "name": "280170cf-c530-4d72-9037-9f04f2905802",
      "status": "Failed",
      "startTime": "2026-09-17T13:00:52.342655089Z",
      "endTime": "2026-09-17T13:25:00.74293906Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssp73alrk9qecl63aq268esm2m41s6e still exists (deletion dispatched at 2026-09-17T13:00:54Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00102e210>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-noedge-r9k66x" in resourcegroup="rg-np-noedge-r9k66x-b6q65mpdbrdl" to finish deleting: GET https://rp.j4930944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4993b33a-2f46-49c4-9ddc-e8dfd573aadd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4993b33a-2f46-49c4-9ddc-e8dfd573aadd",
      "name": "4993b33a-2f46-49c4-9ddc-e8dfd573aadd",
      "status": "Failed",
      "startTime": "2026-09-17T13:07:12.283709141Z",
      "endTime": "2026-09-17T13:31:20.695028077Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssp7ju1gnqcl4frelh23hftlr0i6s8h still exists (deletion dispatched at 2026-09-17T13:07:13Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00102e210>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-noedge-r9k66x" in resourcegroup="rg-np-noedge-r9k66x-b6q65mpdbrdl" to finish deleting: GET https://rp.j4930944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4993b33a-2f46-49c4-9ddc-e8dfd573aadd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4993b33a-2f46-49c4-9ddc-e8dfd573aadd",
      "name": "4993b33a-2f46-49c4-9ddc-e8dfd573aadd",
      "status": "Failed",
      "startTime": "2026-09-17T13:07:12.283709141Z",
      "endTime": "2026-09-17T13:31:20.695028077Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssp7ju1gnqcl4frelh23hftlr0i6s8h still exists (deletion dispatched at 2026-09-17T13:07:13Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0012b0228>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ingress-cert-le" in resourcegroup="ingress-cert-le-q27rc669zqsh" to finish deleting: GET https://rp.j4930944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7fd21058-2067-490c-af21-4f8dcf7eaa2d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7fd21058-2067-490c-af21-4f8dcf7eaa2d",
      "name": "7fd21058-2067-490c-af21-4f8dcf7eaa2d",
      "status": "Failed",
      "startTime": "2026-09-17T13:18:15.92690988Z",
      "endTime": "2026-09-17T13:42:20.66158476Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssp9ceiqjg99tofkj78h4d9u0bhnv3l still exists (deletion dispatched at 2026-09-17T13:18:17Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0012b0228>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ingress-cert-le" in resourcegroup="ingress-cert-le-q27rc669zqsh" to finish deleting: GET https://rp.j4930944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7fd21058-2067-490c-af21-4f8dcf7eaa2d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7fd21058-2067-490c-af21-4f8dcf7eaa2d",
      "name": "7fd21058-2067-490c-af21-4f8dcf7eaa2d",
      "status": "Failed",
      "startTime": "2026-09-17T13:18:15.92690988Z",
      "endTime": "2026-09-17T13:42:20.66158476Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssp9ceiqjg99tofkj78h4d9u0bhnv3l still exists (deletion dispatched at 2026-09-17T13:18:17Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001ddc1b0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-n5x5stk2jr2b" to finish deleting: GET https://rp.j4930944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/581beceb-4aec-4acd-855a-b3a0fe1b475e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/581beceb-4aec-4acd-855a-b3a0fe1b475e",
      "name": "581beceb-4aec-4acd-855a-b3a0fe1b475e",
      "status": "Failed",
      "startTime": "2026-09-17T13:03:11.454975969Z",
      "endTime": "2026-09-17T13:27:20.747941158Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssp7847nq93qpgrlmug2nh4ioibnstg still exists (deletion dispatched at 2026-09-17T13:03:15Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001ddc1b0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-n5x5stk2jr2b" to finish deleting: GET https://rp.j4930944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/581beceb-4aec-4acd-855a-b3a0fe1b475e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/581beceb-4aec-4acd-855a-b3a0fe1b475e",
      "name": "581beceb-4aec-4acd-855a-b3a0fe1b475e",
      "status": "Failed",
      "startTime": "2026-09-17T13:03:11.454975969Z",
      "endTime": "2026-09-17T13:27:20.747941158Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssp7847nq93qpgrlmug2nh4ioibnstg still exists (deletion dispatched at 2026-09-17T13:03:15Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001002300>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-downgrade-rs72gd" in resourcegroup="rg-np-downgrade-rs72gd-h2sg5z57shg4" to finish deleting: GET https://rp.j4930944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cc3d9638-28ce-4c55-a9f7-10d6903cf30f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cc3d9638-28ce-4c55-a9f7-10d6903cf30f",
      "name": "cc3d9638-28ce-4c55-a9f7-10d6903cf30f",
      "status": "Failed",
      "startTime": "2026-09-17T13:18:23.650124176Z",
      "endTime": "2026-09-17T13:42:30.681443783Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssp9bc7t4br61qj7lck2cv5908qu6rv still exists (deletion dispatched at 2026-09-17T13:18:25Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001002300>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-downgrade-rs72gd" in resourcegroup="rg-np-downgrade-rs72gd-h2sg5z57shg4" to finish deleting: GET https://rp.j4930944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cc3d9638-28ce-4c55-a9f7-10d6903cf30f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cc3d9638-28ce-4c55-a9f7-10d6903cf30f",
      "name": "cc3d9638-28ce-4c55-a9f7-10d6903cf30f",
      "status": "Failed",
      "startTime": "2026-09-17T13:18:23.650124176Z",
      "endTime": "2026-09-17T13:42:30.681443783Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssp9bc7t4br61qj7lck2cv5908qu6rv still exists (deletion dispatched at 2026-09-17T13:18:25Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001a16108>: 
    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-cluster-back-version-q8l56f6rlm84", caused by: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-q8l56f6rlm84" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-q8l56f6rlm84, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-q8l56f6rlm84" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-q8l56f6rlm84, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001a16108>: 
    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-cluster-back-version-q8l56f6rlm84", caused by: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-q8l56f6rlm84" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-q8l56f6rlm84, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-q8l56f6rlm84" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-q8l56f6rlm84, error: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:89]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc000e8c940>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-rsjc9lcm2kdb" to finish creating: GET https://rp.j4979456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/08ee0aa6-f26c-4889-ab77-6ed9a097e06b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/08ee0aa6-f26c-4889-ab77-6ed9a097e06b",
      "name": "08ee0aa6-f26c-4889-ab77-6ed9a097e06b",
      "status": "Failed",
      "startTime": "2026-09-17T12:56:47.419533085Z",
      "endTime": "2026-09-17T13:25:52.286605579Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 22m8s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:89]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc000e8c940>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-rsjc9lcm2kdb" to finish creating: GET https://rp.j4979456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/08ee0aa6-f26c-4889-ab77-6ed9a097e06b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/08ee0aa6-f26c-4889-ab77-6ed9a097e06b",
      "name": "08ee0aa6-f26c-4889-ab77-6ed9a097e06b",
      "status": "Failed",
      "startTime": "2026-09-17T12:56:47.419533085Z",
      "endTime": "2026-09-17T13:25:52.286605579Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 22m8s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:127]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc0009e8ec0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-qbcpd5p47nxj" to finish creating: GET https://rp.j4979456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f3bad1b6-65e5-4da7-a96f-1c7006af3c80
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f3bad1b6-65e5-4da7-a96f-1c7006af3c80",
      "name": "f3bad1b6-65e5-4da7-a96f-1c7006af3c80",
      "status": "Failed",
      "startTime": "2026-09-17T13:12:19.756903274Z",
      "endTime": "2026-09-17T13:12:54.770011764Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:127]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc0009e8ec0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-qbcpd5p47nxj" to finish creating: GET https://rp.j4979456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f3bad1b6-65e5-4da7-a96f-1c7006af3c80
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f3bad1b6-65e5-4da7-a96f-1c7006af3c80",
      "name": "f3bad1b6-65e5-4da7-a96f-1c7006af3c80",
      "status": "Failed",
      "startTime": "2026-09-17T13:12:19.756903274Z",
      "endTime": "2026-09-17T13:12:54.770011764Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-22-qhppngpsw5vq/cluster-candidate-4-22-7j8cfp should provision
Unexpected error:
    <*fmt.wrapError | 0xc000bf4500>: 
    failed to create HCP cluster cluster-candidate-4-22-7j8cfp: failed waiting for cluster="cluster-candidate-4-22-7j8cfp" in resourcegroup="rg-candidate-4-22-qhppngpsw5vq" to finish creating: GET https://rp.j4979456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/793cad3e-bb75-488e-8096-19ebf4446508
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/793cad3e-bb75-488e-8096-19ebf4446508",
      "name": "793cad3e-bb75-488e-8096-19ebf4446508",
      "status": "Failed",
      "startTime": "2026-09-17T12:56:38.840875715Z",
      "endTime": "2026-09-17T13:25:42.359594909Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 22m12s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-22-qhppngpsw5vq/cluster-candidate-4-22-7j8cfp should provision
Unexpected error:
    <*fmt.wrapError | 0xc000bf4500>: 
    failed to create HCP cluster cluster-candidate-4-22-7j8cfp: failed waiting for cluster="cluster-candidate-4-22-7j8cfp" in resourcegroup="rg-candidate-4-22-qhppngpsw5vq" to finish creating: GET https://rp.j4979456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/793cad3e-bb75-488e-8096-19ebf4446508
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/793cad3e-bb75-488e-8096-19ebf4446508",
      "name": "793cad3e-bb75-488e-8096-19ebf4446508",
      "status": "Failed",
      "startTime": "2026-09-17T12:56:38.840875715Z",
      "endTime": "2026-09-17T13:25:42.359594909Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 22m12s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-17T13:42:12Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerOOMKilled", cluster="ci01-j4979456-mgmt-2", component="kubernetes-infrastructure", container="fluentbit", endpoint="http", environment="ci01", instance="10.128.64.70:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4979456/providers/microsoft.monitor/accounts/services-j4979456", namespace="arobit", pod="arobit-forwarder-k26lv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="OOMKilled", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="9061149b-822a-4131-9e5f-b554ee1870de"
  Description: Container fluentbit in pod arobit/arobit-forwarder-k26lv on cluster ci01-j4979456-mgmt-2 has been OOMKilled. This indicates the container exceeded its memory limit and was terminated by the kernel.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-17T13:42:12Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerOOMKilled", cluster="ci01-j4979456-mgmt-2", component="kubernetes-infrastructure", container="fluentbit", endpoint="http", environment="ci01", instance="10.128.64.70:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4979456/providers/microsoft.monitor/accounts/services-j4979456", namespace="arobit", pod="arobit-forwarder-k26lv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="OOMKilled", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="9061149b-822a-4131-9e5f-b554ee1870de"
  Description: Container fluentbit in pod arobit/arobit-forwarder-k26lv on cluster ci01-j4979456-mgmt-2 has been OOMKilled. This indicates the container exceeded its memory limit and was terminated by the kernel.

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-22-m7mk4qp8ml44/cluster-candidate-4-22-g8z94g should provision
Unexpected error:
    <*fmt.wrapError | 0xc0001b68a0>: 
    failed to create HCP cluster cluster-candidate-4-22-g8z94g: failed waiting for cluster="cluster-candidate-4-22-g8z94g" in resourcegroup="rg-candidate-4-22-m7mk4qp8ml44" to finish creating: GET https://rp.j7954560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/452782a4-ef29-4cb2-88be-222e3c6d56bb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/452782a4-ef29-4cb2-88be-222e3c6d56bb",
      "name": "452782a4-ef29-4cb2-88be-222e3c6d56bb",
      "status": "Failed",
      "startTime": "2026-09-17T13:07:19.259933821Z",
      "endTime": "2026-09-17T13:36:28.833945428Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 18m12s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-22-m7mk4qp8ml44/cluster-candidate-4-22-g8z94g should provision
Unexpected error:
    <*fmt.wrapError | 0xc0001b68a0>: 
    failed to create HCP cluster cluster-candidate-4-22-g8z94g: failed waiting for cluster="cluster-candidate-4-22-g8z94g" in resourcegroup="rg-candidate-4-22-m7mk4qp8ml44" to finish creating: GET https://rp.j7954560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/452782a4-ef29-4cb2-88be-222e3c6d56bb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/452782a4-ef29-4cb2-88be-222e3c6d56bb",
      "name": "452782a4-ef29-4cb2-88be-222e3c6d56bb",
      "status": "Failed",
      "startTime": "2026-09-17T13:07:19.259933821Z",
      "endTime": "2026-09-17T13:36:28.833945428Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 18m12s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-5-0-rh7dxqwsn6jn/cluster-candidate-5-0-ww9f6h should provision
Unexpected error:
    <*fmt.wrapError | 0xc00101c6c0>: 
    failed to create HCP cluster cluster-candidate-5-0-ww9f6h: failed waiting for cluster="cluster-candidate-5-0-ww9f6h" in resourcegroup="rg-candidate-5-0-rh7dxqwsn6jn" to finish creating: GET https://rp.j7954560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5a10021c-4ce6-4db9-a901-692ad3cb7f6e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5a10021c-4ce6-4db9-a901-692ad3cb7f6e",
      "name": "5a10021c-4ce6-4db9-a901-692ad3cb7f6e",
      "status": "Failed",
      "startTime": "2026-09-17T13:07:17.85124978Z",
      "endTime": "2026-09-17T13:36:18.86048967Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-rc.2 is Partial (want Completed), started 18m14s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-5-0-rh7dxqwsn6jn/cluster-candidate-5-0-ww9f6h should provision
Unexpected error:
    <*fmt.wrapError | 0xc00101c6c0>: 
    failed to create HCP cluster cluster-candidate-5-0-ww9f6h: failed waiting for cluster="cluster-candidate-5-0-ww9f6h" in resourcegroup="rg-candidate-5-0-rh7dxqwsn6jn" to finish creating: GET https://rp.j7954560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5a10021c-4ce6-4db9-a901-692ad3cb7f6e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5a10021c-4ce6-4db9-a901-692ad3cb7f6e",
      "name": "5a10021c-4ce6-4db9-a901-692ad3cb7f6e",
      "status": "Failed",
      "startTime": "2026-09-17T13:07:17.85124978Z",
      "endTime": "2026-09-17T13:36:18.86048967Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-rc.2 is Partial (want Completed), started 18m14s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:123]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc0009e1220>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-l5jw7spk7gfn" to finish creating: GET https://rp.j7954560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/047e1067-3a68-43b0-82bf-f03b03455543
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/047e1067-3a68-43b0-82bf-f03b03455543",
      "name": "047e1067-3a68-43b0-82bf-f03b03455543",
      "status": "Failed",
      "startTime": "2026-09-17T13:07:16.901724077Z",
      "endTime": "2026-09-17T13:36:18.816767345Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 20m15s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:123]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc0009e1220>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-l5jw7spk7gfn" to finish creating: GET https://rp.j7954560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/047e1067-3a68-43b0-82bf-f03b03455543
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/047e1067-3a68-43b0-82bf-f03b03455543",
      "name": "047e1067-3a68-43b0-82bf-f03b03455543",
      "status": "Failed",
      "startTime": "2026-09-17T13:07:16.901724077Z",
      "endTime": "2026-09-17T13:36:18.816767345Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 20m15s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-17T14:18:51Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j7954560-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-th462", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-th462 (namespace arobit) on cluster ci01-j7954560-svc has averaged 125.4% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

alert fired 3 time(s) (1 unknown, 2 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T13:21:21Z
  Ended:       2026-09-17T13:42:26Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7954560-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspg3njcujb3v6v7plhp5kqm1vtammb-z1k3i6i3q2w6p1h", pod="router-5ff4d5db54-z8hcs", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspg3njcujb3v6v7plhp5kqm1vtammb-z1k3i6i3q2w6p1h/router-5ff4d5db54-z8hcs has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000f5c1f8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listct75l7" in resourcegroup="ea-list-rnbsndngbwcp" to finish deleting: GET https://rp.j7653760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cc1b0f96-4766-41c8-a9e1-454bcda18f72
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cc1b0f96-4766-41c8-a9e1-454bcda18f72",
      "name": "cc1b0f96-4766-41c8-a9e1-454bcda18f72",
      "status": "Failed",
      "startTime": "2026-09-17T13:03:05.503917643Z",
      "endTime": "2026-09-17T13:27:11.118627196Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssp8aks7ifc0p8r9jljrcuc8dareu6k still exists (deletion dispatched at 2026-09-17T13:03:06Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000f5c1f8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listct75l7" in resourcegroup="ea-list-rnbsndngbwcp" to finish deleting: GET https://rp.j7653760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cc1b0f96-4766-41c8-a9e1-454bcda18f72
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cc1b0f96-4766-41c8-a9e1-454bcda18f72",
      "name": "cc1b0f96-4766-41c8-a9e1-454bcda18f72",
      "status": "Failed",
      "startTime": "2026-09-17T13:03:05.503917643Z",
      "endTime": "2026-09-17T13:27:11.118627196Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssp8aks7ifc0p8r9jljrcuc8dareu6k still exists (deletion dispatched at 2026-09-17T13:03:06Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001b7e198>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cl" in resourcegroup="cni-cilium-k647zxtr8gg7" to finish deleting: GET https://rp.j7653760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/de57187c-2af7-4266-8cc5-141df40a6c1d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/de57187c-2af7-4266-8cc5-141df40a6c1d",
      "name": "de57187c-2af7-4266-8cc5-141df40a6c1d",
      "status": "Failed",
      "startTime": "2026-09-17T13:03:16.299863658Z",
      "endTime": "2026-09-17T13:27:21.114151007Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssp758fl4976ro8f0vfe0qd8k8vok4b still exists (deletion dispatched at 2026-09-17T13:03:18Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001b7e198>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cl" in resourcegroup="cni-cilium-k647zxtr8gg7" to finish deleting: GET https://rp.j7653760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/de57187c-2af7-4266-8cc5-141df40a6c1d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/de57187c-2af7-4266-8cc5-141df40a6c1d",
      "name": "de57187c-2af7-4266-8cc5-141df40a6c1d",
      "status": "Failed",
      "startTime": "2026-09-17T13:03:16.299863658Z",
      "endTime": "2026-09-17T13:27:21.114151007Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssp758fl4976ro8f0vfe0qd8k8vok4b still exists (deletion dispatched at 2026-09-17T13:03:18Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0012be390>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-ingress" in resourcegroup="private-ingress-szkllwrp2bf5" to finish deleting: GET https://rp.j7653760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0ff3dfcf-120e-4b41-863c-96a7d57c1c10
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0ff3dfcf-120e-4b41-863c-96a7d57c1c10",
      "name": "0ff3dfcf-120e-4b41-863c-96a7d57c1c10",
      "status": "Failed",
      "startTime": "2026-09-17T13:11:18.054099455Z",
      "endTime": "2026-09-17T13:35:21.06741968Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssp77l69rakntvchgj57nf2vu6tgi6d still exists (deletion dispatched at 2026-09-17T13:11:19Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0012be390>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-ingress" in resourcegroup="private-ingress-szkllwrp2bf5" to finish deleting: GET https://rp.j7653760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0ff3dfcf-120e-4b41-863c-96a7d57c1c10
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0ff3dfcf-120e-4b41-863c-96a7d57c1c10",
      "name": "0ff3dfcf-120e-4b41-863c-96a7d57c1c10",
      "status": "Failed",
      "startTime": "2026-09-17T13:11:18.054099455Z",
      "endTime": "2026-09-17T13:35:21.06741968Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssp77l69rakntvchgj57nf2vu6tgi6d still exists (deletion dispatched at 2026-09-17T13:11:19Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0019d00c0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-bv994jltql7w" to finish deleting: GET https://rp.j7653760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/24536fa2-db66-4ae9-88b1-d48cfc097968
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/24536fa2-db66-4ae9-88b1-d48cfc097968",
      "name": "24536fa2-db66-4ae9-88b1-d48cfc097968",
      "status": "Failed",
      "startTime": "2026-09-17T12:59:26.848712279Z",
      "endTime": "2026-09-17T13:23:31.008071443Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssp7k78d1ol49mvc6mdg70kklbeg60h still exists (deletion dispatched at 2026-09-17T12:59:32Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0019d00c0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-bv994jltql7w" to finish deleting: GET https://rp.j7653760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/24536fa2-db66-4ae9-88b1-d48cfc097968
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/24536fa2-db66-4ae9-88b1-d48cfc097968",
      "name": "24536fa2-db66-4ae9-88b1-d48cfc097968",
      "status": "Failed",
      "startTime": "2026-09-17T12:59:26.848712279Z",
      "endTime": "2026-09-17T13:23:31.008071443Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssp7k78d1ol49mvc6mdg70kklbeg60h still exists (deletion dispatched at 2026-09-17T12:59:32Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:198]: failed to create node pool version 4.20.32 for back-level cluster
Unexpected error:
    <*fmt.wrapError | 0xc00105a180>: 
    failed waiting for nodepool="np-ver-4-20-32" for cluster "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-l59lxtj9lwhd" to finish creating: GET https://rp.j7653760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7095545b-bbd0-41f6-961e-6bb475b19413
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7095545b-bbd0-41f6-961e-6bb475b19413",
      "name": "7095545b-bbd0-41f6-961e-6bb475b19413",
      "status": "Failed",
      "startTime": "2026-09-17T13:01:16.192258014Z",
      "endTime": "2026-09-17T13:01:58.244062799Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:198]: failed to create node pool version 4.20.32 for back-level cluster
Unexpected error:
    <*fmt.wrapError | 0xc00105a180>: 
    failed waiting for nodepool="np-ver-4-20-32" for cluster "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-l59lxtj9lwhd" to finish creating: GET https://rp.j7653760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7095545b-bbd0-41f6-961e-6bb475b19413
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7095545b-bbd0-41f6-961e-6bb475b19413",
      "name": "7095545b-bbd0-41f6-961e-6bb475b19413",
      "status": "Failed",
      "startTime": "2026-09-17T13:01:16.192258014Z",
      "endTime": "2026-09-17T13:01:58.244062799Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001221140>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-wk6m45pxns76" to finish deleting: GET https://rp.j7653760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b51040b7-6bf3-4b42-9a38-9f9ed135e56a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b51040b7-6bf3-4b42-9a38-9f9ed135e56a",
      "name": "b51040b7-6bf3-4b42-9a38-9f9ed135e56a",
      "status": "Failed",
      "startTime": "2026-09-17T13:04:07.34911158Z",
      "endTime": "2026-09-17T13:28:11.087818002Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssp79q6249e2dvjoak8t3ao7t9k8ino still exists (deletion dispatched at 2026-09-17T13:04:08Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001221140>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-wk6m45pxns76" to finish deleting: GET https://rp.j7653760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b51040b7-6bf3-4b42-9a38-9f9ed135e56a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b51040b7-6bf3-4b42-9a38-9f9ed135e56a",
      "name": "b51040b7-6bf3-4b42-9a38-9f9ed135e56a",
      "status": "Failed",
      "startTime": "2026-09-17T13:04:07.34911158Z",
      "endTime": "2026-09-17T13:28:11.087818002Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssp79q6249e2dvjoak8t3ao7t9k8ino still exists (deletion dispatched at 2026-09-17T13:04:08Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-5-0-jlgdwlhnhcx7/cluster-candidate-5-0-h6gm8t should provision
Unexpected error:
    <*fmt.wrapError | 0xc00088e480>: 
    failed to create HCP cluster cluster-candidate-5-0-h6gm8t: failed waiting for cluster="cluster-candidate-5-0-h6gm8t" in resourcegroup="rg-candidate-5-0-jlgdwlhnhcx7" to finish creating: GET https://rp.j9627648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/440484a5-098c-4647-932e-d9ebc38c3bc1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/440484a5-098c-4647-932e-d9ebc38c3bc1",
      "name": "440484a5-098c-4647-932e-d9ebc38c3bc1",
      "status": "Failed",
      "startTime": "2026-09-17T12:55:54.30559047Z",
      "endTime": "2026-09-17T13:24:59.62081715Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-rc.2 is Partial (want Completed), started 19m42s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-5-0-jlgdwlhnhcx7/cluster-candidate-5-0-h6gm8t should provision
Unexpected error:
    <*fmt.wrapError | 0xc00088e480>: 
    failed to create HCP cluster cluster-candidate-5-0-h6gm8t: failed waiting for cluster="cluster-candidate-5-0-h6gm8t" in resourcegroup="rg-candidate-5-0-jlgdwlhnhcx7" to finish creating: GET https://rp.j9627648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/440484a5-098c-4647-932e-d9ebc38c3bc1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/440484a5-098c-4647-932e-d9ebc38c3bc1",
      "name": "440484a5-098c-4647-932e-d9ebc38c3bc1",
      "status": "Failed",
      "startTime": "2026-09-17T12:55:54.30559047Z",
      "endTime": "2026-09-17T13:24:59.62081715Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-rc.2 is Partial (want Completed), started 19m42s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T13:27:25Z
  Ended:       2026-09-17T13:33:25Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j9627648-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j9627648-svc).

alert fired 2 time(s) (1 unknown, 1 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T13:12:31Z
  Ended:       2026-09-17T13:37:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9627648-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sspamoc80krnrof9cfovaftgbsh4v70-p4a8m9h3e4b4i4x", pod="router-c6f7b99b5-tvnph", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sspamoc80krnrof9cfovaftgbsh4v70-p4a8m9h3e4b4i4x/router-c6f7b99b5-tvnph has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-5-0-vcz7fgzqgtdm/cluster-candidate-5-0-ztqsnb should provision
Unexpected error:
    <*fmt.wrapError | 0xc000842d80>: 
    failed to create HCP cluster cluster-candidate-5-0-ztqsnb: failed waiting for cluster="cluster-candidate-5-0-ztqsnb" in resourcegroup="rg-candidate-5-0-vcz7fgzqgtdm" to finish creating: GET https://rp.j2573056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8772269d-dde0-426b-b30c-d1cd81cfba32
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8772269d-dde0-426b-b30c-d1cd81cfba32",
      "name": "8772269d-dde0-426b-b30c-d1cd81cfba32",
      "status": "Failed",
      "startTime": "2026-09-17T13:11:09.279842141Z",
      "endTime": "2026-09-17T13:40:12.356762691Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-rc.2 is Partial (want Completed), started 20m53s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-5-0-vcz7fgzqgtdm/cluster-candidate-5-0-ztqsnb should provision
Unexpected error:
    <*fmt.wrapError | 0xc000842d80>: 
    failed to create HCP cluster cluster-candidate-5-0-ztqsnb: failed waiting for cluster="cluster-candidate-5-0-ztqsnb" in resourcegroup="rg-candidate-5-0-vcz7fgzqgtdm" to finish creating: GET https://rp.j2573056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8772269d-dde0-426b-b30c-d1cd81cfba32
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8772269d-dde0-426b-b30c-d1cd81cfba32",
      "name": "8772269d-dde0-426b-b30c-d1cd81cfba32",
      "status": "Failed",
      "startTime": "2026-09-17T13:11:09.279842141Z",
      "endTime": "2026-09-17T13:40:12.356762691Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-rc.2 is Partial (want Completed), started 20m53s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000fc0560>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-w9w5xmhcpq9k" to finish creating: GET https://rp.j2573056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ea230971-c3c1-419c-ac11-4d8ec5c1d9ef
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ea230971-c3c1-419c-ac11-4d8ec5c1d9ef",
      "name": "ea230971-c3c1-419c-ac11-4d8ec5c1d9ef",
      "status": "Failed",
      "startTime": "2026-09-17T13:11:41.071539907Z",
      "endTime": "2026-09-17T13:40:42.303659137Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 18m2s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000fc0560>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-w9w5xmhcpq9k" to finish creating: GET https://rp.j2573056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ea230971-c3c1-419c-ac11-4d8ec5c1d9ef
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ea230971-c3c1-419c-ac11-4d8ec5c1d9ef",
      "name": "ea230971-c3c1-419c-ac11-4d8ec5c1d9ef",
      "status": "Failed",
      "startTime": "2026-09-17T13:11:41.071539907Z",
      "endTime": "2026-09-17T13:40:42.303659137Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 18m2s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-lpqjxws5tpv6/cluster-candidate-4-21-79bc8q should provision
Unexpected error:
    <*fmt.wrapError | 0xc001192360>: 
    failed to create HCP cluster cluster-candidate-4-21-79bc8q: failed waiting for cluster="cluster-candidate-4-21-79bc8q" in resourcegroup="rg-candidate-4-21-lpqjxws5tpv6" to finish creating: GET https://rp.j2573056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0e52bdd9-b621-4bdf-8c7f-88803b26d4b1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0e52bdd9-b621-4bdf-8c7f-88803b26d4b1",
      "name": "0e52bdd9-b621-4bdf-8c7f-88803b26d4b1",
      "status": "Failed",
      "startTime": "2026-09-17T13:11:08.998702999Z",
      "endTime": "2026-09-17T13:40:12.3101134Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 19m28s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-lpqjxws5tpv6/cluster-candidate-4-21-79bc8q should provision
Unexpected error:
    <*fmt.wrapError | 0xc001192360>: 
    failed to create HCP cluster cluster-candidate-4-21-79bc8q: failed waiting for cluster="cluster-candidate-4-21-79bc8q" in resourcegroup="rg-candidate-4-21-lpqjxws5tpv6" to finish creating: GET https://rp.j2573056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0e52bdd9-b621-4bdf-8c7f-88803b26d4b1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0e52bdd9-b621-4bdf-8c7f-88803b26d4b1",
      "name": "0e52bdd9-b621-4bdf-8c7f-88803b26d4b1",
      "status": "Failed",
      "startTime": "2026-09-17T13:11:08.998702999Z",
      "endTime": "2026-09-17T13:40:12.3101134Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 19m28s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc0015ec480>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-4llzpkxqdt5h" to finish creating: GET https://rp.j2573056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/98da8819-6ed6-431a-b25f-78eab8ef8dd9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/98da8819-6ed6-431a-b25f-78eab8ef8dd9",
      "name": "98da8819-6ed6-431a-b25f-78eab8ef8dd9",
      "status": "Failed",
      "startTime": "2026-09-17T13:12:29.574806708Z",
      "endTime": "2026-09-17T13:41:32.177488403Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc0015ec480>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-4llzpkxqdt5h" to finish creating: GET https://rp.j2573056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/98da8819-6ed6-431a-b25f-78eab8ef8dd9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/98da8819-6ed6-431a-b25f-78eab8ef8dd9",
      "name": "98da8819-6ed6-431a-b25f-78eab8ef8dd9",
      "status": "Failed",
      "startTime": "2026-09-17T13:12:29.574806708Z",
      "endTime": "2026-09-17T13:41:32.177488403Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:98]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc001076240>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-zw6vh2x4vdzh" to finish creating: GET https://rp.j2573056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8da3fbe2-2951-4156-b4f5-0f409aecd554
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8da3fbe2-2951-4156-b4f5-0f409aecd554",
      "name": "8da3fbe2-2951-4156-b4f5-0f409aecd554",
      "status": "Failed",
      "startTime": "2026-09-17T13:15:01.412052631Z",
      "endTime": "2026-09-17T13:44:02.315476713Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 21m49s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:98]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc001076240>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-zw6vh2x4vdzh" to finish creating: GET https://rp.j2573056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8da3fbe2-2951-4156-b4f5-0f409aecd554
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8da3fbe2-2951-4156-b4f5-0f409aecd554",
      "name": "8da3fbe2-2951-4156-b4f5-0f409aecd554",
      "status": "Failed",
      "startTime": "2026-09-17T13:15:01.412052631Z",
      "endTime": "2026-09-17T13:44:02.315476713Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 21m49s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-22-2t7ghvmhbmnj/cluster-candidate-4-22-2x7d82 should provision
Unexpected error:
    <*fmt.wrapError | 0xc001158140>: 
    failed to create HCP cluster cluster-candidate-4-22-2x7d82: failed waiting for cluster="cluster-candidate-4-22-2x7d82" in resourcegroup="rg-candidate-4-22-2t7ghvmhbmnj" to finish creating: GET https://rp.j2573056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4da719d0-f4ed-4162-a3ff-e4a6c5af51ba
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4da719d0-f4ed-4162-a3ff-e4a6c5af51ba",
      "name": "4da719d0-f4ed-4162-a3ff-e4a6c5af51ba",
      "status": "Failed",
      "startTime": "2026-09-17T13:11:28.390954588Z",
      "endTime": "2026-09-17T13:40:32.344644366Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 19m22s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-22-2t7ghvmhbmnj/cluster-candidate-4-22-2x7d82 should provision
Unexpected error:
    <*fmt.wrapError | 0xc001158140>: 
    failed to create HCP cluster cluster-candidate-4-22-2x7d82: failed waiting for cluster="cluster-candidate-4-22-2x7d82" in resourcegroup="rg-candidate-4-22-2t7ghvmhbmnj" to finish creating: GET https://rp.j2573056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4da719d0-f4ed-4162-a3ff-e4a6c5af51ba
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4da719d0-f4ed-4162-a3ff-e4a6c5af51ba",
      "name": "4da719d0-f4ed-4162-a3ff-e4a6c5af51ba",
      "status": "Failed",
      "startTime": "2026-09-17T13:11:28.390954588Z",
      "endTime": "2026-09-17T13:40:32.344644366Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 19m22s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:89]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc000a22d00>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-h72955nqztzc" to finish creating: GET https://rp.j2573056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/adbc0b31-5bf0-425f-91ac-209bee804e12
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/adbc0b31-5bf0-425f-91ac-209bee804e12",
      "name": "adbc0b31-5bf0-425f-91ac-209bee804e12",
      "status": "Failed",
      "startTime": "2026-09-17T13:11:42.554116144Z",
      "endTime": "2026-09-17T13:40:42.348888097Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 18m2s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:89]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc000a22d00>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-h72955nqztzc" to finish creating: GET https://rp.j2573056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/adbc0b31-5bf0-425f-91ac-209bee804e12
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/adbc0b31-5bf0-425f-91ac-209bee804e12",
      "name": "adbc0b31-5bf0-425f-91ac-209bee804e12",
      "status": "Failed",
      "startTime": "2026-09-17T13:11:42.554116144Z",
      "endTime": "2026-09-17T13:40:42.348888097Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 18m2s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--r8vk8fhz9wm8/cluster-hs-pre-nrrplm should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f4e3a0>: 
    failed to create HCP cluster cluster-hs-pre-nrrplm: failed waiting for cluster="cluster-hs-pre-nrrplm" in resourcegroup="rg-hs-pre--r8vk8fhz9wm8" to finish creating: GET https://rp.j2573056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f963a38d-9e37-48b7-bd4b-6b88e16b85ab
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f963a38d-9e37-48b7-bd4b-6b88e16b85ab",
      "name": "f963a38d-9e37-48b7-bd4b-6b88e16b85ab",
      "status": "Failed",
      "startTime": "2026-09-17T13:11:28.528887492Z",
      "endTime": "2026-09-17T13:40:32.181099978Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 18m54s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--r8vk8fhz9wm8/cluster-hs-pre-nrrplm should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f4e3a0>: 
    failed to create HCP cluster cluster-hs-pre-nrrplm: failed waiting for cluster="cluster-hs-pre-nrrplm" in resourcegroup="rg-hs-pre--r8vk8fhz9wm8" to finish creating: GET https://rp.j2573056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f963a38d-9e37-48b7-bd4b-6b88e16b85ab
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f963a38d-9e37-48b7-bd4b-6b88e16b85ab",
      "name": "f963a38d-9e37-48b7-bd4b-6b88e16b85ab",
      "status": "Failed",
      "startTime": "2026-09-17T13:11:28.528887492Z",
      "endTime": "2026-09-17T13:40:32.181099978Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 18m54s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc0008688c0>: 
    failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-lpsw2q8vdb2m" to finish creating: GET https://rp.j2573056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/76616663-2962-4c27-a780-127cb9b30572
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/76616663-2962-4c27-a780-127cb9b30572",
      "name": "76616663-2962-4c27-a780-127cb9b30572",
      "status": "Failed",
      "startTime": "2026-09-17T13:11:07.25067562Z",
      "endTime": "2026-09-17T13:40:12.357476827Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 21m8s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc0008688c0>: 
    failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-lpsw2q8vdb2m" to finish creating: GET https://rp.j2573056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/76616663-2962-4c27-a780-127cb9b30572
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/76616663-2962-4c27-a780-127cb9b30572",
      "name": "76616663-2962-4c27-a780-127cb9b30572",
      "status": "Failed",
      "startTime": "2026-09-17T13:11:07.25067562Z",
      "endTime": "2026-09-17T13:40:12.357476827Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 21m8s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc00083a980>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-bjdtmlhkdlb5" to finish creating: GET https://rp.j2573056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e4d64791-d5a0-4688-8e91-8220f53405bc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e4d64791-d5a0-4688-8e91-8220f53405bc",
      "name": "e4d64791-d5a0-4688-8e91-8220f53405bc",
      "status": "Failed",
      "startTime": "2026-09-17T13:13:32.702937524Z",
      "endTime": "2026-09-17T13:42:32.270930101Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 21m25s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc00083a980>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-bjdtmlhkdlb5" to finish creating: GET https://rp.j2573056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e4d64791-d5a0-4688-8e91-8220f53405bc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e4d64791-d5a0-4688-8e91-8220f53405bc",
      "name": "e4d64791-d5a0-4688-8e91-8220f53405bc",
      "status": "Failed",
      "startTime": "2026-09-17T13:13:32.702937524Z",
      "endTime": "2026-09-17T13:42:32.270930101Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 21m25s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc000aea680>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-kmkdlhnxm6qn" to finish creating: GET https://rp.j2573056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/81708d5e-7bcb-4064-a523-2f2429ca57a4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/81708d5e-7bcb-4064-a523-2f2429ca57a4",
      "name": "81708d5e-7bcb-4064-a523-2f2429ca57a4",
      "status": "Failed",
      "startTime": "2026-09-17T13:12:09.723635539Z",
      "endTime": "2026-09-17T13:41:12.177995057Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 18m59s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc000aea680>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-kmkdlhnxm6qn" to finish creating: GET https://rp.j2573056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/81708d5e-7bcb-4064-a523-2f2429ca57a4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/81708d5e-7bcb-4064-a523-2f2429ca57a4",
      "name": "81708d5e-7bcb-4064-a523-2f2429ca57a4",
      "status": "Failed",
      "startTime": "2026-09-17T13:12:09.723635539Z",
      "endTime": "2026-09-17T13:41:12.177995057Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 18m59s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 7 time(s) (1 unknown, 6 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T13:25:00Z
  Ended:       2026-09-17T13:52:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2573056-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssphrhuflst4cpf4hbnhn709qfh15fp-p4k0t8h9d2i2s1j", pod="router-68544bfbb8-v8gnr", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssphrhuflst4cpf4hbnhn709qfh15fp-p4k0t8h9d2i2s1j/router-68544bfbb8-v8gnr has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000e98600>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-20-85x5lk" in resourcegroup="rg-candidate-4-20-vdg5pfb8x96b" to finish deleting: GET https://rp.j8202752.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3be4750b-6d04-4962-b270-180837a66720
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3be4750b-6d04-4962-b270-180837a66720",
      "name": "3be4750b-6d04-4962-b270-180837a66720",
      "status": "Failed",
      "startTime": "2026-09-17T13:08:22.9987657Z",
      "endTime": "2026-09-17T13:32:25.907204967Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssp7n7f9kn1tn0m8nhdclc4oi11ms50 still exists (deletion dispatched at 2026-09-17T13:08:24Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000e98600>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-20-85x5lk" in resourcegroup="rg-candidate-4-20-vdg5pfb8x96b" to finish deleting: GET https://rp.j8202752.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3be4750b-6d04-4962-b270-180837a66720
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3be4750b-6d04-4962-b270-180837a66720",
      "name": "3be4750b-6d04-4962-b270-180837a66720",
      "status": "Failed",
      "startTime": "2026-09-17T13:08:22.9987657Z",
      "endTime": "2026-09-17T13:32:25.907204967Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssp7n7f9kn1tn0m8nhdclc4oi11ms50 still exists (deletion dispatched at 2026-09-17T13:08:24Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000ac2e60>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-mmb8rvzxtd75" to finish creating: GET https://rp.j6121600.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6b20002d-4aaf-4a1a-8670-11b592a70d29
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6b20002d-4aaf-4a1a-8670-11b592a70d29",
      "name": "6b20002d-4aaf-4a1a-8670-11b592a70d29",
      "status": "Failed",
      "startTime": "2026-09-17T11:46:22.660045003Z",
      "endTime": "2026-09-17T12:15:31.401172503Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m23s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000ac2e60>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-mmb8rvzxtd75" to finish creating: GET https://rp.j6121600.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6b20002d-4aaf-4a1a-8670-11b592a70d29
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6b20002d-4aaf-4a1a-8670-11b592a70d29",
      "name": "6b20002d-4aaf-4a1a-8670-11b592a70d29",
      "status": "Failed",
      "startTime": "2026-09-17T11:46:22.660045003Z",
      "endTime": "2026-09-17T12:15:31.401172503Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m23s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.37
Unexpected error:
    <*fmt.wrapError | 0xc000d8b160>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-zj6grj" in resourcegroup="rg-np-version-upgrade-zj6grj-pp9qwfhhwgc8" to finish creating: GET https://rp.j3847296.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/85af6203-800d-4bc5-a875-4385db2526c1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/85af6203-800d-4bc5-a875-4385db2526c1",
      "name": "85af6203-800d-4bc5-a875-4385db2526c1",
      "status": "Failed",
      "startTime": "2026-09-17T11:29:25.794778495Z",
      "endTime": "2026-09-17T11:48:27.334521515Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.37
Unexpected error:
    <*fmt.wrapError | 0xc000d8b160>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-zj6grj" in resourcegroup="rg-np-version-upgrade-zj6grj-pp9qwfhhwgc8" to finish creating: GET https://rp.j3847296.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/85af6203-800d-4bc5-a875-4385db2526c1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/85af6203-800d-4bc5-a875-4385db2526c1",
      "name": "85af6203-800d-4bc5-a875-4385db2526c1",
      "status": "Failed",
      "startTime": "2026-09-17T11:29:25.794778495Z",
      "endTime": "2026-09-17T11:48:27.334521515Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-22-wvq9c9fgtszp/cluster-candidate-4-22-f59zps should provision
Unexpected error:
    <*fmt.wrapError | 0xc001a5a5a0>: 
    failed to create HCP cluster cluster-candidate-4-22-f59zps: failed waiting for cluster="cluster-candidate-4-22-f59zps" in resourcegroup="rg-candidate-4-22-wvq9c9fgtszp" to finish creating: GET https://rp.j5500160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c01af83a-a538-473f-b3be-5e376d524286
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c01af83a-a538-473f-b3be-5e376d524286",
      "name": "c01af83a-a538-473f-b3be-5e376d524286",
      "status": "Failed",
      "startTime": "2026-09-17T10:56:01.960012269Z",
      "endTime": "2026-09-17T11:25:07.989351808Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 18m16s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-22-wvq9c9fgtszp/cluster-candidate-4-22-f59zps should provision
Unexpected error:
    <*fmt.wrapError | 0xc001a5a5a0>: 
    failed to create HCP cluster cluster-candidate-4-22-f59zps: failed waiting for cluster="cluster-candidate-4-22-f59zps" in resourcegroup="rg-candidate-4-22-wvq9c9fgtszp" to finish creating: GET https://rp.j5500160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c01af83a-a538-473f-b3be-5e376d524286
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c01af83a-a538-473f-b3be-5e376d524286",
      "name": "c01af83a-a538-473f-b3be-5e376d524286",
      "status": "Failed",
      "startTime": "2026-09-17T10:56:01.960012269Z",
      "endTime": "2026-09-17T11:25:07.989351808Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 18m16s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:88]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc00061ea60>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-tn4rzxgj24f9" to finish creating: GET https://rp.j5500160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8ff5a3e5-0a3c-4580-9516-ce0a34dbd352
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8ff5a3e5-0a3c-4580-9516-ce0a34dbd352",
      "name": "8ff5a3e5-0a3c-4580-9516-ce0a34dbd352",
      "status": "Failed",
      "startTime": "2026-09-17T10:56:23.301058289Z",
      "endTime": "2026-09-17T11:25:27.989589306Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 21m15s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:88]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc00061ea60>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-tn4rzxgj24f9" to finish creating: GET https://rp.j5500160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8ff5a3e5-0a3c-4580-9516-ce0a34dbd352
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8ff5a3e5-0a3c-4580-9516-ce0a34dbd352",
      "name": "8ff5a3e5-0a3c-4580-9516-ce0a34dbd352",
      "status": "Failed",
      "startTime": "2026-09-17T10:56:23.301058289Z",
      "endTime": "2026-09-17T11:25:27.989589306Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 21m15s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 4 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T11:10:36Z
  Ended:       2026-09-17T11:18:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5500160-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssnii6qo5ta8ipa113mfqoa8cg9ovg6-w5z4j7t8m4l3p8k", pod="router-797f64c956-w5jrz", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssnii6qo5ta8ipa113mfqoa8cg9ovg6-w5z4j7t8m4l3p8k/router-797f64c956-w5jrz has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-17T11:11:36Z
  Ended:       2026-09-17T11:18:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5500160-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssnilsu0lmer0m84rarf0f3ood2jtdl-v20261001", pod="router-7dbd8fdf5f-vrx9x", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssnilsu0lmer0m84rarf0f3ood2jtdl-v20261001/router-7dbd8fdf5f-vrx9x has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-17T11:11:36Z
  Ended:       2026-09-17T11:39:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5500160-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssnilsu0lmer0m84rarf0f3ood2jtdl-v20261001", pod="router-7dbd8fdf5f-8k8kr", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssnilsu0lmer0m84rarf0f3ood2jtdl-v20261001/router-7dbd8fdf5f-8k8kr has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-17T11:13:34Z
  Ended:       2026-09-17T11:37:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5500160-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssnigo1rbmol44repn0nblke58e1aae-y1p9b5x6a2o0p9o", pod="router-cc54f77bc-vv28z", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssnigo1rbmol44repn0nblke58e1aae-y1p9b5x6a2o0p9o/router-cc54f77bc-vv28z has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:188]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-prgsd6"
Unexpected error:
    <*fmt.wrapError | 0xc00106cb20>: 
    DNS for route host agnhost-e2e-sample-app-jbbn8.apps.aro.b9d8s7t1n3o1u4s.irz8.j8665344.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-jbbn8.apps.aro.b9d8s7t1n3o1u4s.irz8.j8665344.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-jbbn8.apps.aro.b9d8s7t1n3o1u4s.irz8.j8665344.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:188]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-prgsd6"
Unexpected error:
    <*fmt.wrapError | 0xc00106cb20>: 
    DNS for route host agnhost-e2e-sample-app-jbbn8.apps.aro.b9d8s7t1n3o1u4s.irz8.j8665344.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-jbbn8.apps.aro.b9d8s7t1n3o1u4s.irz8.j8665344.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-jbbn8.apps.aro.b9d8s7t1n3o1u4s.irz8.j8665344.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc0009864e0>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-hrxjnzlgzfg2" to finish creating: GET https://rp.j8665344.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c0b9ab6a-819e-409c-b5e3-ba11f6fd4008
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c0b9ab6a-819e-409c-b5e3-ba11f6fd4008",
      "name": "c0b9ab6a-819e-409c-b5e3-ba11f6fd4008",
      "status": "Failed",
      "startTime": "2026-09-17T11:01:00.127507794Z",
      "endTime": "2026-09-17T11:30:02.580930241Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 25m36s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc0009864e0>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-hrxjnzlgzfg2" to finish creating: GET https://rp.j8665344.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c0b9ab6a-819e-409c-b5e3-ba11f6fd4008
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c0b9ab6a-819e-409c-b5e3-ba11f6fd4008",
      "name": "c0b9ab6a-819e-409c-b5e3-ba11f6fd4008",
      "status": "Failed",
      "startTime": "2026-09-17T11:01:00.127507794Z",
      "endTime": "2026-09-17T11:30:02.580930241Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 25m36s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-grnnjnv9wr27/cluster-candidate-4-21-fqvxll should provision
Unexpected error:
    <*fmt.wrapError | 0xc00110c280>: 
    failed to create HCP cluster cluster-candidate-4-21-fqvxll: failed waiting for cluster="cluster-candidate-4-21-fqvxll" in resourcegroup="rg-candidate-4-21-grnnjnv9wr27" to finish creating: GET https://rp.j8665344.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0d3816eb-5b98-47cf-a7d0-ab5969f2ff56
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0d3816eb-5b98-47cf-a7d0-ab5969f2ff56",
      "name": "0d3816eb-5b98-47cf-a7d0-ab5969f2ff56",
      "status": "Failed",
      "startTime": "2026-09-17T11:00:55.316971356Z",
      "endTime": "2026-09-17T11:30:02.664970428Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 25m21s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-grnnjnv9wr27/cluster-candidate-4-21-fqvxll should provision
Unexpected error:
    <*fmt.wrapError | 0xc00110c280>: 
    failed to create HCP cluster cluster-candidate-4-21-fqvxll: failed waiting for cluster="cluster-candidate-4-21-fqvxll" in resourcegroup="rg-candidate-4-21-grnnjnv9wr27" to finish creating: GET https://rp.j8665344.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0d3816eb-5b98-47cf-a7d0-ab5969f2ff56
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0d3816eb-5b98-47cf-a7d0-ab5969f2ff56",
      "name": "0d3816eb-5b98-47cf-a7d0-ab5969f2ff56",
      "status": "Failed",
      "startTime": "2026-09-17T11:00:55.316971356Z",
      "endTime": "2026-09-17T11:30:02.664970428Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 25m21s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T11:32:30Z
  Ended:       2026-09-17T11:40:30Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j8665344-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j8665344-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T11:07:39Z
  Ended:       2026-09-17T11:18:41Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6150656-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssnj050ab40fgp4v0g8qea2ljm1rqu0-h3o3s1e2a8m2f5s", pod="router-7c74fbf7c7-6pzzf", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssnj050ab40fgp4v0g8qea2ljm1rqu0-h3o3s1e2a8m2f5s/router-7c74fbf7c7-6pzzf has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T11:32:03Z
  Ended:       2026-09-17T11:40:06Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j6556800-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j6556800-svc).

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--ngdwgjgglhdj/cluster-hs-pre-g8t7gf should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f3e380>: 
    failed to create HCP cluster cluster-hs-pre-g8t7gf: failed waiting for cluster="cluster-hs-pre-g8t7gf" in resourcegroup="rg-hs-pre--ngdwgjgglhdj" to finish creating: GET https://rp.j6993280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5612c188-57de-4476-ba6c-2ed35a77ff92
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5612c188-57de-4476-ba6c-2ed35a77ff92",
      "name": "5612c188-57de-4476-ba6c-2ed35a77ff92",
      "status": "Failed",
      "startTime": "2026-09-17T10:57:35.702404296Z",
      "endTime": "2026-09-17T11:26:44.378654207Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 18m25s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--ngdwgjgglhdj/cluster-hs-pre-g8t7gf should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f3e380>: 
    failed to create HCP cluster cluster-hs-pre-g8t7gf: failed waiting for cluster="cluster-hs-pre-g8t7gf" in resourcegroup="rg-hs-pre--ngdwgjgglhdj" to finish creating: GET https://rp.j6993280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5612c188-57de-4476-ba6c-2ed35a77ff92
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5612c188-57de-4476-ba6c-2ed35a77ff92",
      "name": "5612c188-57de-4476-ba6c-2ed35a77ff92",
      "status": "Failed",
      "startTime": "2026-09-17T10:57:35.702404296Z",
      "endTime": "2026-09-17T11:26:44.378654207Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 18m25s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc000ec6e20>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-jdc8zbmqlkj4" to finish creating: GET https://rp.j6993280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9c0864bb-c64a-426d-afa3-18a243c3bd58
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9c0864bb-c64a-426d-afa3-18a243c3bd58",
      "name": "9c0864bb-c64a-426d-afa3-18a243c3bd58",
      "status": "Failed",
      "startTime": "2026-09-17T10:59:09.348421927Z",
      "endTime": "2026-09-17T11:28:14.430649094Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 20m16s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc000ec6e20>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-jdc8zbmqlkj4" to finish creating: GET https://rp.j6993280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9c0864bb-c64a-426d-afa3-18a243c3bd58
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9c0864bb-c64a-426d-afa3-18a243c3bd58",
      "name": "9c0864bb-c64a-426d-afa3-18a243c3bd58",
      "status": "Failed",
      "startTime": "2026-09-17T10:59:09.348421927Z",
      "endTime": "2026-09-17T11:28:14.430649094Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 20m16s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000c74920>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-r4x9tc6mxhkj" to finish creating: GET https://rp.j6993280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ffe4267e-2d41-4583-961d-347b03291885
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ffe4267e-2d41-4583-961d-347b03291885",
      "name": "ffe4267e-2d41-4583-961d-347b03291885",
      "status": "Failed",
      "startTime": "2026-09-17T11:04:59.768364805Z",
      "endTime": "2026-09-17T11:34:04.384726912Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 24m25s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000c74920>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-r4x9tc6mxhkj" to finish creating: GET https://rp.j6993280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ffe4267e-2d41-4583-961d-347b03291885
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ffe4267e-2d41-4583-961d-347b03291885",
      "name": "ffe4267e-2d41-4583-961d-347b03291885",
      "status": "Failed",
      "startTime": "2026-09-17T11:04:59.768364805Z",
      "endTime": "2026-09-17T11:34:04.384726912Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 24m25s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000f3e540>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-fqqfkcp6hbqk" to finish creating: GET https://rp.j6993280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dae9a503-48dc-4d52-a9af-7dea389a97eb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dae9a503-48dc-4d52-a9af-7dea389a97eb",
      "name": "dae9a503-48dc-4d52-a9af-7dea389a97eb",
      "status": "Failed",
      "startTime": "2026-09-17T10:58:56.172242762Z",
      "endTime": "2026-09-17T11:28:04.350491435Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 19m54s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000f3e540>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-fqqfkcp6hbqk" to finish creating: GET https://rp.j6993280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dae9a503-48dc-4d52-a9af-7dea389a97eb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dae9a503-48dc-4d52-a9af-7dea389a97eb",
      "name": "dae9a503-48dc-4d52-a9af-7dea389a97eb",
      "status": "Failed",
      "startTime": "2026-09-17T10:58:56.172242762Z",
      "endTime": "2026-09-17T11:28:04.350491435Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 19m54s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 2 time(s) (1 unknown, 1 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T11:11:35Z
  Ended:       2026-09-17T11:36:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6993280-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssnih45s28ov98565mqvfb07gsjjimd-d4n2e8g0i2f8m2q", pod="router-5f5d484d45-qwv5z", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssnih45s28ov98565mqvfb07gsjjimd-d4n2e8g0i2f8m2q/router-5f5d484d45-qwv5z has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.38
Unexpected error:
    <*fmt.wrapError | 0xc0010092c0>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-5pzngx" in resourcegroup="rg-np-version-upgrade-5pzngx-z4lzmt6f7cqd" to finish creating: GET https://rp.j6526464.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c65ff35d-8a6d-42e7-be19-ac0fc196634c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c65ff35d-8a6d-42e7-be19-ac0fc196634c",
      "name": "c65ff35d-8a6d-42e7-be19-ac0fc196634c",
      "status": "Failed",
      "startTime": "2026-09-17T11:09:10.677337276Z",
      "endTime": "2026-09-17T11:28:11.131937431Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.38
Unexpected error:
    <*fmt.wrapError | 0xc0010092c0>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-5pzngx" in resourcegroup="rg-np-version-upgrade-5pzngx-z4lzmt6f7cqd" to finish creating: GET https://rp.j6526464.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c65ff35d-8a6d-42e7-be19-ac0fc196634c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c65ff35d-8a6d-42e7-be19-ac0fc196634c",
      "name": "c65ff35d-8a6d-42e7-be19-ac0fc196634c",
      "status": "Failed",
      "startTime": "2026-09-17T11:09:10.677337276Z",
      "endTime": "2026-09-17T11:28:11.131937431Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T11:12:18Z
  Ended:       2026-09-17T11:18:19Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5196928-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="router", endpoint="http", environment="ci01", instance="10.128.65.120:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5196928/providers/microsoft.monitor/accounts/hcps-j5196928", namespace="ocm-arohcpci01-2ssn9fqkkgqtkcb1te9l2nsihdvccpop-x8g8g9w0z4n5j0k", pod="arohcp-monitor-kube-state-metrics-574f657d96-hxhtp", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ssn9fqkkgqtkcb1te9l2nsihdvccpop-x8g8g9w0z4n5j0k/router has not matched the expected number of replicas for longer than 30 minutes.

alert fired 7 time(s) (1 unknown, 6 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T10:47:18Z
  Ended:       2026-09-17T11:18:18Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5196928-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssn9fqkkgqtkcb1te9l2nsihdvccpop-x8g8g9w0z4n5j0k", pod="router-d8f9fc476-52b4m", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssn9fqkkgqtkcb1te9l2nsihdvccpop-x8g8g9w0z4n5j0k/router-d8f9fc476-52b4m has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:89]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc000b110a0>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-sbkwl8vljgpn" to finish creating: GET https://rp.j6748032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d5015369-f740-49f9-9f1d-eb267795b2e8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d5015369-f740-49f9-9f1d-eb267795b2e8",
      "name": "d5015369-f740-49f9-9f1d-eb267795b2e8",
      "status": "Failed",
      "startTime": "2026-09-17T10:19:28.770790207Z",
      "endTime": "2026-09-17T10:48:31.935939484Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 18m51s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:89]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc000b110a0>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-sbkwl8vljgpn" to finish creating: GET https://rp.j6748032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d5015369-f740-49f9-9f1d-eb267795b2e8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d5015369-f740-49f9-9f1d-eb267795b2e8",
      "name": "d5015369-f740-49f9-9f1d-eb267795b2e8",
      "status": "Failed",
      "startTime": "2026-09-17T10:19:28.770790207Z",
      "endTime": "2026-09-17T10:48:31.935939484Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 18m51s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T10:34:40Z
  Ended:       2026-09-17T10:45:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6748032-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssn19t7jsck03qgc5evps6405bso62l-j4d3g2z6s4z7u4d", pod="router-588db7c789-b5465", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssn19t7jsck03qgc5evps6405bso62l-j4d3g2z6s4z7u4d/router-588db7c789-b5465 has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc000ae6d00>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-qjxvxd7q75hr" to finish creating: GET https://rp.j4886144.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e32fd27f-bb18-4481-845a-6650b11bd21d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e32fd27f-bb18-4481-845a-6650b11bd21d",
      "name": "e32fd27f-bb18-4481-845a-6650b11bd21d",
      "status": "Failed",
      "startTime": "2026-09-17T09:57:31.253921777Z",
      "endTime": "2026-09-17T10:26:33.068961561Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m20s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc000ae6d00>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-qjxvxd7q75hr" to finish creating: GET https://rp.j4886144.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e32fd27f-bb18-4481-845a-6650b11bd21d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e32fd27f-bb18-4481-845a-6650b11bd21d",
      "name": "e32fd27f-bb18-4481-845a-6650b11bd21d",
      "status": "Failed",
      "startTime": "2026-09-17T09:57:31.253921777Z",
      "endTime": "2026-09-17T10:26:33.068961561Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m20s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000b183a0>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-54qnc8xlwfhg" to finish creating: GET https://rp.j4886144.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/76e6ba17-d4f1-4462-9b43-6e52575fd8bc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/76e6ba17-d4f1-4462-9b43-6e52575fd8bc",
      "name": "76e6ba17-d4f1-4462-9b43-6e52575fd8bc",
      "status": "Failed",
      "startTime": "2026-09-17T09:57:29.922050024Z",
      "endTime": "2026-09-17T10:26:33.079182372Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m8s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000b183a0>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-54qnc8xlwfhg" to finish creating: GET https://rp.j4886144.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/76e6ba17-d4f1-4462-9b43-6e52575fd8bc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/76e6ba17-d4f1-4462-9b43-6e52575fd8bc",
      "name": "76e6ba17-d4f1-4462-9b43-6e52575fd8bc",
      "status": "Failed",
      "startTime": "2026-09-17T09:57:29.922050024Z",
      "endTime": "2026-09-17T10:26:33.079182372Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m8s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/ksm_hcp_metrics.go:97]: Timed out after 900.001s.
kube_node_info metrics never appeared in Azure Monitor for cluster ""
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/ksm_hcp_metrics.go:95 with:
expected kube_node_info metrics for cluster "" but got no results
Expected
    <[]prometheus.Result | len:0, cap:0>: ...
not to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/ksm_hcp_metrics.go:97]: Timed out after 900.001s.
kube_node_info metrics never appeared in Azure Monitor for cluster ""
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/ksm_hcp_metrics.go:95 with:
expected kube_node_info metrics for cluster "" but got no results
Expected
    <[]prometheus.Result | len:0, cap:0>: ...
not to be empty

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:101]: Timed out after 600.007s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc0018b8060>: 
    failed to get kusto cluster name from config: configuration: key kusto not found
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:101]: Timed out after 600.007s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc0018b8060>: 
    failed to get kusto cluster name from config: configuration: key kusto not found
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/ama_metrics.go:83]: failed to look up SVC Prometheus endpoint for workspace services-j4886144 in resource group hcp-underlay-ci01-j4886144
Unexpected error:
    <*fmt.wrapError | 0xc001b90a00>: 
    failed to get workspace services-j4886144: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4886144/providers/Microsoft.Monitor/accounts/services-j4886144
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceGroupNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceGroupNotFound",
        "message": "Resource group 'hcp-underlay-ci01-j4886144' could not be found."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/ama_metrics.go:83]: failed to look up SVC Prometheus endpoint for workspace services-j4886144 in resource group hcp-underlay-ci01-j4886144
Unexpected error:
    <*fmt.wrapError | 0xc001b90a00>: 
    failed to get workspace services-j4886144: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4886144/providers/Microsoft.Monitor/accounts/services-j4886144
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceGroupNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceGroupNotFound",
        "message": "Resource group 'hcp-underlay-ci01-j4886144' could not be found."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-r5fv6cfx6sx5/cluster-candidate-4-21-bcgg5p should provision
Unexpected error:
    <*fmt.wrapError | 0xc0002b82e0>: 
    failed to create HCP cluster cluster-candidate-4-21-bcgg5p: failed waiting for cluster="cluster-candidate-4-21-bcgg5p" in resourcegroup="rg-candidate-4-21-r5fv6cfx6sx5" to finish creating: GET https://rp.j4886144.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c7e1fc86-dec9-4c9d-bd4b-1089ac2991b7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c7e1fc86-dec9-4c9d-bd4b-1089ac2991b7",
      "name": "c7e1fc86-dec9-4c9d-bd4b-1089ac2991b7",
      "status": "Failed",
      "startTime": "2026-09-17T09:57:20.337198104Z",
      "endTime": "2026-09-17T10:26:23.169599025Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m17s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-r5fv6cfx6sx5/cluster-candidate-4-21-bcgg5p should provision
Unexpected error:
    <*fmt.wrapError | 0xc0002b82e0>: 
    failed to create HCP cluster cluster-candidate-4-21-bcgg5p: failed waiting for cluster="cluster-candidate-4-21-bcgg5p" in resourcegroup="rg-candidate-4-21-r5fv6cfx6sx5" to finish creating: GET https://rp.j4886144.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c7e1fc86-dec9-4c9d-bd4b-1089ac2991b7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c7e1fc86-dec9-4c9d-bd4b-1089ac2991b7",
      "name": "c7e1fc86-dec9-4c9d-bd4b-1089ac2991b7",
      "status": "Failed",
      "startTime": "2026-09-17T09:57:20.337198104Z",
      "endTime": "2026-09-17T10:26:23.169599025Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m17s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000d26480>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-xsdmfkrcw525" to finish creating: GET https://rp.j4886144.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3ef95353-782b-41cc-8d21-8f7baa89c8c9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3ef95353-782b-41cc-8d21-8f7baa89c8c9",
      "name": "3ef95353-782b-41cc-8d21-8f7baa89c8c9",
      "status": "Failed",
      "startTime": "2026-09-17T09:57:31.339099602Z",
      "endTime": "2026-09-17T10:26:32.999749225Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m4s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000d26480>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-xsdmfkrcw525" to finish creating: GET https://rp.j4886144.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3ef95353-782b-41cc-8d21-8f7baa89c8c9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3ef95353-782b-41cc-8d21-8f7baa89c8c9",
      "name": "3ef95353-782b-41cc-8d21-8f7baa89c8c9",
      "status": "Failed",
      "startTime": "2026-09-17T09:57:31.339099602Z",
      "endTime": "2026-09-17T10:26:32.999749225Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m4s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 43 time(s) (37 unknown, 6 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T10:08:08Z
  Ended:       2026-09-17T10:44:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4886144-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssmn1ocgn9okmeulgkqmo7r5v6lch6q-m0m0g4x6i7u6c3d", pod="kube-state-metrics-hcp-8654bffc85-42th9", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssmn1ocgn9okmeulgkqmo7r5v6lch6q-m0m0g4x6i7u6c3d/kube-state-metrics-hcp-8654bffc85-42th9 has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-17T10:08:08Z
  Ended:       2026-09-17T10:30:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4886144-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssmn0a58o3b6d465capumlr6m7i3tmj-p3v9g4r5h1c7b3t", pod="kube-state-metrics-hcp-8654bffc85-48pgr", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssmn0a58o3b6d465capumlr6m7i3tmj-p3v9g4r5h1c7b3t/kube-state-metrics-hcp-8654bffc85-48pgr has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-17T10:08:08Z
  Ended:       2026-09-17T10:26:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4886144-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssmn0d4ev99mqi5thvqom46iu60pu8o-d9l5o4t0k9i2n1x", pod="kube-state-metrics-hcp-8654bffc85-6bltn", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssmn0d4ev99mqi5thvqom46iu60pu8o-d9l5o4t0k9i2n1x/kube-state-metrics-hcp-8654bffc85-6bltn has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-17T10:08:08Z
  Ended:       2026-09-17T10:47:57Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4886144-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssmn25h7ceu6n469ednem3dgrmd8bps-w9b7d7c0w9m1v9m", pod="kube-state-metrics-hcp-8654bffc85-q8cxk", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssmn25h7ceu6n469ednem3dgrmd8bps-w9b7d7c0w9m1v9m/kube-state-metrics-hcp-8654bffc85-q8cxk has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-09-17T10:08:08Z
  Ended:       2026-09-17T10:37:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4886144-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssmn1lcq0sd7l1s8c1f4dqjhp7mthsq-b0m0a3b8d8d1s6c", pod="kube-state-metrics-hcp-8654bffc85-w7zgw", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssmn1lcq0sd7l1s8c1f4dqjhp7mthsq-b0m0a3b8d8d1s6c/kube-state-metrics-hcp-8654bffc85-w7zgw has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-09-17T10:08:08Z
  Ended:       2026-09-17T10:44:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4886144-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssmn2ebkk93lce8296nq51hgn6drh0s-h0v7v4i7f0r3k1n", pod="kube-state-metrics-hcp-8654bffc85-455s6", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssmn2ebkk93lce8296nq51hgn6drh0s-h0v7v4i7f0r3k1n/kube-state-metrics-hcp-8654bffc85-455s6 has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-09-17T10:08:57Z
  Ended:       2026-09-17T10:39:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4886144-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssmn2dtiv895o9opovjsku0grrv59ti-d1u2b7b9s9t1i6v", pod="kube-state-metrics-hcp-8654bffc85-jrdhx", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssmn2dtiv895o9opovjsku0grrv59ti-d1u2b7b9s9t1i6v/kube-state-metrics-hcp-8654bffc85-jrdhx has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-09-17T10:08:57Z
  Ended:       2026-09-17T10:36:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4886144-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssmn1sn5ob4vstc7h5h86vq6ul0ec9l-r4u4j1g4n7o4j5e", pod="kube-state-metrics-hcp-8654bffc85-gzx69", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssmn1sn5ob4vstc7h5h86vq6ul0ec9l-r4u4j1g4n7o4j5e/kube-state-metrics-hcp-8654bffc85-gzx69 has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-09-17T10:08:57Z
  Ended:       2026-09-17T10:16:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4886144-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssmn1l9uim9epfk6ldii7gv21mo14ju-i0d3e1u2c7q6u2i", pod="kube-state-metrics-hcp-8654bffc85-kdm8q", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssmn1l9uim9epfk6ldii7gv21mo14ju-i0d3e1u2c7q6u2i/kube-state-metrics-hcp-8654bffc85-kdm8q has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-09-17T10:08:57Z
  Ended:       2026-09-17T10:30:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4886144-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssmn0a8oetivdoevi265tngl26ffi2p-h2z5a1f2o3b0l9t", pod="kube-state-metrics-hcp-8654bffc85-74m8n", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssmn0a8oetivdoevi265tngl26ffi2p-h2z5a1f2o3b0l9t/kube-state-metrics-hcp-8654bffc85-74m8n has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-09-17T10:08:57Z
  Ended:       2026-09-17T10:37:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4886144-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssmn0dh3betp1ptm9f44oadl6fegvfm-j8p8y9g1m8c4u5q", pod="kube-state-metrics-hcp-8654bffc85-4t685", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssmn0dh3betp1ptm9f44oadl6fegvfm-j8p8y9g1m8c4u5q/kube-state-metrics-hcp-8654bffc85-4t685 has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-09-17T10:08:57Z
  Ended:       2026-09-17T10:23:57Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4886144-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssmmvum7ql5nct035o7hm4r4o8r150u-ea-listn6n2j7", pod="kube-state-metrics-hcp-8654bffc85-zl25q", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssmmvum7ql5nct035o7hm4r4o8r150u-ea-listn6n2j7/kube-state-metrics-hcp-8654bffc85-zl25q has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-09-17T10:08:57Z
  Ended:       2026-09-17T10:31:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4886144-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssmn1c6p35ld76g44ebp5k34j1cf8qt-g8b0m5e6u5p0z3y", pod="kube-state-metrics-hcp-8654bffc85-p4c6z", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssmn1c6p35ld76g44ebp5k34j1cf8qt-g8b0m5e6u5p0z3y/kube-state-metrics-hcp-8654bffc85-p4c6z has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-09-17T10:08:57Z
  Ended:       2026-09-17T10:24:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4886144-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssmn014in935aglqcdgufni4ccv83p7-i2p1y4q9y9l0q5m", pod="kube-state-metrics-hcp-8654bffc85-jpxbn", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssmn014in935aglqcdgufni4ccv83p7-i2p1y4q9y9l0q5m/kube-state-metrics-hcp-8654bffc85-jpxbn has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-09-17T10:08:57Z
  Ended:       2026-09-17T10:40:57Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4886144-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssmn1mrt1i79ptk4i2pr4va3ngq3o3m-j5t3c4o7e0t3l5x", pod="kube-state-metrics-hcp-8654bffc85-gsnkm", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssmn1mrt1i79ptk4i2pr4va3ngq3o3m-j5t3c4o7e0t3l5x/kube-state-metrics-hcp-8654bffc85-gsnkm has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-09-17T10:09:57Z
  Ended:       2026-09-17T10:23:57Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4886144-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssmn1ena1dldp86nrb26jva7n63m03k-sre-hcp-cluster", pod="kube-state-metrics-hcp-8654bffc85-744rg", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssmn1ena1dldp86nrb26jva7n63m03k-sre-hcp-cluster/kube-state-metrics-hcp-8654bffc85-744rg has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Resolved
  Started:     2026-09-17T10:09:57Z
  Ended:       2026-09-17T10:40:57Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4886144-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssmn3du4o1fuau7gvajqpovmd57pvk7-d4y4k6z7w3p1e4v", pod="kube-state-metrics-hcp-8654bffc85-9fzq9", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssmn3du4o1fuau7gvajqpovmd57pvk7-d4y4k6z7w3p1e4v/kube-state-metrics-hcp-8654bffc85-9fzq9 has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Resolved
  Started:     2026-09-17T10:09:57Z
  Ended:       2026-09-17T10:34:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4886144-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssmn1cg383r0cnqbhkk9rq96jb02224-ephemeral-disk", pod="kube-state-metrics-hcp-8654bffc85-4sc5p", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssmn1cg383r0cnqbhkk9rq96jb02224-ephemeral-disk/kube-state-metrics-hcp-8654bffc85-4sc5p has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Resolved
  Started:     2026-09-17T10:09:57Z
  Ended:       2026-09-17T10:44:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4886144-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssmn1lks0dla8jmptpn1827rsg91g09-u7n9o6f3e8a7w8o", pod="kube-state-metrics-hcp-8654bffc85-kvcc7", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssmn1lks0dla8jmptpn1827rsg91g09-u7n9o6f3e8a7w8o/kube-state-metrics-hcp-8654bffc85-kvcc7 has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Resolved
  Started:     2026-09-17T10:09:57Z
  Ended:       2026-09-17T10:19:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4886144-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssmnenguljcne093trac0vrc3vmiqif-b8u4p1e5z5l6f9f", pod="kube-state-metrics-hcp-8654bffc85-mbx9d", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssmnenguljcne093trac0vrc3vmiqif-b8u4p1e5z5l6f9f/kube-state-metrics-hcp-8654bffc85-mbx9d has been in a non-ready state for longer than 5 minutes.
Firing 21:
  State:       Resolved
  Started:     2026-09-17T10:09:57Z
  Ended:       2026-09-17T10:40:57Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4886144-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssmn2fun0lvh3iuc5tcu6acte0905nh-a7v6l2i4p6l1l5z", pod="kube-state-metrics-hcp-8654bffc85-w28ps", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssmn2fun0lvh3iuc5tcu6acte0905nh-a7v6l2i4p6l1l5z/kube-state-metrics-hcp-8654bffc85-w28ps has been in a non-ready state for longer than 5 minutes.
Firing 22:
  State:       Resolved
  Started:     2026-09-17T10:09:57Z
  Ended:       2026-09-17T10:28:57Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4886144-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssmn36hu0p06ncif3i0ji9vedp2ec8n-oidc-wi-cluster", pod="kube-state-metrics-hcp-8654bffc85-v5m62", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssmn36hu0p06ncif3i0ji9vedp2ec8n-oidc-wi-cluster/kube-state-metrics-hcp-8654bffc85-v5m62 has been in a non-ready state for longer than 5 minutes.
Firing 23:
  State:       Resolved
  Started:     2026-09-17T10:09:57Z
  Ended:       2026-09-17T10:33:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4886144-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssmn2i8juokauhkpbv9udmlep1f7kqs-ea-cluster", pod="kube-state-metrics-hcp-8654bffc85-kkfz9", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssmn2i8juokauhkpbv9udmlep1f7kqs-ea-cluster/kube-state-metrics-hcp-8654bffc85-kkfz9 has been in a non-ready state for longer than 5 minutes.
Firing 24:
  State:       Resolved
  Started:     2026-09-17T10:09:57Z
  Ended:       2026-09-17T10:24:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4886144-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssmn0csmnpuis4egfkdtn7dohu0naou-basic-cluster", pod="kube-state-metrics-hcp-8654bffc85-2zh2s", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssmn0csmnpuis4egfkdtn7dohu0naou-basic-cluster/kube-state-metrics-hcp-8654bffc85-2zh2s has been in a non-ready state for longer than 5 minutes.
Firing 25:
  State:       Resolved
  Started:     2026-09-17T10:09:57Z
  Ended:       2026-09-17T10:50:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4886144-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssmn2eo3ft0d2qqqcc969fi4iis8kpa-d4a2v4u3r6l4c2x", pod="kube-state-metrics-hcp-8654bffc85-z6qpw", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssmn2eo3ft0d2qqqcc969fi4iis8kpa-d4a2v4u3r6l4c2x/kube-state-metrics-hcp-8654bffc85-z6qpw has been in a non-ready state for longer than 5 minutes.
Firing 26:
  State:       Resolved
  Started:     2026-09-17T10:09:57Z
  Ended:       2026-09-17T10:36:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4886144-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssmn0f7r1saveq4f93saqmmcarihbga-o3n9l2e0x5h2v6i", pod="router-598fb44f9c-jxfpf", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssmn0f7r1saveq4f93saqmmcarihbga-o3n9l2e0x5h2v6i/router-598fb44f9c-jxfpf has been in a non-ready state for longer than 5 minutes.
Firing 27:
  State:       Resolved
  Started:     2026-09-17T10:09:57Z
  Ended:       2026-09-17T10:27:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4886144-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssmn1bc6l7vdi3pgb6sqmm108clbehj-a8b1s1y4e6n7l6s", pod="kube-state-metrics-hcp-8654bffc85-njk9j", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssmn1bc6l7vdi3pgb6sqmm108clbehj-a8b1s1y4e6n7l6s/kube-state-metrics-hcp-8654bffc85-njk9j has been in a non-ready state for longer than 5 minutes.
Firing 28:
  State:       Resolved
  Started:     2026-09-17T10:09:57Z
  Ended:       2026-09-17T10:28:57Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4886144-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssmn1bm3iapvu18q4u88edtt7ro8tfq-h6x3s7w6l7s0b0u", pod="kube-state-metrics-hcp-8654bffc85-lj5lp", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssmn1bm3iapvu18q4u88edtt7ro8tfq-h6x3s7w6l7s0b0u/kube-state-metrics-hcp-8654bffc85-lj5lp has been in a non-ready state for longer than 5 minutes.
Firing 29:
  State:       Resolved
  Started:     2026-09-17T10:09:57Z
  Ended:       2026-09-17T10:26:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4886144-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssmn054nt2b6ab14cr4f7ba73sjotgn-j2v4q1n9j1w2o3m", pod="kube-state-metrics-hcp-8654bffc85-zjhn6", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssmn054nt2b6ab14cr4f7ba73sjotgn-j2v4q1n9j1w2o3m/kube-state-metrics-hcp-8654bffc85-zjhn6 has been in a non-ready state for longer than 5 minutes.
Firing 30:
  State:       Resolved
  Started:     2026-09-17T10:09:57Z
  Ended:       2026-09-17T10:18:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4886144-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssmnel48n6tj55ort1inpjfgcloo0nb-o1y1y5i1r8x1f6z", pod="kube-state-metrics-hcp-8654bffc85-5tvq8", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssmnel48n6tj55ort1inpjfgcloo0nb-o1y1y5i1r8x1f6z/kube-state-metrics-hcp-8654bffc85-5tvq8 has been in a non-ready state for longer than 5 minutes.
Firing 31:
  State:       Resolved
  Started:     2026-09-17T10:09:57Z
  Ended:       2026-09-17T10:44:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4886144-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssmn2dvkel9cdp8tmh7djfjurd4cbb9-f7d4o7x7m2n2v8n", pod="kube-state-metrics-hcp-8654bffc85-vwpl8", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssmn2dvkel9cdp8tmh7djfjurd4cbb9-f7d4o7x7m2n2v8n/kube-state-metrics-hcp-8654bffc85-vwpl8 has been in a non-ready state for longer than 5 minutes.
Firing 32:
  State:       Resolved
  Started:     2026-09-17T10:09:57Z
  Ended:       2026-09-17T10:33:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4886144-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssmn07khhr5r6rjbrjh69c3akabp0o5-i9o1j9m3z5t7w4c", pod="kube-state-metrics-hcp-8654bffc85-xtdn8", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssmn07khhr5r6rjbrjh69c3akabp0o5-i9o1j9m3z5t7w4c/kube-state-metrics-hcp-8654bffc85-xtdn8 has been in a non-ready state for longer than 5 minutes.
Firing 33:
  State:       Resolved
  Started:     2026-09-17T10:09:57Z
  Ended:       2026-09-17T10:40:57Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4886144-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssmndjv5lflbbttaeo5260nl2skjbjs-t7j8s8i3n9h3y7e", pod="kube-state-metrics-hcp-8654bffc85-vxtdf", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssmndjv5lflbbttaeo5260nl2skjbjs-t7j8s8i3n9h3y7e/kube-state-metrics-hcp-8654bffc85-vxtdf has been in a non-ready state for longer than 5 minutes.
Firing 34:
  State:       Resolved
  Started:     2026-09-17T10:09:57Z
  Ended:       2026-09-17T10:27:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4886144-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssmn394i64jkp8u1qr0g5bup79cthmm-z7d8o5c2y4o4w8y", pod="kube-state-metrics-hcp-8654bffc85-vnrfg", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssmn394i64jkp8u1qr0g5bup79cthmm-z7d8o5c2y4o4w8y/kube-state-metrics-hcp-8654bffc85-vnrfg has been in a non-ready state for longer than 5 minutes.
Firing 35:
  State:       Resolved
  Started:     2026-09-17T10:09:57Z
  Ended:       2026-09-17T10:31:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4886144-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssmn081b3qs4ov26quq8rkkd6kmvgkv-private-ingress", pod="kube-state-metrics-hcp-8654bffc85-2bvxq", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssmn081b3qs4ov26quq8rkkd6kmvgkv-private-ingress/kube-state-metrics-hcp-8654bffc85-2bvxq has been in a non-ready state for longer than 5 minutes.
Firing 36:
  State:       Resolved
  Started:     2026-09-17T10:09:57Z
  Ended:       2026-09-17T10:31:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4886144-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssmn2p7oql665o9t78sdk9ag3gtrap1-f5u5y6g6q2m8s6o", pod="kube-state-metrics-hcp-8654bffc85-f5bph", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssmn2p7oql665o9t78sdk9ag3gtrap1-f5u5y6g6q2m8s6o/kube-state-metrics-hcp-8654bffc85-f5bph has been in a non-ready state for longer than 5 minutes.
Firing 37:
  State:       Resolved
  Started:     2026-09-17T10:09:57Z
  Ended:       2026-09-17T10:16:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4886144-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssmn2qsgbh3jvrunhd5vigk8r5hcn7p-s6h4n3m0q1q5r1v", pod="kube-state-metrics-hcp-8654bffc85-7zchh", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssmn2qsgbh3jvrunhd5vigk8r5hcn7p-s6h4n3m0q1q5r1v/kube-state-metrics-hcp-8654bffc85-7zchh has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T09:42:54Z
  Ended:       2026-09-17T10:05:56Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7297536-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssmahmkh6lr5u979h67h90nmbs1to8p-k8u8t2z1g0o1j4b", pod="router-5b765bf64f-lhx9q", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssmahmkh6lr5u979h67h90nmbs1to8p-k8u8t2z1g0o1j4b/router-5b765bf64f-lhx9q has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000b25000>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-528t9hkgrzh7" to finish creating: GET https://rp.j9752576.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0cd7741f-18ff-46d2-82fa-6693b55efc5d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0cd7741f-18ff-46d2-82fa-6693b55efc5d",
      "name": "0cd7741f-18ff-46d2-82fa-6693b55efc5d",
      "status": "Failed",
      "startTime": "2026-09-17T09:29:27.354777917Z",
      "endTime": "2026-09-17T09:58:36.226237209Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 24m11s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000b25000>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-528t9hkgrzh7" to finish creating: GET https://rp.j9752576.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0cd7741f-18ff-46d2-82fa-6693b55efc5d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0cd7741f-18ff-46d2-82fa-6693b55efc5d",
      "name": "0cd7741f-18ff-46d2-82fa-6693b55efc5d",
      "status": "Failed",
      "startTime": "2026-09-17T09:29:27.354777917Z",
      "endTime": "2026-09-17T09:58:36.226237209Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 24m11s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:89]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc000d1c220>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-hsk75k5rsxz7" to finish creating: GET https://rp.j9752576.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fa3193c1-5588-48f4-a832-82d0c67fad40
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fa3193c1-5588-48f4-a832-82d0c67fad40",
      "name": "fa3193c1-5588-48f4-a832-82d0c67fad40",
      "status": "Failed",
      "startTime": "2026-09-17T09:28:04.923722812Z",
      "endTime": "2026-09-17T09:57:06.067298025Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 22m52s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:89]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc000d1c220>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-hsk75k5rsxz7" to finish creating: GET https://rp.j9752576.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fa3193c1-5588-48f4-a832-82d0c67fad40
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fa3193c1-5588-48f4-a832-82d0c67fad40",
      "name": "fa3193c1-5588-48f4-a832-82d0c67fad40",
      "status": "Failed",
      "startTime": "2026-09-17T09:28:04.923722812Z",
      "endTime": "2026-09-17T09:57:06.067298025Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 22m52s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000988920>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-fhpcws56rm7z" to finish creating: GET https://rp.j9752576.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/88694765-6437-49ac-aac4-0c9114cbb8f3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/88694765-6437-49ac-aac4-0c9114cbb8f3",
      "name": "88694765-6437-49ac-aac4-0c9114cbb8f3",
      "status": "Failed",
      "startTime": "2026-09-17T09:28:10.326777677Z",
      "endTime": "2026-09-17T09:57:16.136118228Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 22m53s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000988920>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-fhpcws56rm7z" to finish creating: GET https://rp.j9752576.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/88694765-6437-49ac-aac4-0c9114cbb8f3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/88694765-6437-49ac-aac4-0c9114cbb8f3",
      "name": "88694765-6437-49ac-aac4-0c9114cbb8f3",
      "status": "Failed",
      "startTime": "2026-09-17T09:28:10.326777677Z",
      "endTime": "2026-09-17T09:57:16.136118228Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 22m53s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T09:38:40Z
  Ended:       2026-09-17T10:01:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9752576-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssm928m721lk4068dgpl6613ujbla6c-d5p5f1n3p4f6a6b", pod="router-54c87cf596-6dx4l", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssm928m721lk4068dgpl6613ujbla6c-d5p5f1n3p4f6a6b/router-54c87cf596-6dx4l has been in a non-ready state for longer than 5 minutes.

time=2026-09-17T09:16:31.171Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n"
time=2026-09-17T09:16:31.636Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy values="map[affinity:map[] auth:map[audience:6dae42f8-4368-4678-94ff-3960e28e3630 authority:https://sts.windows.net/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] azure:map[clientId:53204456-a0a3-4fd2-89f7-36eb569bbb5d cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] credsKeyVault:map[name:aro-hcp-dev-svc-kv secret:sessiongate-cert-ci01-j2597888] deployment:map[replicas:2 requests:map[cpu:250m memory:512Mi]] httpRoute:map[gateway:map[name:ops-ingress-gateway namespace:aks-istio-ingress sectionName:sessiongate-https] host:sessiongate.j2597888.hcpsvc.osadev.cloud] image:map[digest:sha256:dfb4a580c4ae075df76e82458246c28e24407e254922d6c888d0d2793d3c1737 registry:arohcpsvcdev.azurecr.io repository:ci-op-6i9x21qi/pipeline] leaderElection:map[leaseDuration:15s renewDeadline:10s retryPeriod:2s] logLevel:2 mise:map[deploy:false policyLabel:Session Gate] nodeSelector:map[] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-6i9x21qi/pipeline:pull workloadIdentityClientId:0ee3f5a0-d80b-4385-abcc-91385fcf6f70 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] service:map[name:sessiongate port:8080] serviceAccount:map[name:sessiongate workloadIdentityClientId:15494d5f-cf01-4a47-ae22-344126602800 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] tolerations:[]]"
time=2026-09-17T09:16:31.636Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy
time=2026-09-17T09:16:31.786Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy namespace=sessiongate
time=2026-09-17T09:16:31.977Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy dryRun=false
time=2026-09-17T09:16:31.977Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy
time=2026-09-17T09:16:32.031Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy
time=2026-09-17T09:16:37.667Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy err="Internal error occurred: failed calling webhook \"validation.istio.io\": failed to call webhook: Post \"https://istiod-asm-1-29.aks-istio-system.svc:443/validate?timeout=10s\": EOF"
time=2026-09-17T09:16:37.667Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy err="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: Internal error occurred: failed calling webhook \"validation.istio.io\": failed to call webhook: Post \"https://istiod-asm-1-29.aks-istio-system.svc:443/validate?timeout=10s\": EOF"

time=2026-09-17T08:52:09.303Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 description="Step cluster\n  Kind: ARM\n  Template: templates/mgmt-cluster.bicep\n  Parameters: configurations/mgmt-cluster.tmpl.bicepparam"
time=2026-09-17T08:52:10.143Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1
time=2026-09-17T08:52:11.020Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 deployment=e5c452463d5a2ca118d246d0e10c4f76fe32f1478279da2b4450028ca1d7e3f8 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j2728320-mgmt-1%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fe5c452463d5a2ca118d246d0e10c4f76fe32f1478279da2b4450028ca1d7e3f8
time=2026-09-17T08:54:11.408Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 err="stamp 1: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2728320-mgmt-1/providers/Microsoft.Resources/deployments/e5c452463d5a2ca118d246d0e10c4f76fe32f1478279da2b4450028ca1d7e3f8/operationStatuses/08584119717548817375\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"DeploymentActive\\\",\\r\\n    \\\"message\\\": \\\"The deployment with resource id '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2728320-mgmt-1/providers/Microsoft.Resources/deployments/cluster-34wn2l5qqmupi' cannot be saved, because this would overwrite an existing deployment which is still active. If this is an intentional re-deployment, please wait for the original deployment to complete, or cancel it before re-attempting the operation. The previous deployment was started at '9/17/2026 8:52:28 AM' with correlationId 'd7f46ae1-ae89-4906-81b7-ab9a15265766', and will expire at '9/24/2026 8:52:28 AM' if it does not complete before then. Please see https://aka.ms/arm-deploy-resources for usage details.\\\"\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0015dc168>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-52qn5lk4557x" to finish deleting: GET https://rp.j0442496.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2b8b5999-969b-4fd7-a0ef-03666a4c36ea
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2b8b5999-969b-4fd7-a0ef-03666a4c36ea",
      "name": "2b8b5999-969b-4fd7-a0ef-03666a4c36ea",
      "status": "Failed",
      "startTime": "2026-09-17T09:16:16.071243144Z",
      "endTime": "2026-09-17T09:40:24.650679622Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssluvuv49qehd1o6ihpsp6u0fhsuku7 still exists (deletion dispatched at 2026-09-17T09:16:16Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0015dc168>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-52qn5lk4557x" to finish deleting: GET https://rp.j0442496.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2b8b5999-969b-4fd7-a0ef-03666a4c36ea
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2b8b5999-969b-4fd7-a0ef-03666a4c36ea",
      "name": "2b8b5999-969b-4fd7-a0ef-03666a4c36ea",
      "status": "Failed",
      "startTime": "2026-09-17T09:16:16.071243144Z",
      "endTime": "2026-09-17T09:40:24.650679622Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssluvuv49qehd1o6ihpsp6u0fhsuku7 still exists (deletion dispatched at 2026-09-17T09:16:16Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00101a2b8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cl" in resourcegroup="cni-cilium-fth2ns8rql9z" to finish deleting: GET https://rp.j0442496.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e8997c6b-6d2a-4f02-afe2-3927067cda87
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e8997c6b-6d2a-4f02-afe2-3927067cda87",
      "name": "e8997c6b-6d2a-4f02-afe2-3927067cda87",
      "status": "Failed",
      "startTime": "2026-09-17T09:27:47.803109699Z",
      "endTime": "2026-09-17T09:51:54.618583456Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sslv6ma6mn5k5mt36luuom09289pq58 still exists (deletion dispatched at 2026-09-17T09:27:48Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00101a2b8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cl" in resourcegroup="cni-cilium-fth2ns8rql9z" to finish deleting: GET https://rp.j0442496.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e8997c6b-6d2a-4f02-afe2-3927067cda87
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e8997c6b-6d2a-4f02-afe2-3927067cda87",
      "name": "e8997c6b-6d2a-4f02-afe2-3927067cda87",
      "status": "Failed",
      "startTime": "2026-09-17T09:27:47.803109699Z",
      "endTime": "2026-09-17T09:51:54.618583456Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sslv6ma6mn5k5mt36luuom09289pq58 still exists (deletion dispatched at 2026-09-17T09:27:48Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0025dc138>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-2wlqnnchcnfz" to finish deleting: GET https://rp.j0442496.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/06cacc7f-3109-44fc-aa1f-226b6d9287df
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/06cacc7f-3109-44fc-aa1f-226b6d9287df",
      "name": "06cacc7f-3109-44fc-aa1f-226b6d9287df",
      "status": "Failed",
      "startTime": "2026-09-17T09:13:15.932719086Z",
      "endTime": "2026-09-17T09:37:24.699046818Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sslv0qgf0brht8l82lblg9u4aedu59m still exists (deletion dispatched at 2026-09-17T09:13:16Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0025dc138>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-2wlqnnchcnfz" to finish deleting: GET https://rp.j0442496.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/06cacc7f-3109-44fc-aa1f-226b6d9287df
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/06cacc7f-3109-44fc-aa1f-226b6d9287df",
      "name": "06cacc7f-3109-44fc-aa1f-226b6d9287df",
      "status": "Failed",
      "startTime": "2026-09-17T09:13:15.932719086Z",
      "endTime": "2026-09-17T09:37:24.699046818Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sslv0qgf0brht8l82lblg9u4aedu59m still exists (deletion dispatched at 2026-09-17T09:13:16Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00112c090>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-8bctrm" in resourcegroup="rg-zstream-upgrade-4-20-2lc8kzcjbshc" to finish deleting: GET https://rp.j0442496.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/83ec0b10-2ee0-45fb-aaa3-b482a6f739da
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/83ec0b10-2ee0-45fb-aaa3-b482a6f739da",
      "name": "83ec0b10-2ee0-45fb-aaa3-b482a6f739da",
      "status": "Failed",
      "startTime": "2026-09-17T09:36:14.84381025Z",
      "endTime": "2026-09-17T10:00:14.610296792Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sslv1q0m5l3l4h7q2qhv5phss5tml71 still exists (deletion dispatched at 2026-09-17T09:36:15Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00112c090>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-8bctrm" in resourcegroup="rg-zstream-upgrade-4-20-2lc8kzcjbshc" to finish deleting: GET https://rp.j0442496.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/83ec0b10-2ee0-45fb-aaa3-b482a6f739da
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/83ec0b10-2ee0-45fb-aaa3-b482a6f739da",
      "name": "83ec0b10-2ee0-45fb-aaa3-b482a6f739da",
      "status": "Failed",
      "startTime": "2026-09-17T09:36:14.84381025Z",
      "endTime": "2026-09-17T10:00:14.610296792Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sslv1q0m5l3l4h7q2qhv5phss5tml71 still exists (deletion dispatched at 2026-09-17T09:36:15Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0015080a8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-mhtb5rs5742r" to finish deleting: GET https://rp.j0442496.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/52401453-231e-453f-85cc-7914f8e3b298
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/52401453-231e-453f-85cc-7914f8e3b298",
      "name": "52401453-231e-453f-85cc-7914f8e3b298",
      "status": "Failed",
      "startTime": "2026-09-17T09:19:54.501526397Z",
      "endTime": "2026-09-17T09:43:54.654516896Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssluvf28mbs167aoq0cpodshs5mhpdb still exists (deletion dispatched at 2026-09-17T09:19:54Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0015080a8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-mhtb5rs5742r" to finish deleting: GET https://rp.j0442496.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/52401453-231e-453f-85cc-7914f8e3b298
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/52401453-231e-453f-85cc-7914f8e3b298",
      "name": "52401453-231e-453f-85cc-7914f8e3b298",
      "status": "Failed",
      "startTime": "2026-09-17T09:19:54.501526397Z",
      "endTime": "2026-09-17T09:43:54.654516896Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssluvf28mbs167aoq0cpodshs5mhpdb still exists (deletion dispatched at 2026-09-17T09:19:54Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0015da030>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-2vshj8kd4ngg" to finish deleting: GET https://rp.j0442496.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2b25208f-0e8c-46c2-96d1-92647ed090d2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2b25208f-0e8c-46c2-96d1-92647ed090d2",
      "name": "2b25208f-0e8c-46c2-96d1-92647ed090d2",
      "status": "Failed",
      "startTime": "2026-09-17T09:42:31.310516975Z",
      "endTime": "2026-09-17T10:06:34.577233142Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0015da030>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-2vshj8kd4ngg" to finish deleting: GET https://rp.j0442496.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2b25208f-0e8c-46c2-96d1-92647ed090d2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2b25208f-0e8c-46c2-96d1-92647ed090d2",
      "name": "2b25208f-0e8c-46c2-96d1-92647ed090d2",
      "status": "Failed",
      "startTime": "2026-09-17T09:42:31.310516975Z",
      "endTime": "2026-09-17T10:06:34.577233142Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 10 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-17T09:55:32Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0442496-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.4:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0442496/providers/microsoft.monitor/accounts/hcps-j0442496", namespace="ocm-arohcpci01-2sslv0qgf0brht8l82lblg9u4aedu59m-l0t6g2b3x9j1i5r", pod="arohcp-monitor-kube-state-metrics-574f657d96-fstb6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2sslv0qgf0brht8l82lblg9u4aedu59m-l0t6g2b3x9j1i5r/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-17T10:06:32Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0442496-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.4:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0442496/providers/microsoft.monitor/accounts/hcps-j0442496", namespace="ocm-arohcpci01-2sslv0qgf0brht8l82lblg9u4aedu59m-l0t6g2b3x9j1i5r", pod="arohcp-monitor-kube-state-metrics-574f657d96-fstb6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2sslv0qgf0brht8l82lblg9u4aedu59m-l0t6g2b3x9j1i5r/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-17T10:16:31Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0442496-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.4:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0442496/providers/microsoft.monitor/accounts/hcps-j0442496", namespace="ocm-arohcpci01-2sslv1q0m5l3l4h7q2qhv5phss5tml71-w3t5e3f8l5j8d5r", pod="arohcp-monitor-kube-state-metrics-574f657d96-fstb6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2sslv1q0m5l3l4h7q2qhv5phss5tml71-w3t5e3f8l5j8d5r/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-17T10:16:31Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0442496-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.4:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0442496/providers/microsoft.monitor/accounts/hcps-j0442496", namespace="ocm-arohcpci01-2sslv1q0m5l3l4h7q2qhv5phss5tml71-w3t5e3f8l5j8d5r", pod="arohcp-monitor-kube-state-metrics-574f657d96-fstb6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2sslv1q0m5l3l4h7q2qhv5phss5tml71-w3t5e3f8l5j8d5r/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-09-17T10:19:32Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0442496-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.37:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0442496/providers/microsoft.monitor/accounts/hcps-j0442496", namespace="ocm-arohcpci01-2sslv6ma6mn5k5mt36luuom09289pq58-cilium-cl", pod="arohcp-monitor-kube-state-metrics-574f657d96-v4f24", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2sslv6ma6mn5k5mt36luuom09289pq58-cilium-cl/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-09-17T10:19:32Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0442496-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.37:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0442496/providers/microsoft.monitor/accounts/hcps-j0442496", namespace="ocm-arohcpci01-2sslv6ma6mn5k5mt36luuom09289pq58-cilium-cl", pod="arohcp-monitor-kube-state-metrics-574f657d96-v4f24", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2sslv6ma6mn5k5mt36luuom09289pq58-cilium-cl/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 7:
  State:       Fired (not resolved)
  Started:     2026-09-17T10:24:34Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0442496-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.4:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0442496/providers/microsoft.monitor/accounts/hcps-j0442496", namespace="ocm-arohcpci01-2sslv1q0m5l3l4h7q2qhv5phss5tml71-w3t5e3f8l5j8d5r", pod="arohcp-monitor-kube-state-metrics-574f657d96-fstb6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2sslv1q0m5l3l4h7q2qhv5phss5tml71-w3t5e3f8l5j8d5r/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 8:
  State:       Fired (not resolved)
  Started:     2026-09-17T10:24:34Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0442496-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.4:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0442496/providers/microsoft.monitor/accounts/hcps-j0442496", namespace="ocm-arohcpci01-2sslv0qgf0brht8l82lblg9u4aedu59m-l0t6g2b3x9j1i5r", pod="arohcp-monitor-kube-state-metrics-574f657d96-fstb6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2sslv0qgf0brht8l82lblg9u4aedu59m-l0t6g2b3x9j1i5r/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 9:
  State:       Fired (not resolved)
  Started:     2026-09-17T10:24:34Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0442496-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.4:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0442496/providers/microsoft.monitor/accounts/hcps-j0442496", namespace="ocm-arohcpci01-2sslv0qgf0brht8l82lblg9u4aedu59m-l0t6g2b3x9j1i5r", pod="arohcp-monitor-kube-state-metrics-574f657d96-fstb6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2sslv0qgf0brht8l82lblg9u4aedu59m-l0t6g2b3x9j1i5r/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 10:
  State:       Fired (not resolved)
  Started:     2026-09-17T10:24:34Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0442496-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.4:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0442496/providers/microsoft.monitor/accounts/hcps-j0442496", namespace="ocm-arohcpci01-2sslv1q0m5l3l4h7q2qhv5phss5tml71-w3t5e3f8l5j8d5r", pod="arohcp-monitor-kube-state-metrics-574f657d96-fstb6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2sslv1q0m5l3l4h7q2qhv5phss5tml71-w3t5e3f8l5j8d5r/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

time=2026-09-17T08:26:03.998Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 description="Step cluster\n  Kind: ARM\n  Template: templates/mgmt-cluster.bicep\n  Parameters: configurations/mgmt-cluster.tmpl.bicepparam"
time=2026-09-17T08:26:04.656Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2
time=2026-09-17T08:26:05.474Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 deployment=6f88425cb7a829bd0a49692c909958854fd202dee74b8108c03378dc70806d40 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j0087168-mgmt-2%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F6f88425cb7a829bd0a49692c909958854fd202dee74b8108c03378dc70806d40
time=2026-09-17T08:56:03.650Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 err="stamp 2: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded"

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0xc000f1dd40>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-xrnlcrchtggh" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20260901AndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-xrnlcrchtggh, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0xc000f1dd40>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-xrnlcrchtggh" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20260901AndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-xrnlcrchtggh, error: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:480]: Timed out after 34.086s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc001018980>: 
    permission checks failed: SSAR failed for list /services: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /endpoints: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /events: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /persistentvolumeclaims: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for get /persistentvolumes: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /serviceaccounts: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /resourcequotas: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /limitranges: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /replicationcontrollers: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for get /componentstatuses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for get /pods/log: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apps/deployments: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apps/daemonsets: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apps/statefulsets: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apps/replicasets: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list batch/jobs: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list batch/cronjobs: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list networking.k8s.io/networkpolicies: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list networking.k8s.io/ingresses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list rbac.authorization.k8s.io/clusterroles: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list rbac.authorization.k8s.io/clusterrolebindings: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list rbac.authorization.k8s.io/roles: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list rbac.authorization.k8s.io/rolebindings: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list storage.k8s.io/storageclasses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list storage.k8s.io/volumeattachments: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apiextensions.k8s.io/customresourcedefinitions: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list policy/poddisruptionbudgets: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list autoscaling/horizontalpodautoscalers: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list coordination.k8s.io/leases: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list discovery.k8s.io/endpointslices: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list certificates.k8s.io/certificatesigningrequests: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list scheduling.k8s.io/priorityclasses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list node.k8s.io/runtimeclasses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list admissionregistration.k8s.io/mutatingwebhookconfigurations: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list admissionregistration.k8s.io/validatingwebhookconfigurations: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apiregistration.k8s.io/apiservices: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list config.openshift.io/clusterversions: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list config.openshift.io/clusteroperators: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list config.openshift.io/infrastructures: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list security.openshift.io/securitycontextconstraints: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list route.openshift.io/routes: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list image.openshift.io/imagestreams: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:480]: Timed out after 34.086s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc001018980>: 
    permission checks failed: SSAR failed for list /services: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /endpoints: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /events: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /persistentvolumeclaims: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for get /persistentvolumes: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /serviceaccounts: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /resourcequotas: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /limitranges: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /replicationcontrollers: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for get /componentstatuses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for get /pods/log: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apps/deployments: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apps/daemonsets: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apps/statefulsets: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apps/replicasets: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list batch/jobs: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list batch/cronjobs: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list networking.k8s.io/networkpolicies: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list networking.k8s.io/ingresses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list rbac.authorization.k8s.io/clusterroles: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list rbac.authorization.k8s.io/clusterrolebindings: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list rbac.authorization.k8s.io/roles: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list rbac.authorization.k8s.io/rolebindings: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list storage.k8s.io/storageclasses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list storage.k8s.io/volumeattachments: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apiextensions.k8s.io/customresourcedefinitions: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list policy/poddisruptionbudgets: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list autoscaling/horizontalpodautoscalers: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list coordination.k8s.io/leases: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list discovery.k8s.io/endpointslices: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list certificates.k8s.io/certificatesigningrequests: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list scheduling.k8s.io/priorityclasses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list node.k8s.io/runtimeclasses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list admissionregistration.k8s.io/mutatingwebhookconfigurations: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list admissionregistration.k8s.io/validatingwebhookconfigurations: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apiregistration.k8s.io/apiservices: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list config.openshift.io/clusterversions: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list config.openshift.io/clusteroperators: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list config.openshift.io/infrastructures: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list security.openshift.io/securitycontextconstraints: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list route.openshift.io/routes: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list image.openshift.io/imagestreams: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:303]: failed to create HCP cluster np-noedge-x2pvfg
Unexpected error:
    <*fmt.wrapError | 0xc00184eb80>: 
    failed to create HCP cluster np-noedge-x2pvfg: failed waiting for cluster="np-noedge-x2pvfg" in resourcegroup="rg-np-noedge-x2pvfg-jk75vhvpdzlr" to finish creating: GET https://rp.j9433728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/260d3aa8-bf59-42db-ba88-b8ed4aa36bb2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/260d3aa8-bf59-42db-ba88-b8ed4aa36bb2",
      "name": "260d3aa8-bf59-42db-ba88-b8ed4aa36bb2",
      "status": "Failed",
      "startTime": "2026-09-17T08:45:28.535156411Z",
      "endTime": "2026-09-17T09:14:36.182618821Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.38 is Partial (want Completed), started 22m33s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:303]: failed to create HCP cluster np-noedge-x2pvfg
Unexpected error:
    <*fmt.wrapError | 0xc00184eb80>: 
    failed to create HCP cluster np-noedge-x2pvfg: failed waiting for cluster="np-noedge-x2pvfg" in resourcegroup="rg-np-noedge-x2pvfg-jk75vhvpdzlr" to finish creating: GET https://rp.j9433728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/260d3aa8-bf59-42db-ba88-b8ed4aa36bb2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/260d3aa8-bf59-42db-ba88-b8ed4aa36bb2",
      "name": "260d3aa8-bf59-42db-ba88-b8ed4aa36bb2",
      "status": "Failed",
      "startTime": "2026-09-17T08:45:28.535156411Z",
      "endTime": "2026-09-17T09:14:36.182618821Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.38 is Partial (want Completed), started 22m33s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 9 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T08:58:51Z
  Ended:       2026-09-17T09:07:52Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9433728-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sslldae8h4c7a6u4p98esr6dlbc39t7-ea-listmfx27n", pod="router-76c979645c-d27x9", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sslldae8h4c7a6u4p98esr6dlbc39t7-ea-listmfx27n/router-76c979645c-d27x9 has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-17T08:58:51Z
  Ended:       2026-09-17T09:13:52Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9433728-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssll9ss2j4invmhfu1opgkbei7mac58-r3u7r4x4h3m5j1z", pod="router-85fc9c5d89-wp4zl", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssll9ss2j4invmhfu1opgkbei7mac58-r3u7r4x4h3m5j1z/router-85fc9c5d89-wp4zl has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-17T08:58:51Z
  Ended:       2026-09-17T09:21:51Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9433728-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssll67o6fihil7nbt3p62do423jiucc-g1u1f6v4j9t0t1c", pod="router-556889d545-6hhhj", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssll67o6fihil7nbt3p62do423jiucc-g1u1f6v4j9t0t1c/router-556889d545-6hhhj has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-17T08:58:51Z
  Ended:       2026-09-17T09:17:51Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9433728-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sslld54jglp10uirc5ktnad1vi9hv5g-j1j7h5z3o7a6m3n", pod="router-689f7964-vlhkj", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sslld54jglp10uirc5ktnad1vi9hv5g-j1j7h5z3o7a6m3n/router-689f7964-vlhkj has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-09-17T08:58:56Z
  Ended:       2026-09-17T09:07:52Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9433728-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssll8ksdacasa0ge86og97h1bkfg8g6-j9e3k5m4g8q6p6v", pod="router-7b7645ddf-zfnrj", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssll8ksdacasa0ge86og97h1bkfg8g6-j9e3k5m4g8q6p6v/router-7b7645ddf-zfnrj has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-09-17T08:58:56Z
  Ended:       2026-09-17T09:21:51Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9433728-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssll8ksdacasa0ge86og97h1bkfg8g6-j9e3k5m4g8q6p6v", pod="router-7b7645ddf-trkmm", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssll8ksdacasa0ge86og97h1bkfg8g6-j9e3k5m4g8q6p6v/router-7b7645ddf-trkmm has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-09-17T08:59:53Z
  Ended:       2026-09-17T09:21:51Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9433728-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssllau9fs7rn1fpd75a8c36vho4o4k0-k1j9f9n9a6h0e1v", pod="router-69f966859-bzh2g", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssllau9fs7rn1fpd75a8c36vho4o4k0-k1j9f9n9a6h0e1v/router-69f966859-bzh2g has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-09-17T09:00:50Z
  Ended:       2026-09-17T09:07:52Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9433728-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sslla15hhhslgl3hel9urgoo1djjulm-z8t2c0r8w1p2w8f", pod="router-7cb67b4d4-hf9kd", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sslla15hhhslgl3hel9urgoo1djjulm-z8t2c0r8w1p2w8f/router-7cb67b4d4-hf9kd has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-09-17T09:00:52Z
  Ended:       2026-09-17T09:21:51Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9433728-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sslladsa2k556c5vj4v7ho3b6sfht8r-u0r9j5j0p2b8l3d", pod="router-574c565495-9v9wv", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sslladsa2k556c5vj4v7ho3b6sfht8r-u0r9j5j0p2b8l3d/router-574c565495-9v9wv has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:89]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc000888a20>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-sqvbm4kfvhdf" to finish creating: GET https://rp.j3937664.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6dc4e658-28d5-4182-808e-588d74314272
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6dc4e658-28d5-4182-808e-588d74314272",
      "name": "6dc4e658-28d5-4182-808e-588d74314272",
      "status": "Failed",
      "startTime": "2026-09-17T08:39:54.562496471Z",
      "endTime": "2026-09-17T09:08:55.526703283Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 15m10s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:89]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc000888a20>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-sqvbm4kfvhdf" to finish creating: GET https://rp.j3937664.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6dc4e658-28d5-4182-808e-588d74314272
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6dc4e658-28d5-4182-808e-588d74314272",
      "name": "6dc4e658-28d5-4182-808e-588d74314272",
      "status": "Failed",
      "startTime": "2026-09-17T08:39:54.562496471Z",
      "endTime": "2026-09-17T09:08:55.526703283Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 15m10s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-22-ft92xmj5rlrq/cluster-candidate-4-22-txx6kl should provision
Unexpected error:
    <*fmt.wrapError | 0xc000fe4c40>: 
    failed to create HCP cluster cluster-candidate-4-22-txx6kl: failed waiting for cluster="cluster-candidate-4-22-txx6kl" in resourcegroup="rg-candidate-4-22-ft92xmj5rlrq" to finish creating: GET https://rp.j3937664.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ff6f3a6b-6c03-4c6e-abdc-bca5d9188d5e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ff6f3a6b-6c03-4c6e-abdc-bca5d9188d5e",
      "name": "ff6f3a6b-6c03-4c6e-abdc-bca5d9188d5e",
      "status": "Failed",
      "startTime": "2026-09-17T08:39:41.649732234Z",
      "endTime": "2026-09-17T09:08:45.653917596Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 20m42s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-22-ft92xmj5rlrq/cluster-candidate-4-22-txx6kl should provision
Unexpected error:
    <*fmt.wrapError | 0xc000fe4c40>: 
    failed to create HCP cluster cluster-candidate-4-22-txx6kl: failed waiting for cluster="cluster-candidate-4-22-txx6kl" in resourcegroup="rg-candidate-4-22-ft92xmj5rlrq" to finish creating: GET https://rp.j3937664.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ff6f3a6b-6c03-4c6e-abdc-bca5d9188d5e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ff6f3a6b-6c03-4c6e-abdc-bca5d9188d5e",
      "name": "ff6f3a6b-6c03-4c6e-abdc-bca5d9188d5e",
      "status": "Failed",
      "startTime": "2026-09-17T08:39:41.649732234Z",
      "endTime": "2026-09-17T09:08:45.653917596Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 20m42s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:98]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc0008c24e0>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-6whdwr7p9c2x" to finish creating: GET https://rp.j3937664.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8b1a37f4-5980-4efe-8b59-b3d9e8dae7b1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8b1a37f4-5980-4efe-8b59-b3d9e8dae7b1",
      "name": "8b1a37f4-5980-4efe-8b59-b3d9e8dae7b1",
      "status": "Failed",
      "startTime": "2026-09-17T08:39:41.523978916Z",
      "endTime": "2026-09-17T09:08:45.671416842Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 19m19s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:98]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc0008c24e0>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-6whdwr7p9c2x" to finish creating: GET https://rp.j3937664.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8b1a37f4-5980-4efe-8b59-b3d9e8dae7b1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8b1a37f4-5980-4efe-8b59-b3d9e8dae7b1",
      "name": "8b1a37f4-5980-4efe-8b59-b3d9e8dae7b1",
      "status": "Failed",
      "startTime": "2026-09-17T08:39:41.523978916Z",
      "endTime": "2026-09-17T09:08:45.671416842Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 19m19s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc0009fcd80>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-rnh4rjnz5xk5" to finish creating: GET https://rp.j3937664.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2b9bcac2-85ba-45b9-be4a-daba9fd56b63
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2b9bcac2-85ba-45b9-be4a-daba9fd56b63",
      "name": "2b9bcac2-85ba-45b9-be4a-daba9fd56b63",
      "status": "Failed",
      "startTime": "2026-09-17T08:40:13.740985106Z",
      "endTime": "2026-09-17T09:09:15.564920315Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 15m40s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc0009fcd80>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-rnh4rjnz5xk5" to finish creating: GET https://rp.j3937664.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2b9bcac2-85ba-45b9-be4a-daba9fd56b63
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2b9bcac2-85ba-45b9-be4a-daba9fd56b63",
      "name": "2b9bcac2-85ba-45b9-be4a-daba9fd56b63",
      "status": "Failed",
      "startTime": "2026-09-17T08:40:13.740985106Z",
      "endTime": "2026-09-17T09:09:15.564920315Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 15m40s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000cae9a0>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-bq8qv9p78d46" to finish creating: GET https://rp.j3937664.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1a80adef-bbf8-4ef2-a2e6-1b7bc53c480f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1a80adef-bbf8-4ef2-a2e6-1b7bc53c480f",
      "name": "1a80adef-bbf8-4ef2-a2e6-1b7bc53c480f",
      "status": "Failed",
      "startTime": "2026-09-17T08:39:15.923703924Z",
      "endTime": "2026-09-17T09:08:15.68167518Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m3s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000cae9a0>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-bq8qv9p78d46" to finish creating: GET https://rp.j3937664.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1a80adef-bbf8-4ef2-a2e6-1b7bc53c480f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1a80adef-bbf8-4ef2-a2e6-1b7bc53c480f",
      "name": "1a80adef-bbf8-4ef2-a2e6-1b7bc53c480f",
      "status": "Failed",
      "startTime": "2026-09-17T08:39:15.923703924Z",
      "endTime": "2026-09-17T09:08:15.68167518Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m3s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc001064da0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-2wrs4r76bf2t" to finish creating: GET https://rp.j3937664.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3db4a4c9-c362-40b1-b8b6-aa852265f51b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3db4a4c9-c362-40b1-b8b6-aa852265f51b",
      "name": "3db4a4c9-c362-40b1-b8b6-aa852265f51b",
      "status": "Failed",
      "startTime": "2026-09-17T08:41:12.504244073Z",
      "endTime": "2026-09-17T09:10:15.531158614Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 19m32s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc001064da0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-2wrs4r76bf2t" to finish creating: GET https://rp.j3937664.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3db4a4c9-c362-40b1-b8b6-aa852265f51b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3db4a4c9-c362-40b1-b8b6-aa852265f51b",
      "name": "3db4a4c9-c362-40b1-b8b6-aa852265f51b",
      "status": "Failed",
      "startTime": "2026-09-17T08:41:12.504244073Z",
      "endTime": "2026-09-17T09:10:15.531158614Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 19m32s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:123]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc0005e6520>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-r5d5srjmrcwt" to finish creating: GET https://rp.j3937664.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d7cc96eb-b40c-466a-8d83-0360c997832f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d7cc96eb-b40c-466a-8d83-0360c997832f",
      "name": "d7cc96eb-b40c-466a-8d83-0360c997832f",
      "status": "Failed",
      "startTime": "2026-09-17T08:39:41.812447063Z",
      "endTime": "2026-09-17T09:08:45.549760606Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 20m3s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:123]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc0005e6520>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-r5d5srjmrcwt" to finish creating: GET https://rp.j3937664.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d7cc96eb-b40c-466a-8d83-0360c997832f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d7cc96eb-b40c-466a-8d83-0360c997832f",
      "name": "d7cc96eb-b40c-466a-8d83-0360c997832f",
      "status": "Failed",
      "startTime": "2026-09-17T08:39:41.812447063Z",
      "endTime": "2026-09-17T09:08:45.549760606Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 20m3s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f274c0>: 
    failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-qndsd5skx2lj" to finish creating: GET https://rp.j3937664.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8dae3ebe-027c-47e5-aeaa-d0c024830d6e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8dae3ebe-027c-47e5-aeaa-d0c024830d6e",
      "name": "8dae3ebe-027c-47e5-aeaa-d0c024830d6e",
      "status": "Failed",
      "startTime": "2026-09-17T08:43:41.343182951Z",
      "endTime": "2026-09-17T09:12:45.669886954Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 19m11s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f274c0>: 
    failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-qndsd5skx2lj" to finish creating: GET https://rp.j3937664.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8dae3ebe-027c-47e5-aeaa-d0c024830d6e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8dae3ebe-027c-47e5-aeaa-d0c024830d6e",
      "name": "8dae3ebe-027c-47e5-aeaa-d0c024830d6e",
      "status": "Failed",
      "startTime": "2026-09-17T08:43:41.343182951Z",
      "endTime": "2026-09-17T09:12:45.669886954Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 19m11s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T09:11:53Z
  Ended:       2026-09-17T09:18:57Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j3937664-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j3937664-svc).

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:166]: Timed out after 654.956s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:157 with:
RunVMCommand failed for ingress connectivity test
Unexpected error:
    <*errors.errorString | 0xc00277e080>: 
    run command "e2e-runcommand-1789636815233707722" failed with exit code 28 on VM "private-ingress-test-vm" (output: "000")
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:166]: Timed out after 654.956s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:157 with:
RunVMCommand failed for ingress connectivity test
Unexpected error:
    <*errors.errorString | 0xc00277e080>: 
    run command "e2e-runcommand-1789636815233707722" failed with exit code 28 on VM "private-ingress-test-vm" (output: "000")
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:80]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0008a21c0>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-rpjln49gwzjt" to finish creating: GET https://rp.j4509824.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/244583bb-acb0-496e-b778-5295c25e9e3c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/244583bb-acb0-496e-b778-5295c25e9e3c",
      "name": "244583bb-acb0-496e-b778-5295c25e9e3c",
      "status": "Failed",
      "startTime": "2026-09-17T08:48:50.334998393Z",
      "endTime": "2026-09-17T09:17:56.954213085Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 21m35s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:80]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0008a21c0>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-rpjln49gwzjt" to finish creating: GET https://rp.j4509824.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/244583bb-acb0-496e-b778-5295c25e9e3c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/244583bb-acb0-496e-b778-5295c25e9e3c",
      "name": "244583bb-acb0-496e-b778-5295c25e9e3c",
      "status": "Failed",
      "startTime": "2026-09-17T08:48:50.334998393Z",
      "endTime": "2026-09-17T09:17:56.954213085Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 21m35s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 3 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T08:59:33Z
  Ended:       2026-09-17T09:19:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4509824-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sslmj8gp6dp7mu4eoit7s23icdlitkd-g3c4u2y6d0v6k4e", pod="router-7946f56747-hpwrd", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sslmj8gp6dp7mu4eoit7s23icdlitkd-g3c4u2y6d0v6k4e/router-7946f56747-hpwrd has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-17T08:59:33Z
  Ended:       2026-09-17T09:18:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4509824-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sslmlo65mnav23t49hihqisn8g94q9v-o7m5c7i6o3z2x0e", pod="router-55648dd5c6-cjxtr", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sslmlo65mnav23t49hihqisn8g94q9v-o7m5c7i6o3z2x0e/router-55648dd5c6-cjxtr has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-17T09:00:34Z
  Ended:       2026-09-17T09:07:35Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4509824-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sslmquivccejp0n212gvme998ra516l-ingress-cert-le", pod="router-f7754d5f6-sq8fr", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sslmquivccejp0n212gvme998ra516l-ingress-cert-le/router-f7754d5f6-sq8fr has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--f9bkp6bbzfkc/cluster-hs-pre-kdzf97 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000b10c80>: 
    failed to create HCP cluster cluster-hs-pre-kdzf97: failed waiting for cluster="cluster-hs-pre-kdzf97" in resourcegroup="rg-hs-pre--f9bkp6bbzfkc" to finish creating: GET https://rp.j0135552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bbe2937a-de13-4428-8e8f-fc3414b85c27
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bbe2937a-de13-4428-8e8f-fc3414b85c27",
      "name": "bbe2937a-de13-4428-8e8f-fc3414b85c27",
      "status": "Failed",
      "startTime": "2026-09-17T08:49:19.394818225Z",
      "endTime": "2026-09-17T09:18:28.728970828Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 19m24s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--f9bkp6bbzfkc/cluster-hs-pre-kdzf97 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000b10c80>: 
    failed to create HCP cluster cluster-hs-pre-kdzf97: failed waiting for cluster="cluster-hs-pre-kdzf97" in resourcegroup="rg-hs-pre--f9bkp6bbzfkc" to finish creating: GET https://rp.j0135552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bbe2937a-de13-4428-8e8f-fc3414b85c27
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bbe2937a-de13-4428-8e8f-fc3414b85c27",
      "name": "bbe2937a-de13-4428-8e8f-fc3414b85c27",
      "status": "Failed",
      "startTime": "2026-09-17T08:49:19.394818225Z",
      "endTime": "2026-09-17T09:18:28.728970828Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 19m24s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:123]: failed to create node pool ea-np-1
Unexpected error:
    <*fmt.wrapError | 0xc001365160>: 
    failed to create NodePool ea-np-1: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-899r7cpnfdlg" to finish creating: GET https://rp.j0135552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7c43c53b-7ee8-4dbf-9618-9d55032ad955
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7c43c53b-7ee8-4dbf-9618-9d55032ad955",
      "name": "7c43c53b-7ee8-4dbf-9618-9d55032ad955",
      "status": "Failed",
      "startTime": "2026-09-17T09:01:43.266036656Z",
      "endTime": "2026-09-17T09:20:48.655798604Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:123]: failed to create node pool ea-np-1
Unexpected error:
    <*fmt.wrapError | 0xc001365160>: 
    failed to create NodePool ea-np-1: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-899r7cpnfdlg" to finish creating: GET https://rp.j0135552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7c43c53b-7ee8-4dbf-9618-9d55032ad955
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7c43c53b-7ee8-4dbf-9618-9d55032ad955",
      "name": "7c43c53b-7ee8-4dbf-9618-9d55032ad955",
      "status": "Failed",
      "startTime": "2026-09-17T09:01:43.266036656Z",
      "endTime": "2026-09-17T09:20:48.655798604Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 2 time(s) (1 unknown, 1 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T09:04:47Z
  Ended:       2026-09-17T09:24:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0135552-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssln47hvua3f3uv7t7n5471jgo2togc-c2z4h6w5m9z5l2w", pod="router-5589fbdf8-zkfwg", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssln47hvua3f3uv7t7n5471jgo2togc-c2z4h6w5m9z5l2w/router-5589fbdf8-zkfwg has been in a non-ready state for longer than 5 minutes.

time=2026-09-17T08:00:20.528Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 description="Step cluster\n  Kind: ARM\n  Template: templates/mgmt-cluster.bicep\n  Parameters: configurations/mgmt-cluster.tmpl.bicepparam"
time=2026-09-17T08:00:21.209Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1
time=2026-09-17T08:00:22.144Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 deployment=4ef3e2deb84b2c359affabaa340d88f42db65b9a42cf6a67a1d70d0a795763f6 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j4804480-mgmt-1%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F4ef3e2deb84b2c359affabaa340d88f42db65b9a42cf6a67a1d70d0a795763f6
time=2026-09-17T08:30:20.187Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 err="stamp 1: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded"

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:141]: failed to find private KAS internal LB IP in managed resource group "private-kas-pzzdfb6jcgvd--managed"
Unexpected error:
    <*fmt.wrapError | 0xc001100480>: 
    failed to list load balancers in "private-kas-pzzdfb6jcgvd--managed": GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-pzzdfb6jcgvd--managed/providers/Microsoft.Network/loadBalancers
    --------------------------------------------------------------------------------
    RESPONSE 429: 429 Too Many Requests
    ERROR CODE: ResourceCollectionRequestsThrottled
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceCollectionRequestsThrottled",
        "message": "Operation 'Microsoft.Network/loadBalancers/read' failed as server encountered too many requests. Please try after '1' seconds. Tracking Id is '0bca3dc6-65f8-40d2-be6e-d4168ed4b62f'."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:141]: failed to find private KAS internal LB IP in managed resource group "private-kas-pzzdfb6jcgvd--managed"
Unexpected error:
    <*fmt.wrapError | 0xc001100480>: 
    failed to list load balancers in "private-kas-pzzdfb6jcgvd--managed": GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-pzzdfb6jcgvd--managed/providers/Microsoft.Network/loadBalancers
    --------------------------------------------------------------------------------
    RESPONSE 429: 429 Too Many Requests
    ERROR CODE: ResourceCollectionRequestsThrottled
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceCollectionRequestsThrottled",
        "message": "Operation 'Microsoft.Network/loadBalancers/read' failed as server encountered too many requests. Please try after '1' seconds. Tracking Id is '0bca3dc6-65f8-40d2-be6e-d4168ed4b62f'."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...

fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc0014c4860>: 
    failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-mnqlps6hprhr" to finish creating: GET https://rp.j8493568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5f8c19d4-6c42-4be9-873b-345d7742cc94
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5f8c19d4-6c42-4be9-873b-345d7742cc94",
      "name": "5f8c19d4-6c42-4be9-873b-345d7742cc94",
      "status": "Failed",
      "startTime": "2026-09-17T08:40:56.827824237Z",
      "endTime": "2026-09-17T09:10:02.669603289Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 19m41s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc0014c4860>: 
    failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-mnqlps6hprhr" to finish creating: GET https://rp.j8493568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5f8c19d4-6c42-4be9-873b-345d7742cc94
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5f8c19d4-6c42-4be9-873b-345d7742cc94",
      "name": "5f8c19d4-6c42-4be9-873b-345d7742cc94",
      "status": "Failed",
      "startTime": "2026-09-17T08:40:56.827824237Z",
      "endTime": "2026-09-17T09:10:02.669603289Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 19m41s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 2 time(s) (1 unknown, 1 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T08:52:53Z
  Ended:       2026-09-17T09:38:55Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8493568-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssliq0uldorh83h2j628oub1ngtask7-e4t5k1u0q7i8c9p", pod="router-85c9f57496-xx8v2", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssliq0uldorh83h2j628oub1ngtask7-e4t5k1u0q7i8c9p/router-85c9f57496-xx8v2 has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T09:11:27Z
  Ended:       2026-09-17T09:17:28Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j2058496-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j2058496-svc).

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.21.33
Unexpected error:
    <*fmt.wrapErrors | 0xc0017a97a0>: 
    failed waiting for nodepool="npupgrade-4-20" in cluster="np-version-upgrade-cluster-plrmgv" resourcegroup="rg-np-version-upgrade-plrmgv-2td2q96dfsp2" to finish updating, caused by: timeout '45.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool npupgrade-4-20 in cluster np-version-upgrade-cluster-plrmgv in resource group rg-np-version-upgrade-plrmgv-2td2q96dfsp2, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.21.33
Unexpected error:
    <*fmt.wrapErrors | 0xc0017a97a0>: 
    failed waiting for nodepool="npupgrade-4-20" in cluster="np-version-upgrade-cluster-plrmgv" resourcegroup="rg-np-version-upgrade-plrmgv-2td2q96dfsp2" to finish updating, caused by: timeout '45.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool npupgrade-4-20 in cluster np-version-upgrade-cluster-plrmgv in resource group rg-np-version-upgrade-plrmgv-2td2q96dfsp2, error: context deadline exceeded
    ...
occurred

alert fired 7 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T08:54:27Z
  Ended:       2026-09-17T09:00:29Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3118336-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-8t9zj", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-8t9zj (namespace arobit) on cluster ci01-j3118336-mgmt-1 has averaged 131.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-17T08:54:27Z
  Ended:       2026-09-17T09:10:26Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3118336-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-g6d4v", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-g6d4v (namespace arobit) on cluster ci01-j3118336-mgmt-2 has averaged 183.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-17T08:54:27Z
  Ended:       2026-09-17T09:04:28Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3118336-mgmt-1", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-7b547f7d8b-bz92h", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-7b547f7d8b-bz92h (namespace mgmt-agent) on cluster ci01-j3118336-mgmt-1 has averaged 183.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-17T08:54:27Z
  Ended:       2026-09-17T09:04:28Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3118336-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-86c78bc956-rx5ct", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-86c78bc956-rx5ct (namespace clusters-service) on cluster ci01-j3118336-svc has averaged 829.1% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 5:
  State:       Resolved
  Started:     2026-09-17T08:54:27Z
  Ended:       2026-09-17T09:08:29Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3118336-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-vnkdq", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-vnkdq (namespace arobit) on cluster ci01-j3118336-mgmt-1 has averaged 166.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 6:
  State:       Resolved
  Started:     2026-09-17T08:54:27Z
  Ended:       2026-09-17T09:09:27Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3118336-mgmt-2", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-7b547f7d8b-pnvkw", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-7b547f7d8b-pnvkw (namespace mgmt-agent) on cluster ci01-j3118336-mgmt-2 has averaged 166.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 7:
  State:       Resolved
  Started:     2026-09-17T08:59:26Z
  Ended:       2026-09-17T09:17:25Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3118336-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-86c78bc956-ljk5z", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-86c78bc956-ljk5z (namespace clusters-service) on cluster ci01-j3118336-svc has averaged 226.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--4qtgcwsbzxjs/cluster-hs-pre-sxxfcz should provision
Unexpected error:
    <*fmt.wrapError | 0xc00170c380>: 
    failed to create HCP cluster cluster-hs-pre-sxxfcz: failed waiting for cluster="cluster-hs-pre-sxxfcz" in resourcegroup="rg-hs-pre--4qtgcwsbzxjs" to finish creating: GET https://rp.j3085056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/89b60f6f-1529-4076-81d5-cf78bd7c7398
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/89b60f6f-1529-4076-81d5-cf78bd7c7398",
      "name": "89b60f6f-1529-4076-81d5-cf78bd7c7398",
      "status": "Failed",
      "startTime": "2026-09-17T07:42:50.029111515Z",
      "endTime": "2026-09-17T08:11:51.010935417Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 21m20s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--4qtgcwsbzxjs/cluster-hs-pre-sxxfcz should provision
Unexpected error:
    <*fmt.wrapError | 0xc00170c380>: 
    failed to create HCP cluster cluster-hs-pre-sxxfcz: failed waiting for cluster="cluster-hs-pre-sxxfcz" in resourcegroup="rg-hs-pre--4qtgcwsbzxjs" to finish creating: GET https://rp.j3085056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/89b60f6f-1529-4076-81d5-cf78bd7c7398
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/89b60f6f-1529-4076-81d5-cf78bd7c7398",
      "name": "89b60f6f-1529-4076-81d5-cf78bd7c7398",
      "status": "Failed",
      "startTime": "2026-09-17T07:42:50.029111515Z",
      "endTime": "2026-09-17T08:11:51.010935417Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 21m20s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000c95880>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-542pg25pz2bb" to finish creating: GET https://rp.j3085056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/61098ffc-0e1f-4ea9-b0ff-d3f03476730b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/61098ffc-0e1f-4ea9-b0ff-d3f03476730b",
      "name": "61098ffc-0e1f-4ea9-b0ff-d3f03476730b",
      "status": "Failed",
      "startTime": "2026-09-17T07:44:22.269785648Z",
      "endTime": "2026-09-17T08:13:31.026654866Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m1s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000c95880>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-542pg25pz2bb" to finish creating: GET https://rp.j3085056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/61098ffc-0e1f-4ea9-b0ff-d3f03476730b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/61098ffc-0e1f-4ea9-b0ff-d3f03476730b",
      "name": "61098ffc-0e1f-4ea9-b0ff-d3f03476730b",
      "status": "Failed",
      "startTime": "2026-09-17T07:44:22.269785648Z",
      "endTime": "2026-09-17T08:13:31.026654866Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m1s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 3 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T08:17:39Z
  Ended:       2026-09-17T08:27:38Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3085056-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-49zq5", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-49zq5 (namespace arobit) on cluster ci01-j3085056-svc has averaged 168.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-17T08:17:39Z
  Ended:       2026-09-17T08:27:38Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3085056-svc", component="service-memory", container="aro-hcp-backend", namespace="aro-hcp", pod="aro-hcp-backend-d4b6fd9f9-p4w2z", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-backend in pod aro-hcp-backend-d4b6fd9f9-p4w2z (namespace aro-hcp) on cluster ci01-j3085056-svc has averaged 377.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-17T08:17:39Z
  Ended:       2026-09-17T08:27:38Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3085056-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-78587f7f-wtrr7", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-78587f7f-wtrr7 (namespace clusters-service) on cluster ci01-j3085056-svc has averaged 226.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc000f4c440>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-xfhpjp2z56q6" to finish creating: GET https://rp.j9085312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0a0c90ee-9e20-4f29-aca3-8f6c86d2ab47
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0a0c90ee-9e20-4f29-aca3-8f6c86d2ab47",
      "name": "0a0c90ee-9e20-4f29-aca3-8f6c86d2ab47",
      "status": "Failed",
      "startTime": "2026-09-17T07:37:20.85368782Z",
      "endTime": "2026-09-17T08:06:23.778430257Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc000f4c440>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-xfhpjp2z56q6" to finish creating: GET https://rp.j9085312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0a0c90ee-9e20-4f29-aca3-8f6c86d2ab47
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0a0c90ee-9e20-4f29-aca3-8f6c86d2ab47",
      "name": "0a0c90ee-9e20-4f29-aca3-8f6c86d2ab47",
      "status": "Failed",
      "startTime": "2026-09-17T07:37:20.85368782Z",
      "endTime": "2026-09-17T08:06:23.778430257Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T07:58:12Z
  Ended:       2026-09-17T08:12:12Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j9085312-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.31:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9085312/providers/microsoft.monitor/accounts/hcps-j9085312", namespace="ocm-arohcpci01-2ssklcv0l1s8075vhkc4vl3pjtknmrtp-y8w9l0f7i7g1m1f", pod="arohcp-monitor-kube-state-metrics-574f657d96-2qwp8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2ssklcv0l1s8075vhkc4vl3pjtknmrtp-y8w9l0f7i7g1m1f/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-17T07:59:15Z
  Ended:       2026-09-17T08:27:12Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j9085312-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.31:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9085312/providers/microsoft.monitor/accounts/hcps-j9085312", namespace="ocm-arohcpci01-2ssklcv0l1s8075vhkc4vl3pjtknmrtp-y8w9l0f7i7g1m1f", pod="arohcp-monitor-kube-state-metrics-574f657d96-2qwp8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2ssklcv0l1s8075vhkc4vl3pjtknmrtp-y8w9l0f7i7g1m1f/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T08:13:10Z
  Ended:       2026-09-17T08:20:13Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9085312-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.31:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9085312/providers/microsoft.monitor/accounts/hcps-j9085312", namespace="ocm-arohcpci01-2ssklcv0l1s8075vhkc4vl3pjtknmrtp-y8w9l0f7i7g1m1f", pod="arohcp-monitor-kube-state-metrics-574f657d96-2qwp8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ssklcv0l1s8075vhkc4vl3pjtknmrtp-y8w9l0f7i7g1m1f/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T08:00:10Z
  Ended:       2026-09-17T08:08:08Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j9085312-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.31:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9085312/providers/microsoft.monitor/accounts/hcps-j9085312", namespace="ocm-arohcpci01-2ssklcv0l1s8075vhkc4vl3pjtknmrtp-y8w9l0f7i7g1m1f", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="61e3547c-41a9-4b9c-82c4-b22d00b533e0"
  Description: Pod ocm-arohcpci01-2ssklcv0l1s8075vhkc4vl3pjtknmrtp-y8w9l0f7i7g1m1f/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-09-17T08:00:11Z
  Ended:       2026-09-17T08:24:10Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j9085312-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.31:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9085312/providers/microsoft.monitor/accounts/hcps-j9085312", namespace="ocm-arohcpci01-2ssklcv0l1s8075vhkc4vl3pjtknmrtp-y8w9l0f7i7g1m1f", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="61e3547c-41a9-4b9c-82c4-b22d00b533e0"
  Description: Pod ocm-arohcpci01-2ssklcv0l1s8075vhkc4vl3pjtknmrtp-y8w9l0f7i7g1m1f/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").

alert fired 4 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T07:47:54Z
  Ended:       2026-09-17T07:53:52Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6077952-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-648575ffd8-pgfs2", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-648575ffd8-pgfs2 (namespace clusters-service) on cluster ci01-j6077952-svc has averaged 276.8% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-17T07:52:53Z
  Ended:       2026-09-17T08:00:52Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6077952-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-x8286", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-x8286 (namespace arobit) on cluster ci01-j6077952-mgmt-1 has averaged 128.1% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-17T07:54:53Z
  Ended:       2026-09-17T08:01:54Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6077952-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-p74sm", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-p74sm (namespace arobit) on cluster ci01-j6077952-mgmt-1 has averaged 122.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-17T08:17:51Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6077952-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-648575ffd8-526ht", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-648575ffd8-526ht (namespace clusters-service) on cluster ci01-j6077952-svc has averaged 185.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

alert fired 4 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T07:53:16Z
  Ended:       2026-09-17T08:10:12Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7413888-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-st8ft", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-st8ft (namespace arobit) on cluster ci01-j7413888-mgmt-1 has averaged 193.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-17T07:53:16Z
  Ended:       2026-09-17T08:06:16Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7413888-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-585dc64dd6-ltc9d", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-585dc64dd6-ltc9d (namespace clusters-service) on cluster ci01-j7413888-svc has averaged 242.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-17T07:53:16Z
  Ended:       2026-09-17T08:10:13Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7413888-mgmt-1", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-6bd4f4ffc9-zpn8q", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-6bd4f4ffc9-zpn8q (namespace mgmt-agent) on cluster ci01-j7413888-mgmt-1 has averaged 185.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-17T07:53:17Z
  Ended:       2026-09-17T08:02:14Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7413888-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-dqzb5", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-dqzb5 (namespace arobit) on cluster ci01-j7413888-mgmt-1 has averaged 174.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

alert fired 12 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-17T07:21:45Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3910656-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-65bc959db7-ncnk8", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-65bc959db7-ncnk8 (namespace clusters-service) on cluster ci01-j3910656-svc has averaged 841.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-17T07:21:45Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3910656-svc", component="service-memory", container="aro-hcp-backend", namespace="aro-hcp", pod="aro-hcp-backend-6f9547c9d5-nwkbn", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-backend in pod aro-hcp-backend-6f9547c9d5-nwkbn (namespace aro-hcp) on cluster ci01-j3910656-svc has averaged 441.8% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-17T07:21:45Z
  Ended:       2026-09-17T07:41:47Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3910656-mgmt-1", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-7855548969-mxztp", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-7855548969-mxztp (namespace mgmt-agent) on cluster ci01-j3910656-mgmt-1 has averaged 162.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-17T07:21:45Z
  Ended:       2026-09-17T07:42:47Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3910656-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-82s5f", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-82s5f (namespace arobit) on cluster ci01-j3910656-mgmt-1 has averaged 208.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 5:
  State:       Resolved
  Started:     2026-09-17T07:21:45Z
  Ended:       2026-09-17T07:51:46Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3910656-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-n6dpr", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-n6dpr (namespace arobit) on cluster ci01-j3910656-svc has averaged 220.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 6:
  State:       Resolved
  Started:     2026-09-17T07:22:45Z
  Ended:       2026-09-17T07:51:46Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3910656-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-65bc959db7-fwclf", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-65bc959db7-fwclf (namespace clusters-service) on cluster ci01-j3910656-svc has averaged 249.5% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 7:
  State:       Resolved
  Started:     2026-09-17T07:22:45Z
  Ended:       2026-09-17T07:51:46Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3910656-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-65bc959db7-mlkz7", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-65bc959db7-mlkz7 (namespace clusters-service) on cluster ci01-j3910656-svc has averaged 264.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 8:
  State:       Resolved
  Started:     2026-09-17T07:22:45Z
  Ended:       2026-09-17T07:39:48Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3910656-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-jb25r", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-jb25r (namespace arobit) on cluster ci01-j3910656-mgmt-2 has averaged 153.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 9:
  State:       Resolved
  Started:     2026-09-17T07:23:45Z
  Ended:       2026-09-17T07:41:46Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3910656-mgmt-2", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-7855548969-fmbtf", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-7855548969-fmbtf (namespace mgmt-agent) on cluster ci01-j3910656-mgmt-2 has averaged 165.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 10:
  State:       Resolved
  Started:     2026-09-17T07:24:47Z
  Ended:       2026-09-17T07:41:46Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3910656-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-tpp5g", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-tpp5g (namespace arobit) on cluster ci01-j3910656-mgmt-1 has averaged 144.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 11:
  State:       Resolved
  Started:     2026-09-17T07:27:46Z
  Ended:       2026-09-17T07:38:46Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3910656-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-lzpht", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-lzpht (namespace arobit) on cluster ci01-j3910656-mgmt-2 has averaged 132.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 12:
  State:       Resolved
  Started:     2026-09-17T07:28:45Z
  Ended:       2026-09-17T07:38:46Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3910656-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-8thwc", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-8thwc (namespace arobit) on cluster ci01-j3910656-mgmt-1 has averaged 128% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:598]: failed to create nodepool npdg-4-21
Unexpected error:
    <*fmt.wrapError | 0xc0001f1640>: 
    failed to create NodePool npdg-4-21: failed waiting for nodepool="npdg-4-21" for cluster "np-downgrade-tphl5x" in resourcegroup="rg-np-downgrade-tphl5x-crdlsng97qjz" to finish creating: GET https://rp.j5446400.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/277c2008-b4c1-4e9e-b6d0-fc1bfebe17ba
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/277c2008-b4c1-4e9e-b6d0-fc1bfebe17ba",
      "name": "277c2008-b4c1-4e9e-b6d0-fc1bfebe17ba",
      "status": "Failed",
      "startTime": "2026-09-17T07:01:20.069431152Z",
      "endTime": "2026-09-17T07:20:25.508659948Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:598]: failed to create nodepool npdg-4-21
Unexpected error:
    <*fmt.wrapError | 0xc0001f1640>: 
    failed to create NodePool npdg-4-21: failed waiting for nodepool="npdg-4-21" for cluster "np-downgrade-tphl5x" in resourcegroup="rg-np-downgrade-tphl5x-crdlsng97qjz" to finish creating: GET https://rp.j5446400.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/277c2008-b4c1-4e9e-b6d0-fc1bfebe17ba
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/277c2008-b4c1-4e9e-b6d0-fc1bfebe17ba",
      "name": "277c2008-b4c1-4e9e-b6d0-fc1bfebe17ba",
      "status": "Failed",
      "startTime": "2026-09-17T07:01:20.069431152Z",
      "endTime": "2026-09-17T07:20:25.508659948Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 13 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T07:24:05Z
  Ended:       2026-09-17T07:43:05Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5446400-mgmt-1", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-7f4679bf87-cxtkn", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-7f4679bf87-cxtkn (namespace mgmt-agent) on cluster ci01-j5446400-mgmt-1 has averaged 215.1% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-17T07:24:05Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5446400-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-67d4567fb5-9xjlt", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-67d4567fb5-9xjlt (namespace clusters-service) on cluster ci01-j5446400-svc has averaged 826.8% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-17T07:24:05Z
  Ended:       2026-09-17T07:51:04Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5446400-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-67d4567fb5-pd5z9", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-67d4567fb5-pd5z9 (namespace clusters-service) on cluster ci01-j5446400-svc has averaged 286.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-17T07:24:05Z
  Ended:       2026-09-17T07:38:05Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5446400-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-dcdd2", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-dcdd2 (namespace arobit) on cluster ci01-j5446400-mgmt-2 has averaged 149.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 5:
  State:       Resolved
  Started:     2026-09-17T07:24:05Z
  Ended:       2026-09-17T07:42:06Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5446400-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-swmtl", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-swmtl (namespace arobit) on cluster ci01-j5446400-mgmt-2 has averaged 154.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 6:
  State:       Resolved
  Started:     2026-09-17T07:24:05Z
  Ended:       2026-09-17T07:37:07Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5446400-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-8m78k", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-8m78k (namespace arobit) on cluster ci01-j5446400-mgmt-2 has averaged 136.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 7:
  State:       Resolved
  Started:     2026-09-17T07:24:05Z
  Ended:       2026-09-17T07:43:05Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5446400-mgmt-2", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-7f4679bf87-jrw4v", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-7f4679bf87-jrw4v (namespace mgmt-agent) on cluster ci01-j5446400-mgmt-2 has averaged 191.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 8:
  State:       Fired (not resolved)
  Started:     2026-09-17T07:24:05Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5446400-svc", component="service-memory", container="aro-hcp-backend", namespace="aro-hcp", pod="aro-hcp-backend-64877d49d4-64ffd", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-backend in pod aro-hcp-backend-64877d49d4-64ffd (namespace aro-hcp) on cluster ci01-j5446400-svc has averaged 458.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 9:
  State:       Resolved
  Started:     2026-09-17T07:24:05Z
  Ended:       2026-09-17T07:51:04Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5446400-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-67d4567fb5-qj7mp", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-67d4567fb5-qj7mp (namespace clusters-service) on cluster ci01-j5446400-svc has averaged 256.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 10:
  State:       Resolved
  Started:     2026-09-17T07:24:05Z
  Ended:       2026-09-17T07:30:06Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5446400-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-xfwt6", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-xfwt6 (namespace arobit) on cluster ci01-j5446400-mgmt-1 has averaged 183.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 11:
  State:       Resolved
  Started:     2026-09-17T07:24:05Z
  Ended:       2026-09-17T07:36:05Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5446400-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-t9csm", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-t9csm (namespace arobit) on cluster ci01-j5446400-svc has averaged 130.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 12:
  State:       Resolved
  Started:     2026-09-17T07:24:05Z
  Ended:       2026-09-17T07:46:05Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5446400-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-9cw8p", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-9cw8p (namespace arobit) on cluster ci01-j5446400-svc has averaged 196.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 13:
  State:       Resolved
  Started:     2026-09-17T07:24:05Z
  Ended:       2026-09-17T07:40:05Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5446400-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-j4w7g", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-j4w7g (namespace arobit) on cluster ci01-j5446400-mgmt-1 has averaged 182.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T07:19:22Z
  Ended:       2026-09-17T07:25:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9156736-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssjt716mpg6m63d08d6sj0snrqfpk2d-oidc-wi-cluster", pod="openshift-apiserver-74544dc599-7mnfx", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssjt716mpg6m63d08d6sj0snrqfpk2d-oidc-wi-cluster/openshift-apiserver-74544dc599-7mnfx has been in a non-ready state for longer than 5 minutes.

alert fired 11 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T07:20:27Z
  Ended:       2026-09-17T07:34:29Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9156736-mgmt-1", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-677d695895-jblm5", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-677d695895-jblm5 (namespace mgmt-agent) on cluster ci01-j9156736-mgmt-1 has averaged 164.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-17T07:20:27Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9156736-svc", component="service-memory", container="aro-hcp-backend", namespace="aro-hcp", pod="aro-hcp-backend-6dd796c54b-tjcwl", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-backend in pod aro-hcp-backend-6dd796c54b-tjcwl (namespace aro-hcp) on cluster ci01-j9156736-svc has averaged 404.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-17T07:20:27Z
  Ended:       2026-09-17T07:34:29Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9156736-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-fl7w5", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-fl7w5 (namespace arobit) on cluster ci01-j9156736-mgmt-1 has averaged 172.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-17T07:20:27Z
  Ended:       2026-09-17T07:44:26Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9156736-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-7d69566948-vrxfz", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-7d69566948-vrxfz (namespace clusters-service) on cluster ci01-j9156736-svc has averaged 240.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 5:
  State:       Fired (not resolved)
  Started:     2026-09-17T07:20:27Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9156736-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-7d69566948-q5bw5", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-7d69566948-q5bw5 (namespace clusters-service) on cluster ci01-j9156736-svc has averaged 906.8% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 6:
  State:       Resolved
  Started:     2026-09-17T07:20:27Z
  Ended:       2026-09-17T07:40:29Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9156736-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-qpgtp", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-qpgtp (namespace arobit) on cluster ci01-j9156736-svc has averaged 183% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 7:
  State:       Resolved
  Started:     2026-09-17T07:20:27Z
  Ended:       2026-09-17T07:38:27Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9156736-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-6lzzk", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-6lzzk (namespace arobit) on cluster ci01-j9156736-mgmt-2 has averaged 154.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 8:
  State:       Resolved
  Started:     2026-09-17T07:20:27Z
  Ended:       2026-09-17T07:44:26Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9156736-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-7d69566948-gsgx7", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-7d69566948-gsgx7 (namespace clusters-service) on cluster ci01-j9156736-svc has averaged 244.1% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 9:
  State:       Resolved
  Started:     2026-09-17T07:21:27Z
  Ended:       2026-09-17T07:32:28Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9156736-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-4sqn2", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-4sqn2 (namespace arobit) on cluster ci01-j9156736-mgmt-2 has averaged 128.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 10:
  State:       Resolved
  Started:     2026-09-17T07:23:28Z
  Ended:       2026-09-17T07:31:28Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9156736-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-nmkhp", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-nmkhp (namespace arobit) on cluster ci01-j9156736-mgmt-1 has averaged 123.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 11:
  State:       Resolved
  Started:     2026-09-17T07:24:27Z
  Ended:       2026-09-17T07:34:29Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9156736-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-tbpjs", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-tbpjs (namespace arobit) on cluster ci01-j9156736-mgmt-2 has averaged 134.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

alert fired 13 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T06:02:43Z
  Ended:       2026-09-17T06:14:45Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j8615424-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-j5gws", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-j5gws (namespace arobit) on cluster ci01-j8615424-mgmt-1 has averaged 143% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-17T06:02:43Z
  Ended:       2026-09-17T06:13:45Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j8615424-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-bmln4", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-bmln4 (namespace arobit) on cluster ci01-j8615424-svc has averaged 145.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-17T06:02:43Z
  Ended:       2026-09-17T06:24:45Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j8615424-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-fcbf78d68-44rlm", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-fcbf78d68-44rlm (namespace clusters-service) on cluster ci01-j8615424-svc has averaged 249.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-17T06:02:43Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j8615424-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-dqfmn", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-dqfmn (namespace arobit) on cluster ci01-j8615424-mgmt-2 has averaged 256.5% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 5:
  State:       Resolved
  Started:     2026-09-17T06:02:43Z
  Ended:       2026-09-17T06:17:45Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j8615424-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-zsp66", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-zsp66 (namespace arobit) on cluster ci01-j8615424-svc has averaged 172.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 6:
  State:       Resolved
  Started:     2026-09-17T06:02:43Z
  Ended:       2026-09-17T06:23:47Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j8615424-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-fcbf78d68-fppd7", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-fcbf78d68-fppd7 (namespace clusters-service) on cluster ci01-j8615424-svc has averaged 232.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 7:
  State:       Resolved
  Started:     2026-09-17T06:02:43Z
  Ended:       2026-09-17T06:11:45Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j8615424-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-npwtc", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-npwtc (namespace arobit) on cluster ci01-j8615424-mgmt-1 has averaged 211.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 8:
  State:       Resolved
  Started:     2026-09-17T06:02:43Z
  Ended:       2026-09-17T06:14:44Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j8615424-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-f65kz", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-f65kz (namespace arobit) on cluster ci01-j8615424-mgmt-1 has averaged 139.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 9:
  State:       Fired (not resolved)
  Started:     2026-09-17T06:02:43Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j8615424-svc", component="service-memory", container="aro-hcp-backend", namespace="aro-hcp", pod="aro-hcp-backend-78dccff4fc-6c5dm", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-backend in pod aro-hcp-backend-78dccff4fc-6c5dm (namespace aro-hcp) on cluster ci01-j8615424-svc has averaged 411.5% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 10:
  State:       Resolved
  Started:     2026-09-17T06:02:43Z
  Ended:       2026-09-17T06:11:45Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j8615424-mgmt-1", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-5bbcf797b5-tqrgg", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-5bbcf797b5-tqrgg (namespace mgmt-agent) on cluster ci01-j8615424-mgmt-1 has averaged 214.1% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 11:
  State:       Fired (not resolved)
  Started:     2026-09-17T06:02:44Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j8615424-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-fcbf78d68-9x76k", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-fcbf78d68-9x76k (namespace clusters-service) on cluster ci01-j8615424-svc has averaged 920.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 12:
  State:       Resolved
  Started:     2026-09-17T06:02:44Z
  Ended:       2026-09-17T06:20:46Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j8615424-mgmt-2", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-5bbcf797b5-jxs79", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-5bbcf797b5-jxs79 (namespace mgmt-agent) on cluster ci01-j8615424-mgmt-2 has averaged 196.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 13:
  State:       Resolved
  Started:     2026-09-17T06:05:45Z
  Ended:       2026-09-17T06:15:44Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j8615424-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-b9pb4", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-b9pb4 (namespace arobit) on cluster ci01-j8615424-mgmt-2 has averaged 130.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

alert fired 6 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T05:19:56Z
  Ended:       2026-09-17T05:52:58Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j4481536-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-6nhnd", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-6nhnd (namespace arobit) on cluster ci01-j4481536-svc has averaged 193.5% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-17T05:21:54Z
  Ended:       2026-09-17T06:05:56Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j4481536-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-5bd75f6d87-6frgs", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-5bd75f6d87-6frgs (namespace clusters-service) on cluster ci01-j4481536-svc has averaged 671.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-17T05:21:54Z
  Ended:       2026-09-17T05:58:54Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j4481536-svc", component="service-memory", container="aro-hcp-backend", namespace="aro-hcp", pod="aro-hcp-backend-864cf4d9dd-5gkfh", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-backend in pod aro-hcp-backend-864cf4d9dd-5gkfh (namespace aro-hcp) on cluster ci01-j4481536-svc has averaged 380.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-17T05:22:57Z
  Ended:       2026-09-17T05:35:57Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j4481536-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-5bd75f6d87-75jjn", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-5bd75f6d87-75jjn (namespace clusters-service) on cluster ci01-j4481536-svc has averaged 233.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 5:
  State:       Resolved
  Started:     2026-09-17T05:22:57Z
  Ended:       2026-09-17T05:38:53Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j4481536-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-5bd75f6d87-w2tsd", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-5bd75f6d87-w2tsd (namespace clusters-service) on cluster ci01-j4481536-svc has averaged 251.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 6:
  State:       Resolved
  Started:     2026-09-17T05:26:55Z
  Ended:       2026-09-17T05:45:54Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j4481536-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-vgw8g", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-vgw8g (namespace arobit) on cluster ci01-j4481536-mgmt-2 has averaged 140.5% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000fbc120>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-w5shsm9ng8kc" to finish creating: GET https://rp.j5021696.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eb7aec2b-15f3-48ae-bf39-e0b303d7f6f8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eb7aec2b-15f3-48ae-bf39-e0b303d7f6f8",
      "name": "eb7aec2b-15f3-48ae-bf39-e0b303d7f6f8",
      "status": "Failed",
      "startTime": "2026-09-17T04:52:38.625826904Z",
      "endTime": "2026-09-17T05:21:45.494759028Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 2m17s ago; version 4.22.14 is Partial (want Completed), started 5m22s ago; version 4.22.14 is Partial (want Completed), started 12m16s ago; version 4.22.14 is Partial (want Completed), started 15m20s ago; version 4.22.14 is Partial (want Completed), started 17m23s ago; version 4.22.14 is Partial (want Completed), started 17m58s ago; hosted cluster degraded: UnavailableReplicas: openshift-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000fbc120>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-w5shsm9ng8kc" to finish creating: GET https://rp.j5021696.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eb7aec2b-15f3-48ae-bf39-e0b303d7f6f8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eb7aec2b-15f3-48ae-bf39-e0b303d7f6f8",
      "name": "eb7aec2b-15f3-48ae-bf39-e0b303d7f6f8",
      "status": "Failed",
      "startTime": "2026-09-17T04:52:38.625826904Z",
      "endTime": "2026-09-17T05:21:45.494759028Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 2m17s ago; version 4.22.14 is Partial (want Completed), started 5m22s ago; version 4.22.14 is Partial (want Completed), started 12m16s ago; version 4.22.14 is Partial (want Completed), started 15m20s ago; version 4.22.14 is Partial (want Completed), started 17m23s ago; version 4.22.14 is Partial (want Completed), started 17m58s ago; hosted cluster degraded: UnavailableReplicas: openshift-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc0010be220>: 
    failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-r2cp7qnm68cd" to finish creating: GET https://rp.j5021696.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1574465a-b85d-4740-b4e8-67565d3ad14a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1574465a-b85d-4740-b4e8-67565d3ad14a",
      "name": "1574465a-b85d-4740-b4e8-67565d3ad14a",
      "status": "Failed",
      "startTime": "2026-09-17T04:51:44.941548523Z",
      "endTime": "2026-09-17T05:20:45.403813623Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 18m27s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc0010be220>: 
    failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-r2cp7qnm68cd" to finish creating: GET https://rp.j5021696.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1574465a-b85d-4740-b4e8-67565d3ad14a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1574465a-b85d-4740-b4e8-67565d3ad14a",
      "name": "1574465a-b85d-4740-b4e8-67565d3ad14a",
      "status": "Failed",
      "startTime": "2026-09-17T04:51:44.941548523Z",
      "endTime": "2026-09-17T05:20:45.403813623Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 18m27s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 3 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T05:25:34Z
  Ended:       2026-09-17T05:33:33Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5021696-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-5765fc97bc-hmsgv", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-5765fc97bc-hmsgv (namespace clusters-service) on cluster ci01-j5021696-svc has averaged 229.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-17T05:25:34Z
  Ended:       2026-09-17T05:34:32Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5021696-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-5765fc97bc-7rqzw", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-5765fc97bc-7rqzw (namespace clusters-service) on cluster ci01-j5021696-svc has averaged 785% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-17T05:28:32Z
  Ended:       2026-09-17T05:34:32Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5021696-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-dbnnd", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-dbnnd (namespace arobit) on cluster ci01-j5021696-svc has averaged 126.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-22-lkx9zk2x2sg4/cluster-candidate-4-22-v22kq5 should provision
Unexpected error:
    <*fmt.wrapError | 0xc001068640>: 
    failed to create HCP cluster cluster-candidate-4-22-v22kq5: failed waiting for cluster="cluster-candidate-4-22-v22kq5" in resourcegroup="rg-candidate-4-22-lkx9zk2x2sg4" to finish creating: GET https://rp.j4260864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/741d8f04-ed8d-4645-94fa-56f03c47eb8f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/741d8f04-ed8d-4645-94fa-56f03c47eb8f",
      "name": "741d8f04-ed8d-4645-94fa-56f03c47eb8f",
      "status": "Failed",
      "startTime": "2026-09-17T04:52:59.588772018Z",
      "endTime": "2026-09-17T05:21:59.743565867Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 23m13s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-22-lkx9zk2x2sg4/cluster-candidate-4-22-v22kq5 should provision
Unexpected error:
    <*fmt.wrapError | 0xc001068640>: 
    failed to create HCP cluster cluster-candidate-4-22-v22kq5: failed waiting for cluster="cluster-candidate-4-22-v22kq5" in resourcegroup="rg-candidate-4-22-lkx9zk2x2sg4" to finish creating: GET https://rp.j4260864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/741d8f04-ed8d-4645-94fa-56f03c47eb8f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/741d8f04-ed8d-4645-94fa-56f03c47eb8f",
      "name": "741d8f04-ed8d-4645-94fa-56f03c47eb8f",
      "status": "Failed",
      "startTime": "2026-09-17T04:52:59.588772018Z",
      "endTime": "2026-09-17T05:21:59.743565867Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 23m13s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:123]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000ca6240>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-drg9bd4zkzmf" to finish creating: GET https://rp.j4260864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/53c7201f-3e66-49f4-8e9d-3ea9fed2dbcb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/53c7201f-3e66-49f4-8e9d-3ea9fed2dbcb",
      "name": "53c7201f-3e66-49f4-8e9d-3ea9fed2dbcb",
      "status": "Failed",
      "startTime": "2026-09-17T04:51:59.38221738Z",
      "endTime": "2026-09-17T05:20:59.747749137Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 21m58s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:123]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000ca6240>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-drg9bd4zkzmf" to finish creating: GET https://rp.j4260864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/53c7201f-3e66-49f4-8e9d-3ea9fed2dbcb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/53c7201f-3e66-49f4-8e9d-3ea9fed2dbcb",
      "name": "53c7201f-3e66-49f4-8e9d-3ea9fed2dbcb",
      "status": "Failed",
      "startTime": "2026-09-17T04:51:59.38221738Z",
      "endTime": "2026-09-17T05:20:59.747749137Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 21m58s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T05:23:48Z
  Ended:       2026-09-17T05:29:49Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j4260864-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-2vs49", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-2vs49 (namespace arobit) on cluster ci01-j4260864-mgmt-2 has averaged 172.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-17T05:26:47Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j4260864-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-b498k", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-b498k (namespace arobit) on cluster ci01-j4260864-mgmt-1 has averaged 202.5% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:407]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc0014180c0>: 
    VerifyCiliumConnectivityChecks failed: connectivity check failed, not all pods in cilium-connectivity-check-phbtk2 namespace are ready: failed to list pods in cilium-connectivity-check-phbtk2 namespace: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:407]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc0014180c0>: 
    VerifyCiliumConnectivityChecks failed: connectivity check failed, not all pods in cilium-connectivity-check-phbtk2 namespace are ready: failed to list pods in cilium-connectivity-check-phbtk2 namespace: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred

alert fired 8 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T05:21:49Z
  Ended:       2026-09-17T05:28:48Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3898368-mgmt-2", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-6456884798-89rmr", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-6456884798-89rmr (namespace mgmt-agent) on cluster ci01-j3898368-mgmt-2 has averaged 142.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-17T05:21:49Z
  Ended:       2026-09-17T05:28:48Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3898368-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-5957d786fb-xjs47", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-5957d786fb-xjs47 (namespace clusters-service) on cluster ci01-j3898368-svc has averaged 978.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-17T05:21:49Z
  Ended:       2026-09-17T05:36:50Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3898368-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-r86jf", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-r86jf (namespace arobit) on cluster ci01-j3898368-mgmt-1 has averaged 194% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-17T05:21:49Z
  Ended:       2026-09-17T05:29:50Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3898368-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-nbqjq", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-nbqjq (namespace arobit) on cluster ci01-j3898368-svc has averaged 289.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 5:
  State:       Resolved
  Started:     2026-09-17T05:21:49Z
  Ended:       2026-09-17T05:36:50Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3898368-mgmt-1", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-6456884798-h72vz", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-6456884798-h72vz (namespace mgmt-agent) on cluster ci01-j3898368-mgmt-1 has averaged 193.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 6:
  State:       Resolved
  Started:     2026-09-17T05:21:49Z
  Ended:       2026-09-17T05:44:49Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3898368-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-5957d786fb-cz742", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-5957d786fb-cz742 (namespace clusters-service) on cluster ci01-j3898368-svc has averaged 254.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 7:
  State:       Resolved
  Started:     2026-09-17T05:21:49Z
  Ended:       2026-09-17T05:28:48Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3898368-svc", component="service-memory", container="aro-hcp-backend", namespace="aro-hcp", pod="aro-hcp-backend-d6dd4cc9f-qs4kx", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-backend in pod aro-hcp-backend-d6dd4cc9f-qs4kx (namespace aro-hcp) on cluster ci01-j3898368-svc has averaged 491.8% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 8:
  State:       Resolved
  Started:     2026-09-17T05:21:49Z
  Ended:       2026-09-17T05:28:48Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3898368-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-dvnvp", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-dvnvp (namespace arobit) on cluster ci01-j3898368-mgmt-2 has averaged 141.8% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-5-0-mvfnshj95l4p/cluster-candidate-5-0-9ptr9v should provision
Unexpected error:
    <*fmt.wrapError | 0xc0010149a0>: 
    failed to create HCP cluster cluster-candidate-5-0-9ptr9v: failed waiting for cluster="cluster-candidate-5-0-9ptr9v" in resourcegroup="rg-candidate-5-0-mvfnshj95l4p" to finish creating: GET https://rp.j0741376.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2f558a3f-8e1b-4977-a6b0-a4b70992d9bb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2f558a3f-8e1b-4977-a6b0-a4b70992d9bb",
      "name": "2f558a3f-8e1b-4977-a6b0-a4b70992d9bb",
      "status": "Failed",
      "startTime": "2026-09-17T03:35:00.650591919Z",
      "endTime": "2026-09-17T04:04:02.019280064Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-rc.2 is Partial (want Completed), started 21m52s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-5-0-mvfnshj95l4p/cluster-candidate-5-0-9ptr9v should provision
Unexpected error:
    <*fmt.wrapError | 0xc0010149a0>: 
    failed to create HCP cluster cluster-candidate-5-0-9ptr9v: failed waiting for cluster="cluster-candidate-5-0-9ptr9v" in resourcegroup="rg-candidate-5-0-mvfnshj95l4p" to finish creating: GET https://rp.j0741376.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2f558a3f-8e1b-4977-a6b0-a4b70992d9bb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2f558a3f-8e1b-4977-a6b0-a4b70992d9bb",
      "name": "2f558a3f-8e1b-4977-a6b0-a4b70992d9bb",
      "status": "Failed",
      "startTime": "2026-09-17T03:35:00.650591919Z",
      "endTime": "2026-09-17T04:04:02.019280064Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-rc.2 is Partial (want Completed), started 21m52s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1
Unexpected error:
    <*fmt.wrapError | 0xc000b8f5e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-n2zmdkblldlw" to finish creating: GET https://rp.j0741376.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cfa413af-9795-4d14-a521-6638502432ab
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cfa413af-9795-4d14-a521-6638502432ab",
      "name": "cfa413af-9795-4d14-a521-6638502432ab",
      "status": "Failed",
      "startTime": "2026-09-17T04:05:39.31416638Z",
      "endTime": "2026-09-17T04:06:12.119697969Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1
Unexpected error:
    <*fmt.wrapError | 0xc000b8f5e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-n2zmdkblldlw" to finish creating: GET https://rp.j0741376.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cfa413af-9795-4d14-a521-6638502432ab
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cfa413af-9795-4d14-a521-6638502432ab",
      "name": "cfa413af-9795-4d14-a521-6638502432ab",
      "status": "Failed",
      "startTime": "2026-09-17T04:05:39.31416638Z",
      "endTime": "2026-09-17T04:06:12.119697969Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 5 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T04:08:23Z
  Ended:       2026-09-17T04:14:28Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j0741376-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-5b859bc4bf-jjt2z", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-5b859bc4bf-jjt2z (namespace clusters-service) on cluster ci01-j0741376-svc has averaged 832% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-17T04:08:23Z
  Ended:       2026-09-17T04:16:23Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j0741376-mgmt-2", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-5d4d9cb45c-7rb4g", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-5d4d9cb45c-7rb4g (namespace mgmt-agent) on cluster ci01-j0741376-mgmt-2 has averaged 168.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-17T04:08:23Z
  Ended:       2026-09-17T04:19:25Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j0741376-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-c9l25", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-c9l25 (namespace arobit) on cluster ci01-j0741376-mgmt-2 has averaged 140.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-17T04:08:23Z
  Ended:       2026-09-17T04:22:26Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j0741376-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-84nsd", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-84nsd (namespace arobit) on cluster ci01-j0741376-mgmt-2 has averaged 174.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 5:
  State:       Resolved
  Started:     2026-09-17T04:14:27Z
  Ended:       2026-09-17T04:21:23Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j0741376-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-v9grg", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-v9grg (namespace arobit) on cluster ci01-j0741376-mgmt-2 has averaged 123.8% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

alert fired 8 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T04:31:05Z
  Ended:       2026-09-17T04:49:04Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j4346240-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-fjvpp", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-fjvpp (namespace arobit) on cluster ci01-j4346240-svc has averaged 164.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-17T04:31:05Z
  Ended:       2026-09-17T04:49:04Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j4346240-svc", component="service-memory", container="aro-hcp-backend", namespace="aro-hcp", pod="aro-hcp-backend-57c8fd9876-xlstr", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-backend in pod aro-hcp-backend-57c8fd9876-xlstr (namespace aro-hcp) on cluster ci01-j4346240-svc has averaged 388.8% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-17T04:32:03Z
  Ended:       2026-09-17T04:49:04Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j4346240-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-lv9vn", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-lv9vn (namespace arobit) on cluster ci01-j4346240-mgmt-1 has averaged 143.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-17T04:32:03Z
  Ended:       2026-09-17T04:41:03Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j4346240-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-5b7c47c974-6qcss", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-5b7c47c974-6qcss (namespace clusters-service) on cluster ci01-j4346240-svc has averaged 896.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 5:
  State:       Resolved
  Started:     2026-09-17T04:34:06Z
  Ended:       2026-09-17T04:41:02Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j4346240-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-klvzm", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-klvzm (namespace arobit) on cluster ci01-j4346240-svc has averaged 138.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 6:
  State:       Resolved
  Started:     2026-09-17T04:34:07Z
  Ended:       2026-09-17T04:48:05Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j4346240-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-5b7c47c974-8jpp2", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-5b7c47c974-8jpp2 (namespace clusters-service) on cluster ci01-j4346240-svc has averaged 221.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 7:
  State:       Resolved
  Started:     2026-09-17T04:34:07Z
  Ended:       2026-09-17T04:52:07Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j4346240-mgmt-1", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-6bf86fbcd6-47cmn", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-6bf86fbcd6-47cmn (namespace mgmt-agent) on cluster ci01-j4346240-mgmt-1 has averaged 176.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 8:
  State:       Resolved
  Started:     2026-09-17T04:43:05Z
  Ended:       2026-09-17T04:50:06Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j4346240-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-gvr9v", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-gvr9v (namespace arobit) on cluster ci01-j4346240-mgmt-2 has averaged 129% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-22-4jpl9dtwpfzm/cluster-candidate-4-22-shq9rw should provision
Unexpected error:
    <*fmt.wrapError | 0xc0007f6520>: 
    failed to create HCP cluster cluster-candidate-4-22-shq9rw: failed waiting for cluster="cluster-candidate-4-22-shq9rw" in resourcegroup="rg-candidate-4-22-4jpl9dtwpfzm" to finish creating: GET https://rp.j5272576.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8f2fdee1-3fa0-4cad-b40d-396308bf0040
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8f2fdee1-3fa0-4cad-b40d-396308bf0040",
      "name": "8f2fdee1-3fa0-4cad-b40d-396308bf0040",
      "status": "Failed",
      "startTime": "2026-09-17T03:44:24.882818903Z",
      "endTime": "2026-09-17T04:13:25.759327578Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 22m24s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-22-4jpl9dtwpfzm/cluster-candidate-4-22-shq9rw should provision
Unexpected error:
    <*fmt.wrapError | 0xc0007f6520>: 
    failed to create HCP cluster cluster-candidate-4-22-shq9rw: failed waiting for cluster="cluster-candidate-4-22-shq9rw" in resourcegroup="rg-candidate-4-22-4jpl9dtwpfzm" to finish creating: GET https://rp.j5272576.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8f2fdee1-3fa0-4cad-b40d-396308bf0040
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8f2fdee1-3fa0-4cad-b40d-396308bf0040",
      "name": "8f2fdee1-3fa0-4cad-b40d-396308bf0040",
      "status": "Failed",
      "startTime": "2026-09-17T03:44:24.882818903Z",
      "endTime": "2026-09-17T04:13:25.759327578Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 22m24s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 15 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T04:13:05Z
  Ended:       2026-09-17T04:32:04Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5272576-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-lbvfd", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-lbvfd (namespace arobit) on cluster ci01-j5272576-mgmt-1 has averaged 158% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-17T04:13:05Z
  Ended:       2026-09-17T04:34:04Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5272576-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-djr26", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-djr26 (namespace arobit) on cluster ci01-j5272576-svc has averaged 183.8% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-17T04:14:06Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5272576-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-7f68fcd5b9-456ls", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-7f68fcd5b9-456ls (namespace clusters-service) on cluster ci01-j5272576-svc has averaged 831.8% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-17T04:14:06Z
  Ended:       2026-09-17T04:44:06Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5272576-svc", component="service-memory", container="aro-hcp-backend", namespace="aro-hcp", pod="aro-hcp-backend-b6fc9ffcf-w7hl2", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-backend in pod aro-hcp-backend-b6fc9ffcf-w7hl2 (namespace aro-hcp) on cluster ci01-j5272576-svc has averaged 409% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 5:
  State:       Resolved
  Started:     2026-09-17T04:14:06Z
  Ended:       2026-09-17T04:41:05Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5272576-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-7f68fcd5b9-7mv4x", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-7f68fcd5b9-7mv4x (namespace clusters-service) on cluster ci01-j5272576-svc has averaged 271.8% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 6:
  State:       Resolved
  Started:     2026-09-17T04:14:06Z
  Ended:       2026-09-17T04:40:04Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5272576-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-n9r9l", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-n9r9l (namespace arobit) on cluster ci01-j5272576-mgmt-2 has averaged 162.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 7:
  State:       Resolved
  Started:     2026-09-17T04:14:06Z
  Ended:       2026-09-17T04:37:04Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5272576-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-55wzp", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-55wzp (namespace arobit) on cluster ci01-j5272576-mgmt-2 has averaged 196.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 8:
  State:       Resolved
  Started:     2026-09-17T04:14:06Z
  Ended:       2026-09-17T04:38:06Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5272576-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-7f68fcd5b9-d8nlf", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-7f68fcd5b9-d8nlf (namespace clusters-service) on cluster ci01-j5272576-svc has averaged 233.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 9:
  State:       Resolved
  Started:     2026-09-17T04:14:06Z
  Ended:       2026-09-17T04:28:07Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5272576-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-q6c4s", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-q6c4s (namespace arobit) on cluster ci01-j5272576-mgmt-1 has averaged 137.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 10:
  State:       Resolved
  Started:     2026-09-17T04:15:04Z
  Ended:       2026-09-17T04:32:03Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5272576-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-lj9x9", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-lj9x9 (namespace arobit) on cluster ci01-j5272576-mgmt-1 has averaged 145.5% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 11:
  State:       Resolved
  Started:     2026-09-17T04:15:04Z
  Ended:       2026-09-17T04:29:06Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5272576-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-8k7gj", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-8k7gj (namespace arobit) on cluster ci01-j5272576-svc has averaged 138.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 12:
  State:       Resolved
  Started:     2026-09-17T04:15:04Z
  Ended:       2026-09-17T04:38:06Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5272576-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-gj7rn", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-gj7rn (namespace arobit) on cluster ci01-j5272576-mgmt-2 has averaged 161.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 13:
  State:       Resolved
  Started:     2026-09-17T04:15:04Z
  Ended:       2026-09-17T04:33:05Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5272576-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-r2j2n", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-r2j2n (namespace arobit) on cluster ci01-j5272576-mgmt-2 has averaged 141.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 14:
  State:       Resolved
  Started:     2026-09-17T04:16:05Z
  Ended:       2026-09-17T04:30:06Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5272576-mgmt-1", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-5b8f86b79b-jv7v4", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-5b8f86b79b-jv7v4 (namespace mgmt-agent) on cluster ci01-j5272576-mgmt-1 has averaged 161.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 15:
  State:       Resolved
  Started:     2026-09-17T04:16:05Z
  Ended:       2026-09-17T04:33:06Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5272576-mgmt-2", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-5b8f86b79b-7qs69", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-5b8f86b79b-7qs69 (namespace mgmt-agent) on cluster ci01-j5272576-mgmt-2 has averaged 187% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:407]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc00168f0c8>: 
    VerifyCiliumConnectivityChecks failed: connectivity check failed, not all pods in cilium-connectivity-check-txdwxr namespace are ready: failed to list pods in cilium-connectivity-check-txdwxr namespace: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:407]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc00168f0c8>: 
    VerifyCiliumConnectivityChecks failed: connectivity check failed, not all pods in cilium-connectivity-check-txdwxr namespace are ready: failed to list pods in cilium-connectivity-check-txdwxr namespace: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred

alert fired 5 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T02:45:46Z
  Ended:       2026-09-17T03:32:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9076992-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssg777qqnq4cjon6gmqfatr43murnr3-v8z9h0b9l2l6b7a", pod="router-6777b8c95f-lklkr", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssg777qqnq4cjon6gmqfatr43murnr3-v8z9h0b9l2l6b7a/router-6777b8c95f-lklkr has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-17T02:45:46Z
  Ended:       2026-09-17T03:25:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9076992-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssg777qqnq4cjon6gmqfatr43murnr3-v8z9h0b9l2l6b7a", pod="router-6777b8c95f-kwgxc", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssg777qqnq4cjon6gmqfatr43murnr3-v8z9h0b9l2l6b7a/router-6777b8c95f-kwgxc has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-17T02:46:47Z
  Ended:       2026-09-17T03:04:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9076992-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssg7mbkri4692gdg08dld9mm65i0ctd-w7q9x5r4e9d1u2u", pod="router-d6d8587c6-stnlp", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssg7mbkri4692gdg08dld9mm65i0ctd-w7q9x5r4e9d1u2u/router-d6d8587c6-stnlp has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-17T02:48:43Z
  Ended:       2026-09-17T03:04:43Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9076992-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssg85jl4ed13fvm3rmk51oupprdb08b-j1m7o6h4e6k8x0m", pod="router-7cd4947c5d-fcvgq", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssg85jl4ed13fvm3rmk51oupprdb08b-j1m7o6h4e6k8x0m/router-7cd4947c5d-fcvgq has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-09-17T02:48:43Z
  Ended:       2026-09-17T03:04:43Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9076992-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssg8gg615bvgkkgo2s0arenrfb5gltm-agg-cluster", pod="router-866b68f94b-wr5tc", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssg8gg615bvgkkgo2s0arenrfb5gltm-agg-cluster/router-866b68f94b-wr5tc has been in a non-ready state for longer than 5 minutes.

alert fired 11 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T03:10:49Z
  Ended:       2026-09-17T03:28:51Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9076992-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-vv2pt", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-vv2pt (namespace arobit) on cluster ci01-j9076992-mgmt-2 has averaged 161.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-17T03:10:49Z
  Ended:       2026-09-17T03:25:51Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9076992-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-qm88r", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-qm88r (namespace arobit) on cluster ci01-j9076992-mgmt-2 has averaged 178.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-17T03:10:49Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9076992-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-56459cc9f-dddn4", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-56459cc9f-dddn4 (namespace clusters-service) on cluster ci01-j9076992-svc has averaged 831.1% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-17T03:10:49Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9076992-svc", component="service-memory", container="aro-hcp-backend", namespace="aro-hcp", pod="aro-hcp-backend-75748f55df-tzjvh", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-backend in pod aro-hcp-backend-75748f55df-tzjvh (namespace aro-hcp) on cluster ci01-j9076992-svc has averaged 422.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 5:
  State:       Resolved
  Started:     2026-09-17T03:10:49Z
  Ended:       2026-09-17T03:22:52Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9076992-mgmt-1", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-76df9b9f8-lm9mq", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-76df9b9f8-lm9mq (namespace mgmt-agent) on cluster ci01-j9076992-mgmt-1 has averaged 168.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 6:
  State:       Resolved
  Started:     2026-09-17T03:10:49Z
  Ended:       2026-09-17T03:33:49Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9076992-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-56459cc9f-vsgg5", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-56459cc9f-vsgg5 (namespace clusters-service) on cluster ci01-j9076992-svc has averaged 254.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 7:
  State:       Resolved
  Started:     2026-09-17T03:10:49Z
  Ended:       2026-09-17T03:20:49Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9076992-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-2dtcx", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-2dtcx (namespace arobit) on cluster ci01-j9076992-mgmt-1 has averaged 147.5% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 8:
  State:       Resolved
  Started:     2026-09-17T03:10:49Z
  Ended:       2026-09-17T03:31:48Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9076992-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-rrbhs", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-rrbhs (namespace arobit) on cluster ci01-j9076992-svc has averaged 235.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 9:
  State:       Resolved
  Started:     2026-09-17T03:10:49Z
  Ended:       2026-09-17T03:27:52Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9076992-mgmt-2", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-76df9b9f8-qhgv5", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-76df9b9f8-qhgv5 (namespace mgmt-agent) on cluster ci01-j9076992-mgmt-2 has averaged 229.5% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 10:
  State:       Resolved
  Started:     2026-09-17T03:10:49Z
  Ended:       2026-09-17T03:31:48Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9076992-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-mps8r", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-mps8r (namespace arobit) on cluster ci01-j9076992-mgmt-2 has averaged 267.1% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 11:
  State:       Resolved
  Started:     2026-09-17T03:10:49Z
  Ended:       2026-09-17T03:24:51Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9076992-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-6znl8", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-6znl8 (namespace arobit) on cluster ci01-j9076992-mgmt-1 has averaged 155.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T03:10:45Z
  Ended:       2026-09-17T03:32:50Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9076992-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="router", endpoint="http", environment="ci01", instance="10.128.65.31:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9076992/providers/microsoft.monitor/accounts/hcps-j9076992", namespace="ocm-arohcpci01-2ssg777qqnq4cjon6gmqfatr43murnr3-v8z9h0b9l2l6b7a", pod="arohcp-monitor-kube-state-metrics-574f657d96-6q7r5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ssg777qqnq4cjon6gmqfatr43murnr3-v8z9h0b9l2l6b7a/router has not matched the expected number of replicas for longer than 30 minutes.

alert fired 5 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T03:10:30Z
  Ended:       2026-09-17T03:22:31Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9076992-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.65.31:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9076992/providers/microsoft.monitor/accounts/services-j9076992", namespace="klusterlet-2ssg777qqnq4cjon6gmqfatr43murnr3", pod="arohcp-monitor-kube-state-metrics-574f657d96-6q7r5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2ssg777qqnq4cjon6gmqfatr43murnr3/config-policy-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-17T03:10:30Z
  Ended:       2026-09-17T03:22:31Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9076992-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.31:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9076992/providers/microsoft.monitor/accounts/services-j9076992", namespace="klusterlet-2ssg777qqnq4cjon6gmqfatr43murnr3", pod="arohcp-monitor-kube-state-metrics-574f657d96-6q7r5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2ssg777qqnq4cjon6gmqfatr43murnr3/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-17T03:10:30Z
  Ended:       2026-09-17T03:22:31Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9076992-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="governance-policy-framework", endpoint="http", environment="ci01", instance="10.128.65.31:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9076992/providers/microsoft.monitor/accounts/services-j9076992", namespace="klusterlet-2ssg777qqnq4cjon6gmqfatr43murnr3", pod="arohcp-monitor-kube-state-metrics-574f657d96-6q7r5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2ssg777qqnq4cjon6gmqfatr43murnr3/governance-policy-framework has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-17T03:11:28Z
  Ended:       2026-09-17T03:21:28Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9076992-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="governance-policy-framework", endpoint="http", environment="ci01", instance="10.128.65.31:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9076992/providers/microsoft.monitor/accounts/services-j9076992", namespace="klusterlet-2ssg777qqnq4cjon6gmqfatr43murnr3", pod="arohcp-monitor-kube-state-metrics-574f657d96-6q7r5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2ssg777qqnq4cjon6gmqfatr43murnr3/governance-policy-framework has not matched the expected number of replicas for longer than 30 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-09-17T03:11:28Z
  Ended:       2026-09-17T03:21:28Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9076992-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.31:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9076992/providers/microsoft.monitor/accounts/services-j9076992", namespace="klusterlet-2ssg777qqnq4cjon6gmqfatr43murnr3", pod="arohcp-monitor-kube-state-metrics-574f657d96-6q7r5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2ssg777qqnq4cjon6gmqfatr43murnr3/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.

alert fired 6 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T03:36:01Z
  Ended:       2026-09-17T03:45:59Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j0518016-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-5b88c9f94f-r8b5t", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-5b88c9f94f-r8b5t (namespace clusters-service) on cluster ci01-j0518016-svc has averaged 746.5% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-17T03:36:01Z
  Ended:       2026-09-17T03:45:02Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j0518016-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-5b88c9f94f-vh26x", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-5b88c9f94f-vh26x (namespace clusters-service) on cluster ci01-j0518016-svc has averaged 209% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-17T03:36:01Z
  Ended:       2026-09-17T03:43:02Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j0518016-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-krvq7", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-krvq7 (namespace arobit) on cluster ci01-j0518016-mgmt-2 has averaged 218% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-17T03:36:01Z
  Ended:       2026-09-17T03:43:02Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j0518016-mgmt-2", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-5485957bf9-d5zv5", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-5485957bf9-d5zv5 (namespace mgmt-agent) on cluster ci01-j0518016-mgmt-2 has averaged 171.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 5:
  State:       Resolved
  Started:     2026-09-17T03:36:02Z
  Ended:       2026-09-17T03:45:59Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j0518016-mgmt-1", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-5485957bf9-sl6nr", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-5485957bf9-sl6nr (namespace mgmt-agent) on cluster ci01-j0518016-mgmt-1 has averaged 160.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 6:
  State:       Resolved
  Started:     2026-09-17T03:42:01Z
  Ended:       2026-09-17T03:51:04Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j0518016-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-55f47", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-55f47 (namespace arobit) on cluster ci01-j0518016-mgmt-2 has averaged 130.1% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc001340300>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-jvzc8h86q97v" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1730483c-5a34-4a74-b761-00a13efee4d4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1730483c-5a34-4a74-b761-00a13efee4d4",
      "name": "1730483c-5a34-4a74-b761-00a13efee4d4",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:28.276946332Z",
      "endTime": "2026-09-17T02:31:34.003715435Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc001340300>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-jvzc8h86q97v" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1730483c-5a34-4a74-b761-00a13efee4d4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1730483c-5a34-4a74-b761-00a13efee4d4",
      "name": "1730483c-5a34-4a74-b761-00a13efee4d4",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:28.276946332Z",
      "endTime": "2026-09-17T02:31:34.003715435Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc00118e660>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-9gxmxx6fhffd" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a242de7a-6d2e-430d-bb11-b7c9eaee8899
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a242de7a-6d2e-430d-bb11-b7c9eaee8899",
      "name": "a242de7a-6d2e-430d-bb11-b7c9eaee8899",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:28.338798634Z",
      "endTime": "2026-09-17T02:31:33.973889478Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc00118e660>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-9gxmxx6fhffd" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a242de7a-6d2e-430d-bb11-b7c9eaee8899
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a242de7a-6d2e-430d-bb11-b7c9eaee8899",
      "name": "a242de7a-6d2e-430d-bb11-b7c9eaee8899",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:28.338798634Z",
      "endTime": "2026-09-17T02:31:33.973889478Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-fwx5gk" with version 4.21.32 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000a50280>: 
    failed to create HCP cluster cluster-zstream-4-21-fwx5gk: failed waiting for cluster="cluster-zstream-4-21-fwx5gk" in resourcegroup="rg-zstream-upgrade-4-21-b5ssflnhqcdd" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b695f784-3091-465d-8026-55bd201f7208
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b695f784-3091-465d-8026-55bd201f7208",
      "name": "b695f784-3091-465d-8026-55bd201f7208",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:28.383849125Z",
      "endTime": "2026-09-17T02:31:34.01474185Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-fwx5gk" with version 4.21.32 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000a50280>: 
    failed to create HCP cluster cluster-zstream-4-21-fwx5gk: failed waiting for cluster="cluster-zstream-4-21-fwx5gk" in resourcegroup="rg-zstream-upgrade-4-21-b5ssflnhqcdd" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b695f784-3091-465d-8026-55bd201f7208
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b695f784-3091-465d-8026-55bd201f7208",
      "name": "b695f784-3091-465d-8026-55bd201f7208",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:28.383849125Z",
      "endTime": "2026-09-17T02:31:34.01474185Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0xc000d0aba0>: 
    failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-ms9kkl5qh4rm" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7a6f3465-1cd5-4662-8204-0e2dfd9ccda6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7a6f3465-1cd5-4662-8204-0e2dfd9ccda6",
      "name": "7a6f3465-1cd5-4662-8204-0e2dfd9ccda6",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:38.567378599Z",
      "endTime": "2026-09-17T02:31:43.962353158Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0xc000d0aba0>: 
    failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-ms9kkl5qh4rm" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7a6f3465-1cd5-4662-8204-0e2dfd9ccda6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7a6f3465-1cd5-4662-8204-0e2dfd9ccda6",
      "name": "7a6f3465-1cd5-4662-8204-0e2dfd9ccda6",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:38.567378599Z",
      "endTime": "2026-09-17T02:31:43.962353158Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000a10900>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-kf4qgm4z4dts" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/303929de-f9bf-4b87-903f-167135439e71
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/303929de-f9bf-4b87-903f-167135439e71",
      "name": "303929de-f9bf-4b87-903f-167135439e71",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:35.753852827Z",
      "endTime": "2026-09-17T02:31:43.996702977Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000a10900>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-kf4qgm4z4dts" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/303929de-f9bf-4b87-903f-167135439e71
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/303929de-f9bf-4b87-903f-167135439e71",
      "name": "303929de-f9bf-4b87-903f-167135439e71",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:35.753852827Z",
      "endTime": "2026-09-17T02:31:43.996702977Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-j52qlt with version 4.20.38
Unexpected error:
    <*fmt.wrapError | 0xc000d16cc0>: 
    failed to create HCP cluster np-version-upgrade-cluster-j52qlt: failed waiting for cluster="np-version-upgrade-cluster-j52qlt" in resourcegroup="rg-np-version-upgrade-j52qlt-5fp9k8q5vgqd" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9371c790-9005-48c5-ab72-5a96cc6f4ca1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9371c790-9005-48c5-ab72-5a96cc6f4ca1",
      "name": "9371c790-9005-48c5-ab72-5a96cc6f4ca1",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:44.264326971Z",
      "endTime": "2026-09-17T02:31:54.008286613Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-j52qlt with version 4.20.38
Unexpected error:
    <*fmt.wrapError | 0xc000d16cc0>: 
    failed to create HCP cluster np-version-upgrade-cluster-j52qlt: failed waiting for cluster="np-version-upgrade-cluster-j52qlt" in resourcegroup="rg-np-version-upgrade-j52qlt-5fp9k8q5vgqd" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9371c790-9005-48c5-ab72-5a96cc6f4ca1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9371c790-9005-48c5-ab72-5a96cc6f4ca1",
      "name": "9371c790-9005-48c5-ab72-5a96cc6f4ca1",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:44.264326971Z",
      "endTime": "2026-09-17T02:31:54.008286613Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-6hvxbh" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0xc0016d4240>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-6hvxbh: failed waiting for cluster="cp-ystream-upgrade-4-22-6hvxbh" in resourcegroup="rg-cp-ystream-upgrade-4-22-xh4hgks62ljw" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/531e1a1c-60be-4fce-b365-0f884cff7c52
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/531e1a1c-60be-4fce-b365-0f884cff7c52",
      "name": "531e1a1c-60be-4fce-b365-0f884cff7c52",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:46.217261407Z",
      "endTime": "2026-09-17T02:31:53.884522774Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-6hvxbh" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0xc0016d4240>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-6hvxbh: failed waiting for cluster="cp-ystream-upgrade-4-22-6hvxbh" in resourcegroup="rg-cp-ystream-upgrade-4-22-xh4hgks62ljw" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/531e1a1c-60be-4fce-b365-0f884cff7c52
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/531e1a1c-60be-4fce-b365-0f884cff7c52",
      "name": "531e1a1c-60be-4fce-b365-0f884cff7c52",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:46.217261407Z",
      "endTime": "2026-09-17T02:31:53.884522774Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0xc0014fc8a0>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-zvg8dvqqf6zz" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-zvg8dvqqf6zz, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0xc0014fc8a0>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-zvg8dvqqf6zz" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-zvg8dvqqf6zz, error: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:441]: failed to create HCP cluster np-skip-minor-h6jm2t at version 4.22.14
Unexpected error:
    <*fmt.wrapError | 0xc001542b80>: 
    failed to create HCP cluster np-skip-minor-h6jm2t: failed waiting for cluster="np-skip-minor-h6jm2t" in resourcegroup="rg-np-skip-minor-h6jm2t-w4hwhznsjjmk" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b45e983d-6f18-4b9a-9811-927f20f1037e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b45e983d-6f18-4b9a-9811-927f20f1037e",
      "name": "b45e983d-6f18-4b9a-9811-927f20f1037e",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:44.420054544Z",
      "endTime": "2026-09-17T02:31:53.95763018Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:441]: failed to create HCP cluster np-skip-minor-h6jm2t at version 4.22.14
Unexpected error:
    <*fmt.wrapError | 0xc001542b80>: 
    failed to create HCP cluster np-skip-minor-h6jm2t: failed waiting for cluster="np-skip-minor-h6jm2t" in resourcegroup="rg-np-skip-minor-h6jm2t-w4hwhznsjjmk" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b45e983d-6f18-4b9a-9811-927f20f1037e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b45e983d-6f18-4b9a-9811-927f20f1037e",
      "name": "b45e983d-6f18-4b9a-9811-927f20f1037e",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:44.420054544Z",
      "endTime": "2026-09-17T02:31:53.95763018Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-22-6ztsf49s4tcb/cluster-candidate-4-22-6xdcfg should provision
Unexpected error:
    <*fmt.wrapError | 0xc000c0c3e0>: 
    failed to create HCP cluster cluster-candidate-4-22-6xdcfg: failed waiting for cluster="cluster-candidate-4-22-6xdcfg" in resourcegroup="rg-candidate-4-22-6ztsf49s4tcb" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a77c9b3c-5b4c-4e42-aab0-8f5e7e0965c5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a77c9b3c-5b4c-4e42-aab0-8f5e7e0965c5",
      "name": "a77c9b3c-5b4c-4e42-aab0-8f5e7e0965c5",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:48.570738366Z",
      "endTime": "2026-09-17T02:31:53.987437394Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-22-6ztsf49s4tcb/cluster-candidate-4-22-6xdcfg should provision
Unexpected error:
    <*fmt.wrapError | 0xc000c0c3e0>: 
    failed to create HCP cluster cluster-candidate-4-22-6xdcfg: failed waiting for cluster="cluster-candidate-4-22-6xdcfg" in resourcegroup="rg-candidate-4-22-6ztsf49s4tcb" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a77c9b3c-5b4c-4e42-aab0-8f5e7e0965c5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a77c9b3c-5b4c-4e42-aab0-8f5e7e0965c5",
      "name": "a77c9b3c-5b4c-4e42-aab0-8f5e7e0965c5",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:48.570738366Z",
      "endTime": "2026-09-17T02:31:53.987437394Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f7e9e0>: 
    failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-225czhknm7jg" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cb86c217-7005-44ed-b917-c04114674f41
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cb86c217-7005-44ed-b917-c04114674f41",
      "name": "cb86c217-7005-44ed-b917-c04114674f41",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:28.419178407Z",
      "endTime": "2026-09-17T02:31:33.944859309Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f7e9e0>: 
    failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-225czhknm7jg" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cb86c217-7005-44ed-b917-c04114674f41
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cb86c217-7005-44ed-b917-c04114674f41",
      "name": "cb86c217-7005-44ed-b917-c04114674f41",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:28.419178407Z",
      "endTime": "2026-09-17T02:31:33.944859309Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc000e4a660>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-96b6xnkpp85l" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3bd48285-8d33-4820-9697-a776911193ac
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3bd48285-8d33-4820-9697-a776911193ac",
      "name": "3bd48285-8d33-4820-9697-a776911193ac",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:28.231273638Z",
      "endTime": "2026-09-17T02:31:34.016609783Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc000e4a660>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-96b6xnkpp85l" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3bd48285-8d33-4820-9697-a776911193ac
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3bd48285-8d33-4820-9697-a776911193ac",
      "name": "3bd48285-8d33-4820-9697-a776911193ac",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:28.231273638Z",
      "endTime": "2026-09-17T02:31:34.016609783Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:89]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc000fbc360>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-9z5s8nc9xqgx" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b214ddd4-ba76-4a5e-a613-4c36e58e84d0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b214ddd4-ba76-4a5e-a613-4c36e58e84d0",
      "name": "b214ddd4-ba76-4a5e-a613-4c36e58e84d0",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:41.948718676Z",
      "endTime": "2026-09-17T02:31:43.891046142Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:89]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc000fbc360>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-9z5s8nc9xqgx" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b214ddd4-ba76-4a5e-a613-4c36e58e84d0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b214ddd4-ba76-4a5e-a613-4c36e58e84d0",
      "name": "b214ddd4-ba76-4a5e-a613-4c36e58e84d0",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:41.948718676Z",
      "endTime": "2026-09-17T02:31:43.891046142Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:308]: failed to create HCP cluster "sre-hcp-cluster-sc" for serial console test
Unexpected error:
    <*fmt.wrapError | 0xc001048320>: 
    failed to create HCP cluster sre-hcp-cluster-sc: failed waiting for cluster="sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-s6bm4zgqdnrw" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3cbb25a4-33cb-4061-b0a5-4f75945aa4d9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3cbb25a4-33cb-4061-b0a5-4f75945aa4d9",
      "name": "3cbb25a4-33cb-4061-b0a5-4f75945aa4d9",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:38.356951393Z",
      "endTime": "2026-09-17T02:31:43.908352124Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:308]: failed to create HCP cluster "sre-hcp-cluster-sc" for serial console test
Unexpected error:
    <*fmt.wrapError | 0xc001048320>: 
    failed to create HCP cluster sre-hcp-cluster-sc: failed waiting for cluster="sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-s6bm4zgqdnrw" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3cbb25a4-33cb-4061-b0a5-4f75945aa4d9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3cbb25a4-33cb-4061-b0a5-4f75945aa4d9",
      "name": "3cbb25a4-33cb-4061-b0a5-4f75945aa4d9",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:38.356951393Z",
      "endTime": "2026-09-17T02:31:43.908352124Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc000f90a40>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-8x87dzwxvb96" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3edfae6e-fb2d-4e52-a859-1500f9def9f6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3edfae6e-fb2d-4e52-a859-1500f9def9f6",
      "name": "3edfae6e-fb2d-4e52-a859-1500f9def9f6",
      "status": "Failed",
      "startTime": "2026-09-17T02:04:32.691734276Z",
      "endTime": "2026-09-17T02:33:33.968010236Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc000f90a40>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-8x87dzwxvb96" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3edfae6e-fb2d-4e52-a859-1500f9def9f6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3edfae6e-fb2d-4e52-a859-1500f9def9f6",
      "name": "3edfae6e-fb2d-4e52-a859-1500f9def9f6",
      "status": "Failed",
      "startTime": "2026-09-17T02:04:32.691734276Z",
      "endTime": "2026-09-17T02:33:33.968010236Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:80]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000b92880>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-z57jsd9qjtzf" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/84933e1d-1d50-47a1-a019-2e5e7e28d1ba
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/84933e1d-1d50-47a1-a019-2e5e7e28d1ba",
      "name": "84933e1d-1d50-47a1-a019-2e5e7e28d1ba",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:44.303522195Z",
      "endTime": "2026-09-17T02:31:53.875776363Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:80]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000b92880>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-z57jsd9qjtzf" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/84933e1d-1d50-47a1-a019-2e5e7e28d1ba
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/84933e1d-1d50-47a1-a019-2e5e7e28d1ba",
      "name": "84933e1d-1d50-47a1-a019-2e5e7e28d1ba",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:44.303522195Z",
      "endTime": "2026-09-17T02:31:53.875776363Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0xc0012be260>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-f2s6nsbcb4km" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/96826b81-1ed0-4935-b83d-e068c4aaec7e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/96826b81-1ed0-4935-b83d-e068c4aaec7e",
      "name": "96826b81-1ed0-4935-b83d-e068c4aaec7e",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:38.422604462Z",
      "endTime": "2026-09-17T02:31:44.011868414Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0xc0012be260>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-f2s6nsbcb4km" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/96826b81-1ed0-4935-b83d-e068c4aaec7e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/96826b81-1ed0-4935-b83d-e068c4aaec7e",
      "name": "96826b81-1ed0-4935-b83d-e068c4aaec7e",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:38.422604462Z",
      "endTime": "2026-09-17T02:31:44.011868414Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000bee140>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-n2zqcqmlz6pc" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c2786d7d-1f11-4c6b-b3c4-ecb9d5711378
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c2786d7d-1f11-4c6b-b3c4-ecb9d5711378",
      "name": "c2786d7d-1f11-4c6b-b3c4-ecb9d5711378",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:27.795928391Z",
      "endTime": "2026-09-17T02:31:33.962956757Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000bee140>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-n2zqcqmlz6pc" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c2786d7d-1f11-4c6b-b3c4-ecb9d5711378
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c2786d7d-1f11-4c6b-b3c4-ecb9d5711378",
      "name": "c2786d7d-1f11-4c6b-b3c4-ecb9d5711378",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:27.795928391Z",
      "endTime": "2026-09-17T02:31:33.962956757Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc0006d8f20>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-5jtz9dz48t8w" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4a755a8a-369f-4b08-a5a5-89db9b41a341
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4a755a8a-369f-4b08-a5a5-89db9b41a341",
      "name": "4a755a8a-369f-4b08-a5a5-89db9b41a341",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:28.405905627Z",
      "endTime": "2026-09-17T02:31:34.009715425Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc0006d8f20>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-5jtz9dz48t8w" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4a755a8a-369f-4b08-a5a5-89db9b41a341
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4a755a8a-369f-4b08-a5a5-89db9b41a341",
      "name": "4a755a8a-369f-4b08-a5a5-89db9b41a341",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:28.405905627Z",
      "endTime": "2026-09-17T02:31:34.009715425Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc00110a440>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-qr9h99g92762" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c2e04524-dc8f-4310-a549-fb6dc0199f5f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c2e04524-dc8f-4310-a549-fb6dc0199f5f",
      "name": "c2e04524-dc8f-4310-a549-fb6dc0199f5f",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:28.114427313Z",
      "endTime": "2026-09-17T02:31:33.96482384Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc00110a440>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-qr9h99g92762" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c2e04524-dc8f-4310-a549-fb6dc0199f5f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c2e04524-dc8f-4310-a549-fb6dc0199f5f",
      "name": "c2e04524-dc8f-4310-a549-fb6dc0199f5f",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:28.114427313Z",
      "endTime": "2026-09-17T02:31:33.96482384Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-j9twqf with version 4.21.33
Unexpected error:
    <*fmt.wrapError | 0xc000e3a4a0>: 
    failed to create HCP cluster np-version-upgrade-cluster-j9twqf: failed waiting for cluster="np-version-upgrade-cluster-j9twqf" in resourcegroup="rg-np-version-upgrade-j9twqf-b7p8nsv9crgv" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7d04b62a-ee8e-4fce-b798-1257698c18c7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7d04b62a-ee8e-4fce-b798-1257698c18c7",
      "name": "7d04b62a-ee8e-4fce-b798-1257698c18c7",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:44.709948422Z",
      "endTime": "2026-09-17T02:31:54.011799797Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-j9twqf with version 4.21.33
Unexpected error:
    <*fmt.wrapError | 0xc000e3a4a0>: 
    failed to create HCP cluster np-version-upgrade-cluster-j9twqf: failed waiting for cluster="np-version-upgrade-cluster-j9twqf" in resourcegroup="rg-np-version-upgrade-j9twqf-b7p8nsv9crgv" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7d04b62a-ee8e-4fce-b798-1257698c18c7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7d04b62a-ee8e-4fce-b798-1257698c18c7",
      "name": "7d04b62a-ee8e-4fce-b798-1257698c18c7",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:44.709948422Z",
      "endTime": "2026-09-17T02:31:54.011799797Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-kjzx5h with version 4.21.33
Unexpected error:
    <*fmt.wrapError | 0xc000cf0980>: 
    failed to create HCP cluster np-version-upgrade-cluster-kjzx5h: failed waiting for cluster="np-version-upgrade-cluster-kjzx5h" in resourcegroup="rg-np-version-upgrade-kjzx5h-cfkg25kg5pff" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/147a24b3-2ffd-49e8-a97a-6e6fca89fafe
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/147a24b3-2ffd-49e8-a97a-6e6fca89fafe",
      "name": "147a24b3-2ffd-49e8-a97a-6e6fca89fafe",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:24.251677536Z",
      "endTime": "2026-09-17T02:31:24.046042722Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-kjzx5h with version 4.21.33
Unexpected error:
    <*fmt.wrapError | 0xc000cf0980>: 
    failed to create HCP cluster np-version-upgrade-cluster-kjzx5h: failed waiting for cluster="np-version-upgrade-cluster-kjzx5h" in resourcegroup="rg-np-version-upgrade-kjzx5h-cfkg25kg5pff" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/147a24b3-2ffd-49e8-a97a-6e6fca89fafe
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/147a24b3-2ffd-49e8-a97a-6e6fca89fafe",
      "name": "147a24b3-2ffd-49e8-a97a-6e6fca89fafe",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:24.251677536Z",
      "endTime": "2026-09-17T02:31:24.046042722Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...

fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0xc001152b40>: 
    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-4tlzr7kc2655" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/04eb485b-d159-4a7e-a6b0-ef38fce18568
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/04eb485b-d159-4a7e-a6b0-ef38fce18568",
      "name": "04eb485b-d159-4a7e-a6b0-ef38fce18568",
      "status": "Failed",
      "startTime": "2026-09-17T02:26:41.290488511Z",
      "endTime": "2026-09-17T02:55:43.925125024Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0xc001152b40>: 
    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-4tlzr7kc2655" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/04eb485b-d159-4a7e-a6b0-ef38fce18568
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/04eb485b-d159-4a7e-a6b0-ef38fce18568",
      "name": "04eb485b-d159-4a7e-a6b0-ef38fce18568",
      "status": "Failed",
      "startTime": "2026-09-17T02:26:41.290488511Z",
      "endTime": "2026-09-17T02:55:43.925125024Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:92]: failed to create HCP cluster "sre-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000cdc740>: 
    failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-9lhfrbsqkhjk" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/39c45a7a-fb4f-41cf-82c7-e4d2fe160ef8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/39c45a7a-fb4f-41cf-82c7-e4d2fe160ef8",
      "name": "39c45a7a-fb4f-41cf-82c7-e4d2fe160ef8",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:27.808727421Z",
      "endTime": "2026-09-17T02:31:33.989605445Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:92]: failed to create HCP cluster "sre-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000cdc740>: 
    failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-9lhfrbsqkhjk" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/39c45a7a-fb4f-41cf-82c7-e4d2fe160ef8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/39c45a7a-fb4f-41cf-82c7-e4d2fe160ef8",
      "name": "39c45a7a-fb4f-41cf-82c7-e4d2fe160ef8",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:27.808727421Z",
      "endTime": "2026-09-17T02:31:33.989605445Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-zmcbpxcpx6ws/cluster-candidate-4-21-cqz9fw should provision
Unexpected error:
    <*fmt.wrapError | 0xc0010085c0>: 
    failed to create HCP cluster cluster-candidate-4-21-cqz9fw: failed waiting for cluster="cluster-candidate-4-21-cqz9fw" in resourcegroup="rg-candidate-4-21-zmcbpxcpx6ws" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9e9958ce-766f-4036-b7e8-0b661be55619
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9e9958ce-766f-4036-b7e8-0b661be55619",
      "name": "9e9958ce-766f-4036-b7e8-0b661be55619",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:49.673438329Z",
      "endTime": "2026-09-17T02:31:53.992657772Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-zmcbpxcpx6ws/cluster-candidate-4-21-cqz9fw should provision
Unexpected error:
    <*fmt.wrapError | 0xc0010085c0>: 
    failed to create HCP cluster cluster-candidate-4-21-cqz9fw: failed waiting for cluster="cluster-candidate-4-21-cqz9fw" in resourcegroup="rg-candidate-4-21-zmcbpxcpx6ws" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9e9958ce-766f-4036-b7e8-0b661be55619
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9e9958ce-766f-4036-b7e8-0b661be55619",
      "name": "9e9958ce-766f-4036-b7e8-0b661be55619",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:49.673438329Z",
      "endTime": "2026-09-17T02:31:53.992657772Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:720]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc001330460>: 
    failed to create HCP cluster np-dg-minor-ppm5ds: failed waiting for cluster="np-dg-minor-ppm5ds" in resourcegroup="rg-np-dg-minor-ppm5ds-qsf8p9hp8hdm" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/53af4dc3-7144-40af-8942-346948cc38f8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/53af4dc3-7144-40af-8942-346948cc38f8",
      "name": "53af4dc3-7144-40af-8942-346948cc38f8",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:24.259334609Z",
      "endTime": "2026-09-17T02:31:24.01732328Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:720]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc001330460>: 
    failed to create HCP cluster np-dg-minor-ppm5ds: failed waiting for cluster="np-dg-minor-ppm5ds" in resourcegroup="rg-np-dg-minor-ppm5ds-qsf8p9hp8hdm" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/53af4dc3-7144-40af-8942-346948cc38f8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/53af4dc3-7144-40af-8942-346948cc38f8",
      "name": "53af4dc3-7144-40af-8942-346948cc38f8",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:24.259334609Z",
      "endTime": "2026-09-17T02:31:24.01732328Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0xc0008a7420>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-jt498f98p44p" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d24c8171-8d46-426a-8e8c-f4e850903936
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d24c8171-8d46-426a-8e8c-f4e850903936",
      "name": "d24c8171-8d46-426a-8e8c-f4e850903936",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:37.263414559Z",
      "endTime": "2026-09-17T02:31:43.998529874Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0xc0008a7420>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-jt498f98p44p" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d24c8171-8d46-426a-8e8c-f4e850903936
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d24c8171-8d46-426a-8e8c-f4e850903936",
      "name": "d24c8171-8d46-426a-8e8c-f4e850903936",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:37.263414559Z",
      "endTime": "2026-09-17T02:31:43.998529874Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-22-v7k2jv" with version 4.22.13 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc001184300>: 
    failed to create HCP cluster cluster-zstream-4-22-v7k2jv: failed waiting for cluster="cluster-zstream-4-22-v7k2jv" in resourcegroup="rg-zstream-upgrade-4-22-w2swgqxn7qgg" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3738bf13-d603-464c-9ea8-a79b06759e18
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3738bf13-d603-464c-9ea8-a79b06759e18",
      "name": "3738bf13-d603-464c-9ea8-a79b06759e18",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:28.426824456Z",
      "endTime": "2026-09-17T02:31:33.942611429Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-22-v7k2jv" with version 4.22.13 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc001184300>: 
    failed to create HCP cluster cluster-zstream-4-22-v7k2jv: failed waiting for cluster="cluster-zstream-4-22-v7k2jv" in resourcegroup="rg-zstream-upgrade-4-22-w2swgqxn7qgg" to finish creating: GET https://rp.j7881088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3738bf13-d603-464c-9ea8-a79b06759e18
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3738bf13-d603-464c-9ea8-a79b06759e18",
      "name": "3738bf13-d603-464c-9ea8-a79b06759e18",
      "status": "Failed",
      "startTime": "2026-09-17T02:02:28.426824456Z",
      "endTime": "2026-09-17T02:31:33.942611429Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 7 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T02:38:15Z
  Ended:       2026-09-17T03:00:15Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7881088-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-t9b25", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-t9b25 (namespace arobit) on cluster ci01-j7881088-svc has averaged 218.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-17T02:38:15Z
  Ended:       2026-09-17T03:01:19Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7881088-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-db48c88df-mqr6n", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-db48c88df-mqr6n (namespace clusters-service) on cluster ci01-j7881088-svc has averaged 233% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-17T02:38:15Z
  Ended:       2026-09-17T02:56:14Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7881088-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-ph5tc", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-ph5tc (namespace arobit) on cluster ci01-j7881088-mgmt-2 has averaged 159.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-17T02:38:15Z
  Ended:       2026-09-17T02:58:15Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7881088-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-6gsds", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-6gsds (namespace arobit) on cluster ci01-j7881088-mgmt-2 has averaged 188% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 5:
  State:       Fired (not resolved)
  Started:     2026-09-17T02:38:15Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7881088-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-db48c88df-nsd25", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-db48c88df-nsd25 (namespace clusters-service) on cluster ci01-j7881088-svc has averaged 766.5% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 6:
  State:       Resolved
  Started:     2026-09-17T02:38:15Z
  Ended:       2026-09-17T02:47:15Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7881088-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-sdthl", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-sdthl (namespace arobit) on cluster ci01-j7881088-mgmt-1 has averaged 234.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 7:
  State:       Resolved
  Started:     2026-09-17T02:38:15Z
  Ended:       2026-09-17T03:01:19Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7881088-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-db48c88df-mbwff", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-db48c88df-mbwff (namespace clusters-service) on cluster ci01-j7881088-svc has averaged 251.1% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-17T02:39:34Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j7881088-mgmt-1", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.14:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7881088/providers/microsoft.monitor/accounts/services-j7881088", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-574f657d96-rcwzm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-17T02:40:35Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j7881088-mgmt-1", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.14:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7881088/providers/microsoft.monitor/accounts/services-j7881088", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-574f657d96-rcwzm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-17T02:46:35Z
  Ended:       2026-09-17T02:54:36Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j7881088-mgmt-1", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.14:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7881088/providers/microsoft.monitor/accounts/services-j7881088", namespace="klusterlet-2ssfo2opt0bd9ml6dnilop3d3i40c0r7", pod="arohcp-monitor-kube-state-metrics-574f657d96-rcwzm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment klusterlet-2ssfo2opt0bd9ml6dnilop3d3i40c0r7/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-17T02:46:35Z
  Ended:       2026-09-17T02:59:37Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j7881088-mgmt-1", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.14:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7881088/providers/microsoft.monitor/accounts/services-j7881088", namespace="klusterlet-2ssfo2opt0bd9ml6dnilop3d3i40c0r7", pod="arohcp-monitor-kube-state-metrics-574f657d96-rcwzm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment klusterlet-2ssfo2opt0bd9ml6dnilop3d3i40c0r7/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-17T02:04:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7881088-mgmt-1", component="kubernetes-infrastructure", namespace="open-cluster-management-agent-addon", pod="klusterlet-addon-workmgr-6f59c7c94f-hsk8t", region="canadacentral", severity="warning"
  Description: Pod open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f59c7c94f-hsk8t has been in a non-ready state for longer than 5 minutes.

alert fired 16 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-17T02:29:37Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7881088-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.14:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7881088/providers/microsoft.monitor/accounts/services-j7881088", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-574f657d96-rcwzm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-17T02:30:34Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7881088-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.14:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7881088/providers/microsoft.monitor/accounts/services-j7881088", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-574f657d96-rcwzm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-17T02:35:35Z
  Ended:       2026-09-17T02:42:38Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7881088-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.14:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7881088/providers/microsoft.monitor/accounts/services-j7881088", namespace="klusterlet-2ssfodlh52rjsn8v0i6tqm6t35c2l0p0", pod="arohcp-monitor-kube-state-metrics-574f657d96-rcwzm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2ssfodlh52rjsn8v0i6tqm6t35c2l0p0/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-17T02:35:35Z
  Ended:       2026-09-17T02:42:38Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7881088-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.14:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7881088/providers/microsoft.monitor/accounts/services-j7881088", namespace="klusterlet-2ssfoc87t8ajfhu9n4pqjtbphn862gj1", pod="arohcp-monitor-kube-state-metrics-574f657d96-rcwzm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2ssfoc87t8ajfhu9n4pqjtbphn862gj1/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-09-17T02:35:35Z
  Ended:       2026-09-17T02:42:39Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7881088-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.14:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7881088/providers/microsoft.monitor/accounts/services-j7881088", namespace="klusterlet-2ssfoc8cioja4t99v8qf8fgteg8hp8cq", pod="arohcp-monitor-kube-state-metrics-574f657d96-rcwzm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2ssfoc8cioja4t99v8qf8fgteg8hp8cq/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-09-17T02:35:35Z
  Ended:       2026-09-17T02:42:38Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7881088-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.14:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7881088/providers/microsoft.monitor/accounts/services-j7881088", namespace="klusterlet-2ssfofif68524qhonc25k6st2irl6nvr", pod="arohcp-monitor-kube-state-metrics-574f657d96-rcwzm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2ssfofif68524qhonc25k6st2irl6nvr/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-09-17T02:36:36Z
  Ended:       2026-09-17T02:42:37Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7881088-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.14:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7881088/providers/microsoft.monitor/accounts/services-j7881088", namespace="klusterlet-2ssfodboatgvtrijvb9itq8g6nv3a0lj", pod="arohcp-monitor-kube-state-metrics-574f657d96-rcwzm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2ssfodboatgvtrijvb9itq8g6nv3a0lj/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-09-17T02:36:36Z
  Ended:       2026-09-17T02:42:37Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7881088-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.14:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7881088/providers/microsoft.monitor/accounts/services-j7881088", namespace="klusterlet-2ssfod4unnpb3fidghon0kvgohtrm0gi", pod="arohcp-monitor-kube-state-metrics-574f657d96-rcwzm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2ssfod4unnpb3fidghon0kvgohtrm0gi/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-09-17T02:36:36Z
  Ended:       2026-09-17T02:42:38Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7881088-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.14:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7881088/providers/microsoft.monitor/accounts/services-j7881088", namespace="klusterlet-2ssfod7a9s6uqchgri0b4afl9dc7n19f", pod="arohcp-monitor-kube-state-metrics-574f657d96-rcwzm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2ssfod7a9s6uqchgri0b4afl9dc7n19f/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-09-17T02:36:36Z
  Ended:       2026-09-17T02:42:38Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7881088-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.14:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7881088/providers/microsoft.monitor/accounts/services-j7881088", namespace="klusterlet-2ssfoda33s4gbilpj50f3r2392mijctf", pod="arohcp-monitor-kube-state-metrics-574f657d96-rcwzm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2ssfoda33s4gbilpj50f3r2392mijctf/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-09-17T02:36:36Z
  Ended:       2026-09-17T02:43:34Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7881088-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.14:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7881088/providers/microsoft.monitor/accounts/services-j7881088", namespace="klusterlet-2ssfodmtcgprbt25mkk7n1f31gp57v5p", pod="arohcp-monitor-kube-state-metrics-574f657d96-rcwzm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2ssfodmtcgprbt25mkk7n1f31gp57v5p/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-09-17T02:36:36Z
  Ended:       2026-09-17T02:59:37Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7881088-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.14:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7881088/providers/microsoft.monitor/accounts/services-j7881088", namespace="klusterlet-2ssfo2opt0bd9ml6dnilop3d3i40c0r7", pod="arohcp-monitor-kube-state-metrics-574f657d96-rcwzm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2ssfo2opt0bd9ml6dnilop3d3i40c0r7/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-09-17T02:36:36Z
  Ended:       2026-09-17T02:42:39Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7881088-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.14:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7881088/providers/microsoft.monitor/accounts/services-j7881088", namespace="klusterlet-2ssfofsi9cr1hrnf33j5m2j2v0anob00", pod="arohcp-monitor-kube-state-metrics-574f657d96-rcwzm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2ssfofsi9cr1hrnf33j5m2j2v0anob00/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-09-17T02:36:36Z
  Ended:       2026-09-17T02:54:35Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7881088-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.14:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7881088/providers/microsoft.monitor/accounts/services-j7881088", namespace="klusterlet-2ssfo2opt0bd9ml6dnilop3d3i40c0r7", pod="arohcp-monitor-kube-state-metrics-574f657d96-rcwzm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2ssfo2opt0bd9ml6dnilop3d3i40c0r7/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-09-17T02:36:36Z
  Ended:       2026-09-17T02:42:38Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7881088-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.14:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7881088/providers/microsoft.monitor/accounts/services-j7881088", namespace="klusterlet-2ssfof0kh0mr490ilc2tusdfihbkudsb", pod="arohcp-monitor-kube-state-metrics-574f657d96-rcwzm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2ssfof0kh0mr490ilc2tusdfihbkudsb/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-09-17T02:37:34Z
  Ended:       2026-09-17T02:43:34Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7881088-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.14:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7881088/providers/microsoft.monitor/accounts/services-j7881088", namespace="klusterlet-2ssfohaljjdbuho1cfahj1o9d3dj04u4", pod="arohcp-monitor-kube-state-metrics-574f657d96-rcwzm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2ssfohaljjdbuho1cfahj1o9d3dj04u4/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000a844a0>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-mdr5vl8z5h7q" to finish creating: GET https://rp.j6437632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1372c7a4-2df6-4180-8631-c7a6144c4c99
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1372c7a4-2df6-4180-8631-c7a6144c4c99",
      "name": "1372c7a4-2df6-4180-8631-c7a6144c4c99",
      "status": "Failed",
      "startTime": "2026-09-17T02:44:45.592258788Z",
      "endTime": "2026-09-17T03:13:52.849263315Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 21m6s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000a844a0>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-mdr5vl8z5h7q" to finish creating: GET https://rp.j6437632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1372c7a4-2df6-4180-8631-c7a6144c4c99
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1372c7a4-2df6-4180-8631-c7a6144c4c99",
      "name": "1372c7a4-2df6-4180-8631-c7a6144c4c99",
      "status": "Failed",
      "startTime": "2026-09-17T02:44:45.592258788Z",
      "endTime": "2026-09-17T03:13:52.849263315Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 21m6s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-vl7c2rjpvjq7/cluster-candidate-4-21-p7js6n should provision
Unexpected error:
    <*fmt.wrapError | 0xc000b60e20>: 
    failed to create HCP cluster cluster-candidate-4-21-p7js6n: failed waiting for cluster="cluster-candidate-4-21-p7js6n" in resourcegroup="rg-candidate-4-21-vl7c2rjpvjq7" to finish creating: GET https://rp.j6437632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/67d55c5a-1df5-45a1-9d8a-80663ee1244f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/67d55c5a-1df5-45a1-9d8a-80663ee1244f",
      "name": "67d55c5a-1df5-45a1-9d8a-80663ee1244f",
      "status": "Failed",
      "startTime": "2026-09-17T02:44:45.387823311Z",
      "endTime": "2026-09-17T03:13:52.879414342Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 19m50s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-vl7c2rjpvjq7/cluster-candidate-4-21-p7js6n should provision
Unexpected error:
    <*fmt.wrapError | 0xc000b60e20>: 
    failed to create HCP cluster cluster-candidate-4-21-p7js6n: failed waiting for cluster="cluster-candidate-4-21-p7js6n" in resourcegroup="rg-candidate-4-21-vl7c2rjpvjq7" to finish creating: GET https://rp.j6437632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/67d55c5a-1df5-45a1-9d8a-80663ee1244f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/67d55c5a-1df5-45a1-9d8a-80663ee1244f",
      "name": "67d55c5a-1df5-45a1-9d8a-80663ee1244f",
      "status": "Failed",
      "startTime": "2026-09-17T02:44:45.387823311Z",
      "endTime": "2026-09-17T03:13:52.879414342Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 19m50s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:80]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000c60b00>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-qfm9q98mfcpr" to finish creating: GET https://rp.j6437632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3dcca7a1-32ce-4f14-918d-eb4b07fb190f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3dcca7a1-32ce-4f14-918d-eb4b07fb190f",
      "name": "3dcca7a1-32ce-4f14-918d-eb4b07fb190f",
      "status": "Failed",
      "startTime": "2026-09-17T02:44:19.790541307Z",
      "endTime": "2026-09-17T03:13:21.798847533Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 18m37s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:80]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000c60b00>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-qfm9q98mfcpr" to finish creating: GET https://rp.j6437632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3dcca7a1-32ce-4f14-918d-eb4b07fb190f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3dcca7a1-32ce-4f14-918d-eb4b07fb190f",
      "name": "3dcca7a1-32ce-4f14-918d-eb4b07fb190f",
      "status": "Failed",
      "startTime": "2026-09-17T02:44:19.790541307Z",
      "endTime": "2026-09-17T03:13:21.798847533Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 18m37s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T02:59:00Z
  Ended:       2026-09-17T03:27:06Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6437632-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssgc7v2iqv4rfbckr31morckojl5215-private-ingress", pod="router-667f75c547-v8wkq", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssgc7v2iqv4rfbckr31morckojl5215-private-ingress/router-667f75c547-v8wkq has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-17T03:01:04Z
  Ended:       2026-09-17T03:24:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6437632-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssgc7n9j9qjhfviqdpt5g6ua6btqmcn-h8g3i4v4t3m5a2s", pod="router-76b66bfcb5-vt5tw", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssgc7n9j9qjhfviqdpt5g6ua6btqmcn-h8g3i4v4t3m5a2s/router-76b66bfcb5-vt5tw has been in a non-ready state for longer than 5 minutes.

alert fired 15 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-17T03:16:00Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6437632-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-797c67bcdc-pfftn", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-797c67bcdc-pfftn (namespace clusters-service) on cluster ci01-j6437632-svc has averaged 242.1% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-17T03:16:00Z
  Ended:       2026-09-17T03:36:58Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6437632-mgmt-1", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-66f56f4b8b-mmnm8", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-66f56f4b8b-mmnm8 (namespace mgmt-agent) on cluster ci01-j6437632-mgmt-1 has averaged 172.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-17T03:16:00Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6437632-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-797c67bcdc-m7rk4", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-797c67bcdc-m7rk4 (namespace clusters-service) on cluster ci01-j6437632-svc has averaged 751.8% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-17T03:16:00Z
  Ended:       2026-09-17T03:43:58Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6437632-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-gnk8z", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-gnk8z (namespace arobit) on cluster ci01-j6437632-svc has averaged 186.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 5:
  State:       Fired (not resolved)
  Started:     2026-09-17T03:16:00Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6437632-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-797c67bcdc-n48gs", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-797c67bcdc-n48gs (namespace clusters-service) on cluster ci01-j6437632-svc has averaged 286.5% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 6:
  State:       Resolved
  Started:     2026-09-17T03:16:00Z
  Ended:       2026-09-17T03:32:57Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6437632-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-5snrs", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-5snrs (namespace arobit) on cluster ci01-j6437632-mgmt-2 has averaged 140.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 7:
  State:       Fired (not resolved)
  Started:     2026-09-17T03:16:00Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6437632-svc", component="service-memory", container="aro-hcp-backend", namespace="aro-hcp", pod="aro-hcp-backend-5f5946cd66-9mq4p", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-backend in pod aro-hcp-backend-5f5946cd66-9mq4p (namespace aro-hcp) on cluster ci01-j6437632-svc has averaged 424.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 8:
  State:       Resolved
  Started:     2026-09-17T03:16:00Z
  Ended:       2026-09-17T03:32:57Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6437632-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-55c58", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-55c58 (namespace arobit) on cluster ci01-j6437632-mgmt-1 has averaged 141.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 9:
  State:       Resolved
  Started:     2026-09-17T03:16:56Z
  Ended:       2026-09-17T03:35:58Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6437632-mgmt-2", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-66f56f4b8b-9fks7", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-66f56f4b8b-9fks7 (namespace mgmt-agent) on cluster ci01-j6437632-mgmt-2 has averaged 164.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 10:
  State:       Resolved
  Started:     2026-09-17T03:17:56Z
  Ended:       2026-09-17T03:36:59Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6437632-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-rdp9x", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-rdp9x (namespace arobit) on cluster ci01-j6437632-mgmt-1 has averaged 147.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 11:
  State:       Resolved
  Started:     2026-09-17T03:18:57Z
  Ended:       2026-09-17T03:40:56Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6437632-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-ljkp7", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-ljkp7 (namespace arobit) on cluster ci01-j6437632-mgmt-2 has averaged 165.1% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 12:
  State:       Resolved
  Started:     2026-09-17T03:22:00Z
  Ended:       2026-09-17T03:29:56Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6437632-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-fcrcj", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-fcrcj (namespace arobit) on cluster ci01-j6437632-svc has averaged 128.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 13:
  State:       Resolved
  Started:     2026-09-17T03:22:00Z
  Ended:       2026-09-17T03:34:59Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6437632-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-2hwp7", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-2hwp7 (namespace arobit) on cluster ci01-j6437632-mgmt-2 has averaged 129% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 14:
  State:       Resolved
  Started:     2026-09-17T03:23:58Z
  Ended:       2026-09-17T03:38:56Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6437632-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-62bl5", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-62bl5 (namespace arobit) on cluster ci01-j6437632-mgmt-1 has averaged 142.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 15:
  State:       Resolved
  Started:     2026-09-17T03:25:58Z
  Ended:       2026-09-17T03:31:58Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6437632-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-hbpdt", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-hbpdt (namespace arobit) on cluster ci01-j6437632-mgmt-1 has averaged 120.1% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000985200>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-4vtg2dzg4tsm" to finish creating: GET https://rp.j6537856.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/67e33856-41f6-4643-ae45-a797cf841c28
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/67e33856-41f6-4643-ae45-a797cf841c28",
      "name": "67e33856-41f6-4643-ae45-a797cf841c28",
      "status": "Failed",
      "startTime": "2026-09-17T02:46:55.291101558Z",
      "endTime": "2026-09-17T03:15:59.322698766Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 19m15s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000985200>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-4vtg2dzg4tsm" to finish creating: GET https://rp.j6537856.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/67e33856-41f6-4643-ae45-a797cf841c28
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/67e33856-41f6-4643-ae45-a797cf841c28",
      "name": "67e33856-41f6-4643-ae45-a797cf841c28",
      "status": "Failed",
      "startTime": "2026-09-17T02:46:55.291101558Z",
      "endTime": "2026-09-17T03:15:59.322698766Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 19m15s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc0010b2f60>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-hk9sm6mhjqks" to finish creating: GET https://rp.j6537856.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6a50ceeb-54cc-4ca7-b087-10be0245475a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6a50ceeb-54cc-4ca7-b087-10be0245475a",
      "name": "6a50ceeb-54cc-4ca7-b087-10be0245475a",
      "status": "Failed",
      "startTime": "2026-09-17T02:46:17.393539683Z",
      "endTime": "2026-09-17T03:15:19.381708079Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 18m38s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc0010b2f60>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-hk9sm6mhjqks" to finish creating: GET https://rp.j6537856.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6a50ceeb-54cc-4ca7-b087-10be0245475a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6a50ceeb-54cc-4ca7-b087-10be0245475a",
      "name": "6a50ceeb-54cc-4ca7-b087-10be0245475a",
      "status": "Failed",
      "startTime": "2026-09-17T02:46:17.393539683Z",
      "endTime": "2026-09-17T03:15:19.381708079Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 18m38s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:123]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc001712600>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-2vff89ssj75w" to finish creating: GET https://rp.j6537856.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/433f8dbb-9a26-40e6-b561-8f8f52095538
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/433f8dbb-9a26-40e6-b561-8f8f52095538",
      "name": "433f8dbb-9a26-40e6-b561-8f8f52095538",
      "status": "Failed",
      "startTime": "2026-09-17T02:46:24.868779354Z",
      "endTime": "2026-09-17T03:15:29.379097107Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 18m44s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:123]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc001712600>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-2vff89ssj75w" to finish creating: GET https://rp.j6537856.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/433f8dbb-9a26-40e6-b561-8f8f52095538
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/433f8dbb-9a26-40e6-b561-8f8f52095538",
      "name": "433f8dbb-9a26-40e6-b561-8f8f52095538",
      "status": "Failed",
      "startTime": "2026-09-17T02:46:24.868779354Z",
      "endTime": "2026-09-17T03:15:29.379097107Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 18m44s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000cdc120>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-p2sqgdjzvkvn" to finish creating: GET https://rp.j6537856.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6185eb10-51bb-4696-a53a-c5eaad9dc31d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6185eb10-51bb-4696-a53a-c5eaad9dc31d",
      "name": "6185eb10-51bb-4696-a53a-c5eaad9dc31d",
      "status": "Failed",
      "startTime": "2026-09-17T02:45:22.801463389Z",
      "endTime": "2026-09-17T03:14:29.359358725Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 18m20s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000cdc120>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-p2sqgdjzvkvn" to finish creating: GET https://rp.j6537856.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6185eb10-51bb-4696-a53a-c5eaad9dc31d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6185eb10-51bb-4696-a53a-c5eaad9dc31d",
      "name": "6185eb10-51bb-4696-a53a-c5eaad9dc31d",
      "status": "Failed",
      "startTime": "2026-09-17T02:45:22.801463389Z",
      "endTime": "2026-09-17T03:14:29.359358725Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 18m20s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc000a10e80>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-8nfj4dg6xpxk" to finish creating: GET https://rp.j6537856.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/063e2ac2-9588-4f9c-82b9-899a69317856
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/063e2ac2-9588-4f9c-82b9-899a69317856",
      "name": "063e2ac2-9588-4f9c-82b9-899a69317856",
      "status": "Failed",
      "startTime": "2026-09-17T02:48:07.392021884Z",
      "endTime": "2026-09-17T03:17:09.326897061Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc000a10e80>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-8nfj4dg6xpxk" to finish creating: GET https://rp.j6537856.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/063e2ac2-9588-4f9c-82b9-899a69317856
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/063e2ac2-9588-4f9c-82b9-899a69317856",
      "name": "063e2ac2-9588-4f9c-82b9-899a69317856",
      "status": "Failed",
      "startTime": "2026-09-17T02:48:07.392021884Z",
      "endTime": "2026-09-17T03:17:09.326897061Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc00103c200>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-zg4fwpppdnrm" to finish creating: GET https://rp.j6537856.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f9c0f6f2-b804-42ec-a112-a611d120c5aa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f9c0f6f2-b804-42ec-a112-a611d120c5aa",
      "name": "f9c0f6f2-b804-42ec-a112-a611d120c5aa",
      "status": "Failed",
      "startTime": "2026-09-17T02:45:44.397135399Z",
      "endTime": "2026-09-17T03:14:49.364484388Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 18m57s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc00103c200>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-zg4fwpppdnrm" to finish creating: GET https://rp.j6537856.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f9c0f6f2-b804-42ec-a112-a611d120c5aa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f9c0f6f2-b804-42ec-a112-a611d120c5aa",
      "name": "f9c0f6f2-b804-42ec-a112-a611d120c5aa",
      "status": "Failed",
      "startTime": "2026-09-17T02:45:44.397135399Z",
      "endTime": "2026-09-17T03:14:49.364484388Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 18m57s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000fb8740>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-4xqds56zcttv" to finish creating: GET https://rp.j6537856.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bb2db9bc-c46e-47f3-966f-caceef2356cf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bb2db9bc-c46e-47f3-966f-caceef2356cf",
      "name": "bb2db9bc-c46e-47f3-966f-caceef2356cf",
      "status": "Failed",
      "startTime": "2026-09-17T02:45:13.233682082Z",
      "endTime": "2026-09-17T03:14:19.374594663Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 18m6s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000fb8740>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-4xqds56zcttv" to finish creating: GET https://rp.j6537856.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bb2db9bc-c46e-47f3-966f-caceef2356cf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bb2db9bc-c46e-47f3-966f-caceef2356cf",
      "name": "bb2db9bc-c46e-47f3-966f-caceef2356cf",
      "status": "Failed",
      "startTime": "2026-09-17T02:45:13.233682082Z",
      "endTime": "2026-09-17T03:14:19.374594663Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 18m6s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 11 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T03:17:31Z
  Ended:       2026-09-17T03:23:29Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6537856-mgmt-2", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-7855548969-ghlbw", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-7855548969-ghlbw (namespace mgmt-agent) on cluster ci01-j6537856-mgmt-2 has averaged 166.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-17T03:17:31Z
  Ended:       2026-09-17T03:36:30Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6537856-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-9pnhk", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-9pnhk (namespace arobit) on cluster ci01-j6537856-mgmt-1 has averaged 157.5% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-17T03:17:31Z
  Ended:       2026-09-17T03:43:30Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6537856-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-8vbzs", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-8vbzs (namespace arobit) on cluster ci01-j6537856-mgmt-1 has averaged 182.8% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-17T03:17:31Z
  Ended:       2026-09-17T03:30:31Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6537856-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-fdc4t", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-fdc4t (namespace arobit) on cluster ci01-j6537856-mgmt-2 has averaged 177.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 5:
  State:       Resolved
  Started:     2026-09-17T03:17:31Z
  Ended:       2026-09-17T03:23:29Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6537856-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-7gz8n", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-7gz8n (namespace arobit) on cluster ci01-j6537856-mgmt-2 has averaged 174.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 6:
  State:       Fired (not resolved)
  Started:     2026-09-17T03:17:31Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6537856-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-77db6dcf77-r98d2", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-77db6dcf77-r98d2 (namespace clusters-service) on cluster ci01-j6537856-svc has averaged 836.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 7:
  State:       Resolved
  Started:     2026-09-17T03:17:31Z
  Ended:       2026-09-17T03:44:28Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6537856-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-l75wc", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-l75wc (namespace arobit) on cluster ci01-j6537856-svc has averaged 205.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 8:
  State:       Resolved
  Started:     2026-09-17T03:17:31Z
  Ended:       2026-09-17T03:47:30Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6537856-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-77db6dcf77-gc7h2", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-77db6dcf77-gc7h2 (namespace clusters-service) on cluster ci01-j6537856-svc has averaged 258.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 9:
  State:       Fired (not resolved)
  Started:     2026-09-17T03:17:31Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6537856-svc", component="service-memory", container="aro-hcp-backend", namespace="aro-hcp", pod="aro-hcp-backend-c4ccdfb64-796h2", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-backend in pod aro-hcp-backend-c4ccdfb64-796h2 (namespace aro-hcp) on cluster ci01-j6537856-svc has averaged 408.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 10:
  State:       Resolved
  Started:     2026-09-17T03:17:31Z
  Ended:       2026-09-17T03:46:29Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6537856-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-77db6dcf77-qq22x", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-77db6dcf77-qq22x (namespace clusters-service) on cluster ci01-j6537856-svc has averaged 242.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 11:
  State:       Resolved
  Started:     2026-09-17T03:17:32Z
  Ended:       2026-09-17T03:36:30Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6537856-mgmt-1", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-7855548969-zkvgl", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-7855548969-zkvgl (namespace mgmt-agent) on cluster ci01-j6537856-mgmt-1 has averaged 144.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc000f81380>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-cw4qg8klp5jk" to finish creating: GET https://rp.j2233600.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0a8f0e67-43cc-40f1-a32a-616ff94b6ac1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0a8f0e67-43cc-40f1-a32a-616ff94b6ac1",
      "name": "0a8f0e67-43cc-40f1-a32a-616ff94b6ac1",
      "status": "Failed",
      "startTime": "2026-09-17T01:51:55.906063696Z",
      "endTime": "2026-09-17T02:20:56.559080431Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m43s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc000f81380>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-cw4qg8klp5jk" to finish creating: GET https://rp.j2233600.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0a8f0e67-43cc-40f1-a32a-616ff94b6ac1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0a8f0e67-43cc-40f1-a32a-616ff94b6ac1",
      "name": "0a8f0e67-43cc-40f1-a32a-616ff94b6ac1",
      "status": "Failed",
      "startTime": "2026-09-17T01:51:55.906063696Z",
      "endTime": "2026-09-17T02:20:56.559080431Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m43s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 9 time(s) (1 unknown, 8 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T02:03:30Z
  Ended:       2026-09-17T02:34:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2233600-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssfjf9nv6ct1kgbo2ppvo99o553qc8o-v20260901", pod="router-585848dd75-gbvhv", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssfjf9nv6ct1kgbo2ppvo99o553qc8o-v20260901/router-585848dd75-gbvhv has been in a non-ready state for longer than 5 minutes.

alert fired 12 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T02:28:55Z
  Ended:       2026-09-17T02:36:55Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j2233600-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-jxhxg", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-jxhxg (namespace arobit) on cluster ci01-j2233600-mgmt-2 has averaged 138.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-17T02:28:55Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j2233600-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-89c4fbff5-l2x67", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-89c4fbff5-l2x67 (namespace clusters-service) on cluster ci01-j2233600-svc has averaged 834.5% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-17T02:28:55Z
  Ended:       2026-09-17T02:38:58Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j2233600-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-9wzdj", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-9wzdj (namespace arobit) on cluster ci01-j2233600-mgmt-1 has averaged 152.8% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-17T02:28:55Z
  Ended:       2026-09-17T02:40:00Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j2233600-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-c4vts", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-c4vts (namespace arobit) on cluster ci01-j2233600-mgmt-2 has averaged 145.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 5:
  State:       Resolved
  Started:     2026-09-17T02:28:55Z
  Ended:       2026-09-17T02:43:55Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j2233600-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-p8rsk", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-p8rsk (namespace arobit) on cluster ci01-j2233600-mgmt-1 has averaged 171.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 6:
  State:       Fired (not resolved)
  Started:     2026-09-17T02:28:55Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j2233600-svc", component="service-memory", container="aro-hcp-backend", namespace="aro-hcp", pod="aro-hcp-backend-6b8ff47d7-pwxfw", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-backend in pod aro-hcp-backend-6b8ff47d7-pwxfw (namespace aro-hcp) on cluster ci01-j2233600-svc has averaged 407.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 7:
  State:       Resolved
  Started:     2026-09-17T02:28:55Z
  Ended:       2026-09-17T02:40:56Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j2233600-mgmt-2", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-5d4d9cb45c-fgrxt", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-5d4d9cb45c-fgrxt (namespace mgmt-agent) on cluster ci01-j2233600-mgmt-2 has averaged 183.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 8:
  State:       Resolved
  Started:     2026-09-17T02:28:55Z
  Ended:       2026-09-17T02:52:58Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j2233600-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-89c4fbff5-52qjp", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-89c4fbff5-52qjp (namespace clusters-service) on cluster ci01-j2233600-svc has averaged 291.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 9:
  State:       Resolved
  Started:     2026-09-17T02:28:55Z
  Ended:       2026-09-17T02:42:57Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j2233600-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-cm5m2", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-cm5m2 (namespace arobit) on cluster ci01-j2233600-mgmt-2 has averaged 168.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 10:
  State:       Resolved
  Started:     2026-09-17T02:28:55Z
  Ended:       2026-09-17T02:49:59Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j2233600-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-89c4fbff5-2h4r9", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-89c4fbff5-2h4r9 (namespace clusters-service) on cluster ci01-j2233600-svc has averaged 241.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 11:
  State:       Resolved
  Started:     2026-09-17T02:28:55Z
  Ended:       2026-09-17T02:47:57Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j2233600-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-w8b66", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-w8b66 (namespace arobit) on cluster ci01-j2233600-svc has averaged 216.8% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 12:
  State:       Resolved
  Started:     2026-09-17T02:28:55Z
  Ended:       2026-09-17T02:40:00Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j2233600-mgmt-1", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-5d4d9cb45c-gh2st", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-5d4d9cb45c-gh2st (namespace mgmt-agent) on cluster ci01-j2233600-mgmt-1 has averaged 173.8% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T02:28:29Z
  Ended:       2026-09-17T02:34:28Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2233600-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="router", endpoint="http", environment="ci01", instance="10.128.64.195:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2233600/providers/microsoft.monitor/accounts/hcps-j2233600", namespace="ocm-arohcpci01-2ssfjf9nv6ct1kgbo2ppvo99o553qc8o-v20260901", pod="arohcp-monitor-kube-state-metrics-574f657d96-gzzmc", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ssfjf9nv6ct1kgbo2ppvo99o553qc8o-v20260901/router has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-17T02:28:29Z
  Ended:       2026-09-17T02:34:28Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2233600-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="router", endpoint="http", environment="ci01", instance="10.128.64.195:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2233600/providers/microsoft.monitor/accounts/hcps-j2233600", namespace="ocm-arohcpci01-2ssfjf9nv6ct1kgbo2ppvo99o553qc8o-v20260901", pod="arohcp-monitor-kube-state-metrics-574f657d96-gzzmc", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ssfjf9nv6ct1kgbo2ppvo99o553qc8o-v20260901/router has not matched the expected number of replicas for longer than 30 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:162]: failed to verify HCP cluster ephemeral-disk is viable
Unexpected error:
    <*errors.joinError | 0xc0017886a8>: 
    VerifyBasicAccess failed: failed to list services: Get "https://api.ephemeral-disk.foc3.westus3.aroapp-hcp.azure-test.net:443/api/v1/namespaces/default/services": dial tcp: lookup api.ephemeral-disk.foc3.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: read udp 10.130.98.17:49368->172.30.0.10:53: i/o timeout
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.ephemeral-disk.foc3.westus3.aroapp-hcp.azure-test.net:443/apis/apiregistration.k8s.io/v1/apiservices": dial tcp: lookup api.ephemeral-disk.foc3.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: read udp 10.130.98.17:49368->172.30.0.10:53: i/o timeout
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:162]: failed to verify HCP cluster ephemeral-disk is viable
Unexpected error:
    <*errors.joinError | 0xc0017886a8>: 
    VerifyBasicAccess failed: failed to list services: Get "https://api.ephemeral-disk.foc3.westus3.aroapp-hcp.azure-test.net:443/api/v1/namespaces/default/services": dial tcp: lookup api.ephemeral-disk.foc3.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: read udp 10.130.98.17:49368->172.30.0.10:53: i/o timeout
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.ephemeral-disk.foc3.westus3.aroapp-hcp.azure-test.net:443/apis/apiregistration.k8s.io/v1/apiservices": dial tcp: lookup api.ephemeral-disk.foc3.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: read udp 10.130.98.17:49368->172.30.0.10:53: i/o timeout
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:612]: failed to get node release images before downgrade
Unexpected error:
    <*fmt.wrapError | 0xc0019a0220>: 
    list nodes (nodePool=npdg-4-21): Get "https://api.a4m2z1c5e2c5i9i.2o87.westus3.aroapp-hcp.azure-test.net:443/api/v1/nodes": dial tcp: lookup api.a4m2z1c5e2c5i9i.2o87.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: read udp 10.130.98.17:59019->172.30.0.10:53: i/o timeout
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:612]: failed to get node release images before downgrade
Unexpected error:
    <*fmt.wrapError | 0xc0019a0220>: 
    list nodes (nodePool=npdg-4-21): Get "https://api.a4m2z1c5e2c5i9i.2o87.westus3.aroapp-hcp.azure-test.net:443/api/v1/nodes": dial tcp: lookup api.a4m2z1c5e2c5i9i.2o87.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: read udp 10.130.98.17:59019->172.30.0.10:53: i/o timeout
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:319]: failed to create storage account diagnostic setting for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-cv4zb86lrk99/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*exported.ResponseError | 0xc000f21140>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-cv4zb86lrk99/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster/providers/Microsoft.Insights/diagnosticSettings/shoebox-storage-diag
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: SubscriptionNotRegistered
    --------------------------------------------------------------------------------
    {
      "code": "SubscriptionNotRegistered",
      "message": "The subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' is not registered to use microsoft.insights."
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:319]: failed to create storage account diagnostic setting for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-cv4zb86lrk99/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*exported.ResponseError | 0xc000f21140>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-cv4zb86lrk99/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster/providers/Microsoft.Insights/diagnosticSettings/shoebox-storage-diag
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: SubscriptionNotRegistered
    --------------------------------------------------------------------------------
    {
      "code": "SubscriptionNotRegistered",
      "message": "The subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' is not registered to use microsoft.insights."
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-h9khdk" for cluster "cluster-nightly-4-20-h9khdk"
Unexpected error:
    <*fmt.wrapError | 0xc000adc920>: 
    failed to create NodePool np-h9khdk: failed starting nodepool creation "np-h9khdk" for cluster "cluster-nightly-4-20-h9khdk" in resourcegroup="rg-nightly-4-20-9v8vn2pfzhs7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-9v8vn2pfzhs7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-h9khdk/nodePools/np-h9khdk
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-16-210648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-h9khdk" for cluster "cluster-nightly-4-20-h9khdk"
Unexpected error:
    <*fmt.wrapError | 0xc000adc920>: 
    failed to create NodePool np-h9khdk: failed starting nodepool creation "np-h9khdk" for cluster "cluster-nightly-4-20-h9khdk" in resourcegroup="rg-nightly-4-20-9v8vn2pfzhs7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-9v8vn2pfzhs7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-h9khdk/nodePools/np-h9khdk
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-16-210648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc00079eb40>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-8mb2dp88brbx": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-8mb2dp88brbx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.22\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc00079eb40>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-8mb2dp88brbx": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-8mb2dp88brbx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.22\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-16-210648
Unexpected error:
    <*fmt.wrapError | 0xc000996700>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-r7m2fk" in resourcegroup="rg-np-version-upgrade-r7m2fk-5jlmq454vn2n": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-r7m2fk-5jlmq454vn2n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-r7m2fk/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-16-210648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-16-210648
Unexpected error:
    <*fmt.wrapError | 0xc000996700>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-r7m2fk" in resourcegroup="rg-np-version-upgrade-r7m2fk-5jlmq454vn2n": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-r7m2fk-5jlmq454vn2n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-r7m2fk/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-16-210648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:766]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-16-210648
Unexpected error:
    <*fmt.wrapError | 0xc001466840>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-psjtgg" resourcegroup="rg-np-dg-minor-psjtgg-2t4zvgfl9nx7": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-psjtgg-2t4zvgfl9nx7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-psjtgg/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-16-210648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:766]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-16-210648
Unexpected error:
    <*fmt.wrapError | 0xc001466840>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-psjtgg" resourcegroup="rg-np-dg-minor-psjtgg-2t4zvgfl9nx7": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-psjtgg-2t4zvgfl9nx7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-psjtgg/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-16-210648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc0020c7110>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-9jm8bqptv2wt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc0020c7110>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-9jm8bqptv2wt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc0014a8de0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-pzlb2tkmb5c5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc0014a8de0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-pzlb2tkmb5c5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:222]: failed to create nodepool "np-1" for authorized CIDRs cluster
Unexpected error:
    <*fmt.wrapError | 0xc00062e0a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-khvwxc7hm457" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9a805e6d-f4c1-448f-ac8d-eb4629159049
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9a805e6d-f4c1-448f-ac8d-eb4629159049",
      "name": "9a805e6d-f4c1-448f-ac8d-eb4629159049",
      "status": "Failed",
      "startTime": "2026-09-17T23:14:40.679038392Z",
      "endTime": "2026-09-17T23:33:46.186414821Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:222]: failed to create nodepool "np-1" for authorized CIDRs cluster
Unexpected error:
    <*fmt.wrapError | 0xc00062e0a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-khvwxc7hm457" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9a805e6d-f4c1-448f-ac8d-eb4629159049
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9a805e6d-f4c1-448f-ac8d-eb4629159049",
      "name": "9a805e6d-f4c1-448f-ac8d-eb4629159049",
      "status": "Failed",
      "startTime": "2026-09-17T23:14:40.679038392Z",
      "endTime": "2026-09-17T23:33:46.186414821Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.009s.
Expected success, but got an error:
    <*errors.joinError | 0xc000fe8ae0>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-16-151028, targetMinor=5.1.0-0.nightly-multi-2026-09-15-135250) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-09-15-135250"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.009s.
Expected success, but got an error:
    <*errors.joinError | 0xc000fe8ae0>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-16-151028, targetMinor=5.1.0-0.nightly-multi-2026-09-15-135250) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-09-15-135250"
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0xc0013e8bd0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-n8nkjmlflmx2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0xc0013e8bd0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-n8nkjmlflmx2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-16-210648
Unexpected error:
    <*fmt.wrapError | 0xc0007184c0>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-cqdwqx" in resourcegroup="rg-np-skip-minor-cqdwqx-w27zwxzrpszc": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-cqdwqx-w27zwxzrpszc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-cqdwqx/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-16-210648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-16-210648
Unexpected error:
    <*fmt.wrapError | 0xc0007184c0>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-cqdwqx" in resourcegroup="rg-np-skip-minor-cqdwqx-w27zwxzrpszc": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-cqdwqx-w27zwxzrpszc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-cqdwqx/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-16-210648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:274]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0xc002b70300>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-9mx94tspcdqp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:274]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0xc002b70300>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-9mx94tspcdqp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc002740c00>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-5ddg8tcqt24f" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a27ca6a0-c950-4180-9609-7a9ade0dfaf2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a27ca6a0-c950-4180-9609-7a9ade0dfaf2",
      "name": "a27ca6a0-c950-4180-9609-7a9ade0dfaf2",
      "status": "Failed",
      "startTime": "2026-09-17T22:32:08.724911504Z",
      "endTime": "2026-09-17T22:51:12.585683878Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc002740c00>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-5ddg8tcqt24f" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a27ca6a0-c950-4180-9609-7a9ade0dfaf2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a27ca6a0-c950-4180-9609-7a9ade0dfaf2",
      "name": "a27ca6a0-c950-4180-9609-7a9ade0dfaf2",
      "status": "Failed",
      "startTime": "2026-09-17T22:32:08.724911504Z",
      "endTime": "2026-09-17T22:51:12.585683878Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:319]: failed to create nodepool npnoedge-4-20 at version 4.20.37
Unexpected error:
    <*fmt.wrapError | 0xc000a4edc0>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-rrzr5s" in resourcegroup="rg-np-noedge-rrzr5s-tw6bdclngxdp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4e383065-95e4-447a-8c07-f8b7ac95870b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4e383065-95e4-447a-8c07-f8b7ac95870b",
      "name": "4e383065-95e4-447a-8c07-f8b7ac95870b",
      "status": "Failed",
      "startTime": "2026-09-17T22:45:09.171165434Z",
      "endTime": "2026-09-17T23:04:12.617623397Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:319]: failed to create nodepool npnoedge-4-20 at version 4.20.37
Unexpected error:
    <*fmt.wrapError | 0xc000a4edc0>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-rrzr5s" in resourcegroup="rg-np-noedge-rrzr5s-tw6bdclngxdp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4e383065-95e4-447a-8c07-f8b7ac95870b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4e383065-95e4-447a-8c07-f8b7ac95870b",
      "name": "4e383065-95e4-447a-8c07-f8b7ac95870b",
      "status": "Failed",
      "startTime": "2026-09-17T22:45:09.171165434Z",
      "endTime": "2026-09-17T23:04:12.617623397Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-rwd7s5" for cluster "cluster-candidate-4-20-rwd7s5"
Unexpected error:
    <*fmt.wrapError | 0xc0014fd540>: 
    failed to create NodePool np-rwd7s5: failed waiting for nodepool="np-rwd7s5" for cluster "cluster-candidate-4-20-rwd7s5" in resourcegroup="rg-candidate-4-20-dgf456bm6cvc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a5770130-02b8-4165-803f-f4f44190a860
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a5770130-02b8-4165-803f-f4f44190a860",
      "name": "a5770130-02b8-4165-803f-f4f44190a860",
      "status": "Failed",
      "startTime": "2026-09-17T22:08:49.966569897Z",
      "endTime": "2026-09-17T22:27:49.362137457Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-rwd7s5" for cluster "cluster-candidate-4-20-rwd7s5"
Unexpected error:
    <*fmt.wrapError | 0xc0014fd540>: 
    failed to create NodePool np-rwd7s5: failed waiting for nodepool="np-rwd7s5" for cluster "cluster-candidate-4-20-rwd7s5" in resourcegroup="rg-candidate-4-20-dgf456bm6cvc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a5770130-02b8-4165-803f-f4f44190a860
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a5770130-02b8-4165-803f-f4f44190a860",
      "name": "a5770130-02b8-4165-803f-f4f44190a860",
      "status": "Failed",
      "startTime": "2026-09-17T22:08:49.966569897Z",
      "endTime": "2026-09-17T22:27:49.362137457Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:139]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
    <*fmt.wrapError | 0xc000ddaa80>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-mkcwd2qnbq2q" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eef2b4be-3845-4db8-8a9e-a82745bc8bcd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eef2b4be-3845-4db8-8a9e-a82745bc8bcd",
      "name": "eef2b4be-3845-4db8-8a9e-a82745bc8bcd",
      "status": "Failed",
      "startTime": "2026-09-17T02:13:07.43461625Z",
      "endTime": "2026-09-17T02:32:13.121368333Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:139]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
    <*fmt.wrapError | 0xc000ddaa80>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-mkcwd2qnbq2q" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eef2b4be-3845-4db8-8a9e-a82745bc8bcd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eef2b4be-3845-4db8-8a9e-a82745bc8bcd",
      "name": "eef2b4be-3845-4db8-8a9e-a82745bc8bcd",
      "status": "Failed",
      "startTime": "2026-09-17T02:13:07.43461625Z",
      "endTime": "2026-09-17T02:32:13.121368333Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:148]: failed to verify cluster "cp-ystream-upgrade-4-22-j5vkzr" is viable before upgrade
Unexpected error:
    <*errors.joinError | 0xc0015d2858>: 
    VerifyBasicAccess failed: failed to list services: Get "https://api.z1t1t5s4i6u3r6h.s9td.uksouthstaging.aroapp-hcp.io:443/api/v1/namespaces/default/services": dial tcp: lookup api.z1t1t5s4i6u3r6h.s9td.uksouthstaging.aroapp-hcp.io on 172.30.0.10:53: read udp 10.130.98.22:39829->172.30.0.10:53: i/o timeout
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.z1t1t5s4i6u3r6h.s9td.uksouthstaging.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": dial tcp: lookup api.z1t1t5s4i6u3r6h.s9td.uksouthstaging.aroapp-hcp.io on 172.30.0.10:53: read udp 10.130.98.22:39829->172.30.0.10:53: i/o timeout
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:148]: failed to verify cluster "cp-ystream-upgrade-4-22-j5vkzr" is viable before upgrade
Unexpected error:
    <*errors.joinError | 0xc0015d2858>: 
    VerifyBasicAccess failed: failed to list services: Get "https://api.z1t1t5s4i6u3r6h.s9td.uksouthstaging.aroapp-hcp.io:443/api/v1/namespaces/default/services": dial tcp: lookup api.z1t1t5s4i6u3r6h.s9td.uksouthstaging.aroapp-hcp.io on 172.30.0.10:53: read udp 10.130.98.22:39829->172.30.0.10:53: i/o timeout
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.z1t1t5s4i6u3r6h.s9td.uksouthstaging.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": dial tcp: lookup api.z1t1t5s4i6u3r6h.s9td.uksouthstaging.aroapp-hcp.io on 172.30.0.10:53: read udp 10.130.98.22:39829->172.30.0.10:53: i/o timeout
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:139]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0035265a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-rcr2wwpjvcjv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5b397553-ee61-4674-b26e-e3cf8ae2dc9d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5b397553-ee61-4674-b26e-e3cf8ae2dc9d",
      "name": "5b397553-ee61-4674-b26e-e3cf8ae2dc9d",
      "status": "Failed",
      "startTime": "2026-09-17T05:39:36.012772677Z",
      "endTime": "2026-09-17T05:58:42.168749593Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:139]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0035265a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-rcr2wwpjvcjv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5b397553-ee61-4674-b26e-e3cf8ae2dc9d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5b397553-ee61-4674-b26e-e3cf8ae2dc9d",
      "name": "5b397553-ee61-4674-b26e-e3cf8ae2dc9d",
      "status": "Failed",
      "startTime": "2026-09-17T05:39:36.012772677Z",
      "endTime": "2026-09-17T05:58:42.168749593Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000fc2640>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-4xfvdn4xsffw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/98b08bb9-0eee-4840-9535-91464d8458a2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/98b08bb9-0eee-4840-9535-91464d8458a2",
      "name": "98b08bb9-0eee-4840-9535-91464d8458a2",
      "status": "Failed",
      "startTime": "2026-09-17T05:40:34.02322608Z",
      "endTime": "2026-09-17T05:59:42.272250066Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000fc2640>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-4xfvdn4xsffw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/98b08bb9-0eee-4840-9535-91464d8458a2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/98b08bb9-0eee-4840-9535-91464d8458a2",
      "name": "98b08bb9-0eee-4840-9535-91464d8458a2",
      "status": "Failed",
      "startTime": "2026-09-17T05:40:34.02322608Z",
      "endTime": "2026-09-17T05:59:42.272250066Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-72xhwq" for cluster "cluster-candidate-4-22-72xhwq"
Unexpected error:
    <*fmt.wrapError | 0xc0009bb400>: 
    failed to create NodePool np-72xhwq: failed waiting for nodepool="np-72xhwq" for cluster "cluster-candidate-4-22-72xhwq" in resourcegroup="rg-candidate-4-22-ffxlnxmpk4tb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8ee3e506-9091-4aa0-ad2c-045d3e78731e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8ee3e506-9091-4aa0-ad2c-045d3e78731e",
      "name": "8ee3e506-9091-4aa0-ad2c-045d3e78731e",
      "status": "Failed",
      "startTime": "2026-09-17T06:05:40.036081638Z",
      "endTime": "2026-09-17T06:24:44.492140979Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-72xhwq" for cluster "cluster-candidate-4-22-72xhwq"
Unexpected error:
    <*fmt.wrapError | 0xc0009bb400>: 
    failed to create NodePool np-72xhwq: failed waiting for nodepool="np-72xhwq" for cluster "cluster-candidate-4-22-72xhwq" in resourcegroup="rg-candidate-4-22-ffxlnxmpk4tb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8ee3e506-9091-4aa0-ad2c-045d3e78731e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8ee3e506-9091-4aa0-ad2c-045d3e78731e",
      "name": "8ee3e506-9091-4aa0-ad2c-045d3e78731e",
      "status": "Failed",
      "startTime": "2026-09-17T06:05:40.036081638Z",
      "endTime": "2026-09-17T06:24:44.492140979Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred